Robot control method and device, electronic equipment and storage medium

Abstract

The invention provides a robot control method and device, electronic equipment and a storage medium. Comprising the following steps: performing voice state recognition on a voice frame sequence of collected continuous voices to obtain voiced voice segments in the continuous voices; performing feature extraction on the voiced voice segments to obtain voice feature vectors; performing attention processing on the voice feature vector to obtain an attention feature vector of the voiced voice segment; determining a voice attribute recognition result and a voice content recognition result of the voiced voice segment based on the attention feature vector; and controlling the robot to execute a target behavior matched with the voice attribute recognition result and the voice content recognition result. The voice recognition accuracy can be improved through attention processing, and the accurate voice attribute recognition result and the voice content recognition result are obtained, so that the robot can more accurately understand and respond to the requirements of the user through the voice attribute recognition result and the voice content recognition result, and the interaction ability of the robot is improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot control method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application of robots in multiple fields has become more and more extensive. Robots can learn users' needs and preferences through interaction, and thus provide more personalized and accurate services. However, during the interaction process, it is necessary to accurately recognize the voice content and make corresponding responses based on the recognition results.

[0003] In related technologies, conventional machine learning algorithms are usually used to recognize user voices, which can only recognize the text content corresponding to the voice, but cannot recognize the user's emotions and sound events, thereby reducing the accuracy of the robot's understanding of user needs and reducing the robot's interactive capabilities. Summary of the invention

[0004] The embodiments of the present application provide a robot control method, device, electronic device and storage medium, which can improve the accuracy of speech recognition through attention processing, and obtain accurate speech attribute recognition results and speech content recognition results. Therefore, through the speech attribute recognition results and speech content recognition results, the robot can more accurately understand and respond to user needs, thereby improving the robot's interactive ability.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a robot control method, which includes: performing speech state recognition on a speech frame sequence of collected continuous speech to obtain an audible speech segment in the continuous speech; performing feature extraction on the audible speech segment to obtain a speech feature vector; performing attention processing on the speech feature vector to obtain an attention feature vector of the audible speech segment; determining a speech attribute recognition result and a speech content recognition result of the audible speech segment based on the attention feature vector; and controlling the robot to execute a target behavior that matches the speech attribute recognition result and the speech content recognition result.

[0007] An embodiment of the present application provides a robot control device, including: a recognition module, used to perform voice state recognition on a voice frame sequence of collected continuous speech to obtain an audible speech segment in the continuous speech; a feature extraction module, used to perform feature extraction on the audible speech segment to obtain a voice feature vector; an attention processing module, used to perform attention processing on the speech feature vector to obtain an attention feature vector of the audible speech segment; a determination module, used to determine a voice attribute recognition result and a voice content recognition result of the audible speech segment based on the attention feature vector; and a control module, used to control the robot to perform a target behavior that matches the voice attribute recognition result and the voice content recognition result.

[0008] In the above scheme, the recognition module is also used to: perform a first preprocessing on the collected speech frame sequence of continuous speech to obtain a preprocessed speech frame sequence; call a pre-trained speech activity detection model to perform a first feature mapping on the preprocessed speech frame sequence to obtain a speech probability value of each speech frame in the preprocessed speech frame sequence; the speech probability value is used to characterize the probability of the presence of sound in the speech frame; based on the speech probability value of each speech frame, determine the voiced speech segment in the continuous speech.

[0009] In the above scheme, the recognition module is also used to: obtain a preset first speech probability threshold and a preset second speech probability threshold; wherein the second speech probability threshold is less than the first speech probability threshold; determine the speech frame whose speech probability value is greater than the first speech probability threshold as a voiced speech frame; in response to the first preceding frame of two adjacent speech frames being a silent speech frame, and the first succeeding frame of the two speech frames being the voiced speech frame, determine the first succeeding frame as a voiced speech start frame, and cache the speech frames starting from the voiced speech start frame; in the process of caching the speech frames, in response to When the second previous frame between two adjacent speech frames is the voiced speech frame, and the speech probability value of the second next frame between the two speech frames is less than the second speech probability threshold, the speech frame duration from the voiced speech start frame to the second next frame is obtained; in response to the speech frame duration being greater than a preset detection silence duration, the second previous frame is determined as a voiced speech end frame, and speech frames are stopped from being cached after the voiced speech end frame is cached; the voiced speech start frame, the voiced speech end frame, and the speech frames between the voiced speech start frame and the voiced speech end frame are determined as the voiced speech segment.

[0010] In the above scheme, the recognition module is also used to: in response to the voice frame duration being less than or equal to the preset detection silence duration, determine the second subsequent frame as the voiced voice frame, and continue to cache the second subsequent frame.

[0011] In the above scheme, the feature extraction module is also used to: perform a second preprocessing on the voiced speech segment to obtain a preprocessed sound feature; perform feature extraction on the preprocessed sound feature to obtain a preprocessed sound feature vector; obtain preset index data and perform data encoding on the index data to obtain a coded feature vector; perform a first feature splicing on the preprocessed sound feature vector and the coded feature vector to obtain the speech feature vector.

[0012] In the above scheme, the second preprocessing includes a first preprocessing operation and a second preprocessing operation; the feature extraction module is also used to: perform the first preprocessing operation on the voiced speech segment to obtain preprocessing features; perform the second preprocessing operation on the preprocessing features to obtain the preprocessed sound features.

[0013] In the above scheme, the feature extraction module is also used to: perform frame processing on the voiced speech segment to obtain N analysis frames, where N is an integer greater than 1; perform normalization processing on the N analysis frames to obtain normalized features; perform fast Fourier transform on the normalized features to obtain an amplitude spectrum; perform Mel feature extraction on the amplitude spectrum to obtain a Mel spectrum feature; and determine the preprocessing feature based on the Mel spectrum feature.

[0014] In the above scheme, the feature extraction module is also used to: downsample the preprocessing features to obtain downsampled features; and normalize the downsampled features to obtain the preprocessing sound features.

[0015] In the above scheme, the determination module is also used to: decode the attention feature vector to obtain an attribute label probability value set and a content probability value set; each label probability value in the attribute label probability value set corresponds to a preset label in the preset label set; the label probability value is used to characterize the probability that the corresponding preset label matches the voiced speech segment; the preset label corresponding to the maximum label probability value in the attribute label probability value set is determined as the target label; based on the content probability value in the content probability value set, the index of each text unit in the text content is determined; based on the target label, the speech attribute recognition result of the voiced speech segment is determined, and based on the index, the speech content recognition result of the voiced speech segment is determined.

[0016] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer executable instructions; and a processor for implementing the robot control method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium storing a computer program or executable instructions. When the computer program or computer executable instructions are executed by a processor, the robot control method provided in the embodiment of the present application is implemented.

[0018] An embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the computer executable instructions from the computer-readable storage medium and executes the computer executable instructions, the robot control method provided in the embodiment of the present application is implemented.

[0019] The embodiments of the present application have the following beneficial effects:

[0020] When controlling the robot, through voice state recognition, silent or invalid voice segments can be effectively filtered out in continuous speech, and only voiced voice segments can be extracted, thereby reducing the interference of useless data; moreover, by introducing the attention mechanism, the voice feature vector can focus on the key feature part, thereby improving the expressiveness of the voice features and improving the accuracy of voice recognition; in addition, the voice recognition results of the present application include voice attribute recognition results and voice content recognition results, realizing multi-level analysis of voice information, so that the robot can understand user needs more accurately; and, according to the voice attribute recognition results and the voice content recognition results, the robot behavior is controlled, so that the robot can perform precise and personalized target behaviors according to the voice input, thereby improving the robot's interactive ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is an optional flowchart of the robot control method provided in the embodiment of the present application;

[0022] Figure 2 It is a flowchart of determining a voiced speech segment in continuous speech based on the speech probability of each speech frame provided by an embodiment of the present application;

[0023] Figure 3 It is a flow chart of performing a first preprocessing operation on a voiced speech segment to obtain preprocessing features provided by an embodiment of the present application;

[0024] Figure 4 It is a flow chart of performing a second preprocessing operation on a preprocessing feature to obtain a preprocessing sound feature provided by an embodiment of the present application;

[0025] Figure 5 It is a structural schematic diagram of an attention feature vector of a voiced speech segment obtained by performing attention processing on a speech feature vector provided in an embodiment of the present application;

[0026] Figure 6 It is a flowchart of determining the speech attribute recognition result and the speech content recognition result of the voiced speech segment based on the attention feature vector provided by the embodiment of the present application;

[0027] Figure 7 This is a diagram of the speech recognition architecture provided by an embodiment of the present application;

[0028] Figure 8 is a structural block diagram of a robot control device provided in an embodiment of the present application;

[0029] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] If similar descriptions of "first / second" appear in the application documents, the following instructions are added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0034] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] In the embodiment of the present application, when controlling the robot, first, the voice state recognition is performed on the voice frame sequence of the collected continuous speech to obtain the voiced voice segment in the continuous speech; the voiced voice segment is feature extracted to obtain the voice feature vector; the voice feature vector is subjected to attention processing to obtain the attention feature vector of the voiced voice segment; based on the attention feature vector, the voice attribute recognition result and the voice content recognition result of the voiced voice segment are determined; the robot is controlled to perform the target behavior matching the voice attribute recognition result and the voice content recognition result. In this way, through voice state recognition, the silent or invalid voice segment can be effectively filtered out in the continuous speech, and only the voiced voice segment can be extracted, thereby reducing the interference of useless data; moreover, by introducing the attention mechanism, the voice feature vector can focus on the key feature part, thereby improving the expression ability of the voice feature and improving the accuracy of voice recognition; in addition, the voice recognition result of the present application includes the voice attribute recognition result and the voice content recognition result, realizing the multi-level analysis of the voice information, so that the robot can understand the user needs more accurately; and, according to the voice attribute recognition result and the voice content recognition result, the robot behavior is controlled, so that the robot can achieve accurate and personalized target behavior according to the voice input, and improve the robot's interactive ability.

[0036] The robot control method provided in the embodiments of the present application can be applied to electronic devices such as robots, laptop computers, tablet computers, desktop computers, smart home appliances, and smart vehicle-mounted devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0037] The robot control method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0038] Figure 1 1 is an optional flow chart of a robot control method provided in an embodiment of the present application, and the method can be applied to electronic devices. The following will be described by taking the electronic device as a robot as an example. Figure 1 As shown, the method includes the following steps S101 to S105:

[0039] Step S101, performing speech state recognition on the collected speech frame sequence of continuous speech to obtain the voiced speech segments in the continuous speech.

[0040] Continuous speech refers to natural, smooth, and uninterrupted speech information, which may contain pauses between speeches, background noise, etc. Continuous speech can have a preset sampling rate, for example, the preset sampling rate can often be 16000HZ or 8000HZ. Continuous speech can be collected through the robot's built-in microphone array (single or multiple), and multiple microphone arrays can achieve noise reduction, speech enhancement, and sound source localization. The robot can adapt to a variety of speech collection environments, including quiet indoor scenes and noisy outdoor scenes. The microphone array uses noise suppression technology and signal enhancement algorithms to ensure the quality of the continuous speech collected by the robot.

[0041] Speech frame sequence refers to a series of short-term speech frames that are divided into a continuous speech signal at fixed time intervals, and each speech frame contains a certain number of sampling points. Speech state recognition refers to analyzing the characteristics of speech frames in a speech frame sequence through algorithms and determining whether each speech frame is voiced or silent.

[0042] A voiced speech segment refers to an audio segment containing speech content extracted from continuous speech. A voiced speech segment is composed of multiple consecutive voiced speech frames.

[0043] In some embodiments, step S101 can be implemented by the following method: first, a first preprocessing is performed on the collected speech frame sequence of continuous speech to obtain a preprocessed speech frame sequence; then, a pre-trained speech activity detection model is called to perform a first feature mapping on the preprocessed speech frame sequence to obtain a speech probability value for each speech frame in the preprocessed speech frame sequence; the speech probability value is used to characterize the probability of the presence of sound in the speech frame; finally, based on the speech probability value of each speech frame, the voiced speech segment in the continuous speech is determined.

[0044] The first preprocessing refers to the preliminary processing of the speech frame sequence of the collected continuous speech. The first preprocessing includes noise reduction, pre-emphasis and windowing, among which noise reduction refers to reducing environmental noise through a noise reduction algorithm. Specific noise reduction algorithms include spectral subtraction, Wiener filtering or noise reduction through a deep learning model; pre-emphasis refers to amplifying the high-frequency components in each speech frame, weakening the influence of low-frequency components on the speech signal, and enhancing the high-frequency characteristics of the speech signal; windowing refers to applying a window function to each speech frame to reduce spectrum leakage. Commonly used windowing functions include Hamming window, rectangular window, etc.; the specific preprocessing operations are not limited in this application.

[0045] A pre-trained voice activity detection model refers to a model that has been trained on a large-scale data set. The pre-trained voice activity detection model is used to detect whether there is effective sound in each frame of the voice signal. The pre-trained voice activity detection model can be implemented as a traditional machine learning model, such as a hidden Markov model, a support vector machine, etc.; the pre-trained voice activity detection model can also be implemented as a deep learning model, such as a deep neural network model, a recurrent neural network model, etc.; this application does not limit this. Effective sound refers to a sound that has practical meaning in a voice signal, usually including human voices, music, alarm sounds, and other sound signals with specific information or functions.

[0046] The first feature mapping refers to extracting features from each frame of speech signal through a pre-trained speech activity detection model, mapping each frame of speech signal to a speech probability space, and outputting a speech probability value for each frame of speech signal; the speech probability value is used to characterize the probability of the presence of sound in a speech frame.

[0047] In some embodiments, see Figure 2 , Figure 2 is a flow chart of determining a voiced speech segment in continuous speech based on the speech probability of each speech frame provided by an embodiment of the present application, Figure 2 It is shown that based on the speech probability value of each speech frame, determining the voiced speech segment in the continuous speech can be achieved by the following steps S201 to S206:

[0048] Step S201, obtaining a preset first speech probability threshold and a preset second speech probability threshold; wherein the second speech probability threshold is smaller than the first speech probability threshold.

[0049] In some embodiments, the preset first speech probability threshold is a probability threshold used to determine whether a speech frame significantly belongs to a voiced speech segment; the preset second speech probability threshold is a probability threshold used to determine whether a speech frame may belong to a voiced speech segment; the second speech probability threshold is less than the first speech probability threshold.

[0050] Step S202: Determine a speech frame whose speech probability value is greater than a first speech probability threshold as a voiced speech frame.

[0051] In some embodiments, when the speech probability value of speech frame A is greater than a first speech probability threshold, speech frame A is determined to be a voiced speech frame.

[0052] Step S203, in response to the first preceding frame of two adjacent speech frames being an unvoiced speech frame and the first succeeding frame of the two speech frames being a voiced speech frame, the first succeeding frame is determined as a voiced speech start frame, and speech frames are cached starting from the voiced speech start frame.

[0053] A silent speech frame refers to a speech frame without any sound signal; a voiced speech start frame refers to the first frame where the speech frame switches from a silent state to a voiced state, and is used to mark the starting point of a voiced speech segment.

[0054] In some embodiments, there are adjacent speech frames A and B, where speech frame A is the first previous frame and speech frame B is the first subsequent frame; wherein speech frame A is a silent speech frame, that is, there is no valid sound in speech frame A, and speech frame B is determined to be a voiced speech frame through a speech probability value, then speech frame B is determined to be a voiced speech start frame, and speech frames are cached starting from speech frame B.

[0055] Step S204, in the process of caching speech frames, in response to the second previous frame of two adjacent speech frames being a voiced speech frame, and the speech probability value of the second after frame of the two speech frames being less than the second speech probability threshold, obtain the speech frame duration from the voiced speech start frame to the second after frame.

[0056] The speech frame duration refers to the duration between the sound start frame and the second subsequent frame. The calculation method of the speech frame duration is: speech frame duration = number of frames × single frame duration.

[0057] In some embodiments, after caching starts from the start frame B of the voiced speech, there are adjacent speech frames C and speech frames D, where speech frame C is the second previous frame and speech frame D is the second subsequent frame; when the speech probability value of speech frame D is less than a preset second speech probability threshold, the duration from the start frame B of the voiced speech to speech frame D is determined.

[0058] Step S205, in response to the voice frame duration being greater than the preset detection silence duration, determining the second previous frame as a voiced voice termination frame, and stopping caching voice frames after caching the voiced voice termination frame.

[0059] The preset detection silence duration is a preset duration threshold, which is used to determine whether the current voice segment has ended; the voiced speech termination frame refers to the last voiced speech frame of the voiced speech segment, and the subsequent speech frames after the voiced speech termination frame are considered to be silent speech frames.

[0060] In some embodiments, assuming that the preset detection silence duration is 100ms, the duration from the voiced speech start frame B to the speech frame D is calculated to be 125ms. At this time, the speech frame duration is greater than the preset detection silence duration, then the speech frame C is determined as the voiced speech termination frame, and the speech frame caching is stopped after the speech frame C is cached.

[0061] Step S206: determine the voiced speech start frame, the voiced speech end frame, and the speech frames between the voiced speech start frame and the voiced speech end frame as voiced speech segments.

[0062] In some embodiments, assuming that there are 5 speech frames between the voiced speech start frame B and the voiced speech end frame C, namely speech frames 1 to 5, the voiced speech start frame B, the voiced speech end frame C and speech frames 1 to 5 are spliced ​​to obtain a voiced speech segment.

[0063] In some embodiments, in response to the speech frame duration being less than or equal to a preset detected silence duration, the second subsequent frame is determined as a voiced speech frame, and the second subsequent frame continues to be cached.

[0064] Here, when the preset detection silence duration is 100ms, by calculating that the duration from the voiced speech start frame B to the speech frame D is 75ms, at this time, the speech frame duration is less than the preset detection silence duration, then the speech frame D is determined as a voiced speech frame, and the speech frame D is cached.

[0065] In some embodiments, after determining the voiced speech termination frame, it can be determined that the speech frame after the voiced speech termination frame is a silent speech frame, and the speech frame after the voiced speech termination frame can be used as the first previous frame in the next voiced speech segment recognition process.

[0066] Step S102, extracting features from the voiced speech segment to obtain a speech feature vector.

[0067] Feature extraction refers to the process of extracting the core features that can represent the input data from the input data; the speech feature vector refers to the numerical representation of the core information of the speech signal.

[0068] In some embodiments, step S102 can be implemented by the following method: first, perform a second preprocessing on the voiced speech segment to obtain preprocessed sound features; then, perform feature extraction on the preprocessed sound features to obtain a preprocessed sound feature vector; then, obtain preset index data and perform data encoding on the index data to obtain a coded feature vector; finally, perform a first feature splicing on the preprocessed sound feature vector and the coded feature vector to obtain a speech feature vector.

[0069] The second preprocessing refers to the preliminary processing of the voiced speech segment, including framing, windowing, pre-emphasis, etc. The specific preprocessing method is not limited in this application.

[0070] Index data refers to index tags associated with voiced speech segments, which may include language category index, speech emotion index, acoustic event detection index, and digital regularization index, etc. The specific index tags are not limited in this application. Data encoding refers to converting index data into a fixed format or numerical representation, which can be performed by one-hot encoding, numerical normalization encoding, etc.; the encoded feature vector refers to the numerical representation obtained after encoding the index data.

[0071] As an example of step S102, first, the voiced speech segment is preprocessed to obtain preprocessed sound features; then, the preprocessed sound features are feature extracted by a pre-trained feature extraction module to obtain a preprocessed sound feature vector; then, preset index data related to the voiced speech segment is obtained, including but not limited to a speech category index, a speech emotion index, and an acoustic event detection index, and feature encoding is performed on the index data to obtain an encoded feature vector; finally, feature concatenation is performed on the encoded feature vector and the preprocessed sound feature vector to obtain a speech feature vector; for example: the encoded feature vector is [0.2, 0.5, 0.8], and the preprocessed sound feature vector is [0.4, 0.4, 0.5, 0.3], and the encoded feature vector and the preprocessed sound feature vector are feature concatenated to obtain a speech feature vector [0.2, 0.5, 0.8, 0.4, 0.4, 0.5, 0.3].

[0072] In some embodiments, the second preprocessing includes a first preprocessing operation and a second preprocessing operation; preprocessing the voiced speech segment to obtain preprocessed sound features can be achieved by the following method: first, performing a first preprocessing operation on the voiced speech segment to obtain preprocessed features; then, performing a second preprocessing operation on the preprocessed features to obtain preprocessed sound features.

[0073] Here, the first preprocessing operation includes operations such as framing, DC removal, pre-emphasis and windowing; the second preprocessing operation includes downsampling and normalization operations; the specific processes of the first preprocessing operation and the second preprocessing operation are shown below.

[0074] In some embodiments, see Figure 3 , Figure 3 It is a flow chart of performing a first preprocessing operation on a voiced speech segment to obtain preprocessing features provided by an embodiment of the present application; Figure 3 It is shown that a first preprocessing operation is performed on the voiced speech segment to obtain preprocessing features, which can be achieved by the following steps S301 to S305:

[0075] Step S301, performing frame processing on the voiced speech segment to obtain N analysis frames, where N is an integer greater than 1.

[0076] Framing processing refers to dividing the voiced speech segments according to fixed time windows (frame length) and frame shifts to form analysis frames of short time periods; frame length refers to the length of time covered by each frame of audio data in the framing processing, usually in milliseconds. For example, a frame length of 25ms means that each frame contains 25ms of speech signal; frame shift refers to the time interval between the starting positions of two adjacent frames in the framing processing.

[0077] In some embodiments, step S301 can be implemented by the following method: first, the voiced speech segment is sampled to obtain a sampling point sequence; then, a preset frame length and frame shift are obtained, such as a frame length of 25ms (corresponding to 400 sampling points) and a frame shift of 10ms (corresponding to 160 sampling points); then, the voiced speech is intercepted at the starting position according to the preset frame length, and 400 sampling points are intercepted as the first analysis frame; then, the frame is moved according to the frame shift, and after moving 160 sampling points, 400 sampling points are continued to be intercepted as the second analysis frame, and so on, the entire sampling point sequence is intercepted to obtain N analysis frames.

[0078] Step S302, normalizing the N analysis frames to obtain normalized features.

[0079] In some embodiments, the normalization process includes performing DC removal, pre-emphasis, windowing, and backward zero padding on the analysis frame in sequence. The DC removal operation refers to removing the DC component in the speech signal during the speech signal processing process, and the DC component refers to the part of the speech signal that does not change with time; the calculation formula of the DC removal operation is shown in Formula 1:

[0080]

[0081] Wherein, n represents the sampling point; N represents the total number of sampling points of the analysis frame x; x(n) represents the amplitude of the analysis frame x; and x′(n) represents the amplitude of the analysis frame x after removing DC.

[0082] The pre-emphasis operation refers to enhancing the high frequency part in each analysis frame so that the high frequency part is more prominent than the low frequency part. The calculation formula of the pre-emphasis operation is shown in Formula 2:

[0083] x″(n)=x′(n)-α·x′(n-1) Formula 2

[0084] Wherein, α is the pre-emphasis coefficient, which is preset and usually close to 1 but cannot be equal to 1; x′(n-1) represents the amplitude of the previous frame of the analysis frame x after DC removal; x″(n) represents the amplitude after the pre-emphasis operation is performed on the analysis frame x after DC removal.

[0085] The windowing operation refers to the operation of applying a window function to the framed speech signal for weighted processing during the speech signal processing process; common window functions include: rectangular window, Hamming window and Gaussian window, etc., and the specific windowing function is not limited in this application; taking the Hamming window as an example, the formula for the windowing operation by the Hamming window function is shown in Formula 3:

[0086] x″′(n)=x″(n)·w(n) Formula 3

[0087] Wherein, x″′(n) represents the amplitude after the Hamming window operation is performed on the analysis frame x after the pre-emphasis operation; w(n) is the Hamming window weight, and the calculation formula of the Hamming window weight is shown in Formula 4:

[0088]

[0089] Where n represents the sampling point number; N represents the total number of sampling points of the analysis frame x.

[0090] Backward zero padding refers to the process of adding zero values ​​to the end of each analysis frame during speech signal processing; since the analysis frame needs to be fast Fourier transformed, and the input of the fast Fourier transform must be 2 n Therefore, each analysis frame needs to be backfilled with zeros so that each analysis frame can be fast Fourier transformed. For example, if the current analysis frame has 400 sampling points, 112 sampling points need to be backfilled.

[0091] In some embodiments, after normalization processing is performed on the N analysis frames respectively, a normalized feature corresponding to each analysis frame is obtained.

[0092] Step S303, performing fast Fourier transform on the normalized features to obtain an amplitude spectrum.

[0093] In some embodiments, the fast Fourier transform is a mathematical transformation used to convert a time domain signal into a frequency domain signal; the amplitude spectrum is the modulus of the fast Fourier transform result, that is, the intensity of each frequency component in the frequency domain signal, and the amplitude spectrum is used to characterize the energy distribution of each frequency component in the speech signal. The calculation formula of step S303 is shown in formula 5:

[0094] |X(k)|=|FFT{x″′(n)}| Formula 5

[0095] Wherein, X(k) represents the result of fast Fourier transform of the normalized features of the analysis frame x; |X(k)| represents the amplitude spectrum; and FFT{x″′(n)} represents the result of fast Fourier transform of the normalized features of the analysis frame x.

[0096] Step S304: perform Mel feature extraction on the amplitude spectrum to obtain Mel spectrum features.

[0097] In some embodiments, Mel feature extraction is a process of converting a linear frequency scale into a Mel frequency scale that conforms to the human ear perception characteristics; Mel spectrum feature is a frequency domain representation obtained after Mel frequency filtering of the amplitude spectrum, and Mel spectrum feature is used to characterize the energy distribution of the signal on the Mel frequency; wherein, the amplitude spectrum is usually filtered using a triangular filter, and each triangular filter covers a certain frequency range. The calculation formula for filtering the amplitude spectrum using a triangular filter is shown in Formula 6:

[0098]

[0099] Among them, Mel(m) represents the Mel spectrum feature; m refers to the number of triangular filters; k represents the number of frequency points, that is, the sampling points; K represents the total number of frequency points, that is, the total number of sampling points; M L (m) refers to the Mel frequency of the vertex to the left of triangular filter m; M R (m) refers to the Mel frequency of the vertex to the right of triangular filter m; M C (m) refers to the Mel frequency of the vertex on the triangular filter m; M M (k) refers to the Mel frequency corresponding to frequency point k.

[0100] Here, M L (m), M C (m) and M R The calculation formulas of (m) are shown in Formula 7, Formula 8 and Formula 9 respectively:

[0101] M L (m) = M l +m·M Δ Formula 7

[0102] M C (m) = M l +(m+1)·M Δ Formula 8

[0103] M R (m) = M l +(m+2)·M Δ Formula 9

[0104] Among them, M l Refers to the lower limit of the total Mel frequency; M Δ Refers to the Mel frequency interval; M l The calculation formula is shown in formula 10, M Δ The calculation formula is shown in Formula 11:

[0105]

[0106] Among them, f l is the lower limit of the Mel spectrum; f h is the upper limit of the Mel spectrum; M is the dimension of the Mel spectrum feature (the total number of triangular filters), which is a fixed value of 80.

[0107] Here, M M The calculation formula of (k) is shown in Formula 12:

[0108]

[0109] Where k represents the frequency point; W k Indicates the spectrum width of frequency k, W k The calculation formula is shown in Formula 13:

[0110]

[0111] Among them, K represents the total number of frequency points; F represents the sampling rate of the collected audio.

[0112] Step S305: determining preprocessing features based on the mel spectrum features.

[0113] In some embodiments, step S305 can be implemented by the following method: taking the logarithm of the mel-spectrum feature to obtain the mel-log spectrum feature; and determining the mel-log spectrum feature as the preprocessing feature. The formula for determining the mel-log spectrum feature is shown in Formula 14:

[0114] log Mel (m) = ln {Mel (m)} Formula 14

[0115] Among them, log Mel(m) represents the Mel logarithmic spectrum feature; Mel(m) represents the Mel spectrum feature.

[0116] It should be noted that the dimension of the preprocessing feature is {T, M}, where T is the number of voiced speech frames, which is determined by framing the voiced speech segments, and M is the dimension of the Mel-spectrogram feature, which is a fixed value of 80.

[0117] In some embodiments, see Figure 4 , Figure 4 It is a flow chart of performing a second preprocessing operation on a preprocessing feature to obtain a preprocessing sound feature provided by an embodiment of the present application; Figure 4 It shows that a second preprocessing operation is performed on the preprocessing feature to obtain the preprocessing sound feature, which can be achieved by the following steps S401 to S402:

[0118] Step S401, down-sample the pre-processed features to obtain down-sampled features.

[0119] In some embodiments, downsampling refers to merging the speech frame sequences obtained after the voiced speech segment is framed according to a certain ratio to reduce the number of speech frames; the commonly used downsampling method is low frame rate (LFR) calculation; the formula for LFR calculation is shown in Formula 15:

[0120]

[0121] Where i represents the speech block; j represents the frame; m represents the triangular filter; u represents the number of frames covered by each speech block, which is 7 by default; v represents the number of frames stepped by each speech block, which is 6 by default; r represents the number of first frames supplemented by the first speech block; T represents the number of voiced speech frames; C() represents the speech frame; M is the total number of triangular filters (the dimension of the Mel spectrum feature); Indicates rounding up.

[0122] In some embodiments, assuming that the number of voiced speech frames is 121 frames, the dimension of the preprocessing feature is {111, 80}, u=7, v=6, r=3, p=3; wherein the calculation formula of r is shown in Formula 16, p is the number of the last speech block to supplement the last frame, and the calculation formula of p is shown in Formula 17:

[0123]

[0124] At this time, speech block 0 is composed of speech frames [C(0), C(0), C(0), C(0), C(1), C(2), C(3)]; speech block 1 is composed of speech frames [C(3), C(4), C(5), C(6), C(7), C(8), C(9)]; and so on. It consists of speech frames [C(111), C(112), C(113), C(114), C(115), C(116), C(117)]; speech block It consists of speech frames [C(117), C(118), C(119), C(120), C(120), C(120), C(120)].

[0125] Step S402, normalize the downsampled features to obtain preprocessed sound features.

[0126] In some embodiments, in speech signal processing, cepstral mean and variance normalization (CMVN) is usually used to normalize the downsampled features. The calculation formula of the CMVN normalization method is shown in Formula 18:

[0127] cmvn(i,m&u)=[lfr(i,m&y)+mean cmvn (m&u)]·v cmvn (m&y) Formula 18

[0128] Among them, mean cmvn is the mean value; v cmvn is the variance; both the mean and variance are predetermined.

[0129] Step S103, performing attention processing on the speech feature vector to obtain the attention feature vector of the voiced speech segment.

[0130] Attention processing is used to capture key information or key areas in the input features and weight them according to their importance. The attention feature vector refers to the result of attention processing. Compared with the speech feature vector, the attention feature vector focuses more on key information and can better reflect the speech characteristics.

[0131] In some embodiments, see Figure 5 , Figure 5 It is a structural diagram of the attention feature vector of the voiced speech segment obtained by attention processing of the speech feature vector provided in an embodiment of the present application; step S103 can be implemented by the following method: first, linear mapping is performed on the speech feature vector 501 to obtain a query vector 502, a key vector 503 and a value vector 504; then, a dot product operation is performed on the query vector 502 and the key vector 503 to obtain a similarity score; and the similarity score is normalized to obtain an attention mechanism weight; then, a weighted summation is performed on the value vector 504 by the attention mechanism weight to obtain a weighted feature vector; at the same time, the value vector 504 is input into a pre-trained deep fully connected sequence memory module 506 to obtain a time series feature vector; then, the weighted feature vector is input into a multi-head attention mechanism module 505 to obtain a multi-head attention feature vector; finally, feature concatenation is performed on the time series feature vector and the multi-head attention feature vector to obtain an attention feature vector 507 of the voiced speech segment.

[0132] Here, linear mapping refers to the linear transformation of input features through a preset weight matrix, converting the input features into query vectors, key vectors and value vectors; the query vector represents the demand for specific information or query conditions, and the query vector is used to measure which parts of the input features need to be focused on; the key vector represents the information label or index of each part of the input feature, and the key vector is used to help the attention mechanism find relevant content by matching the query vector; the value vector represents the information actually carried in the input feature.

[0133] The similarity score is used to measure the relationship or similarity between two vectors. It is usually obtained by performing a dot product operation on the query vector and the key vector. The formula for the click operation is shown in Formula 19:

[0134] score = Q·K T Formula 19

[0135] Among them, score represents the similarity score; Q represents the query vector; K represents the key vector; T represents the matrix transpose.

[0136] The pre-trained deep fully connected sequence memory module refers to a pre-trained network module for extracting time series features. The deep fully connected sequence memory module is used to further mine the time series information of speech features and capture the laws of how speech features change over time.

[0137] The multi-head attention mechanism module is a deep learning computing module that aims to process input data in parallel through multiple attention heads, extract different feature relationships from multiple feature subspaces, and thus capture richer feature representations.

[0138] Step S104: determining a speech attribute recognition result and a speech content recognition result of the voiced speech segment based on the attention feature vector.

[0139] The sound attribute recognition results include language category labels, speech emotion labels, acoustic event labels, and digital regularization labels, etc. The specific sound attribute recognition results are not limited in this application. Among them, the language category label refers to the language type corresponding to the audio speech segment, and the language category label can include Chinese, English, Cantonese, etc.; the speech emotion label refers to the user's current emotion type, which can include happy, angry, and sad, etc.; the acoustic event label refers to what the user is doing, such as singing, crying, roaring, etc.; the digital regularization label refers to whether the Chinese characters in the speech content recognition results are converted into Arabic numerals.

[0140] In some embodiments, see Figure 6 , Figure 6 It is a flowchart of determining the speech attribute recognition result and the speech content recognition result of the voiced speech segment based on the attention feature vector provided by the embodiment of the present application; Figure 6 It is shown that in step S104, based on the attention feature vector, the speech attribute recognition result and the speech content recognition result of the voiced speech segment are determined, which can be achieved by the following steps S1041 to S1044:

[0141] Step S1041, decode the attention feature vector to obtain an attribute label probability value set and a content probability value set; each label probability value in the attribute label probability value set corresponds to a preset label in the preset label set; the label probability value is used to characterize the probability that the corresponding preset label matches the voiced speech segment.

[0142] In some embodiments, decoding refers to the process of converting complex, high-dimensional feature vectors into specific probability values, and decoding can usually be completed through a fully connected layer and a nonlinear activation function. The attribute label probability value set includes multiple groups of attribute label probability values, and each label probability value in the attribute label probability value set corresponds to a preset label in the preset label set; the content probability value set refers to a set of probability values ​​corresponding to labels representing the semantic content of speech; the label probability value is used to characterize the probability of matching the preset label in the corresponding preset label set with the voiced speech segment.

[0143] The preset label set includes multiple groups of preset labels, and the preset label set may include a language category label group, a speech emotion label group, an acoustic event label group, and a digital regularization label group.

[0144] Step S1042: determine the preset label corresponding to the maximum label probability value in the attribute label probability value set as the target label.

[0145] In some embodiments, it is assumed that the attribute label probability value set contains a total of 4 groups of attribute label probability values, corresponding to the language category label group, the speech emotion label group, the acoustic event label group and the digital regularization label group respectively; each group of attribute label probability values ​​is traversed respectively, and the preset label corresponding to the maximum label probability value in each group of attribute label probability values ​​is determined as the target label, such as: the preset label corresponding to the maximum label probability value in the language category label group is 24884; the preset label corresponding to the maximum label probability value in the speech emotion label group is 25004; the preset label corresponding to the maximum label probability value in the acoustic event label group is 24993; the preset label corresponding to the maximum label probability value in the digital regularization label group is 25017; then labels 24884, 25004, 24993 and 25017 are used as target labels.

[0146] Step S1043: determining the index of each text unit in the text content based on the content probability value in the content probability value set.

[0147] In some embodiments, the content probability value set includes multiple groups of content probability values, and each group of content probability values ​​is traversed to obtain the maximum probability value in each group of content probability values ​​as the index of each text unit. For example, for text unit 1, the first group of content probability values ​​is traversed to obtain the index 12226 corresponding to the maximum probability value, and the index 12226 is used as the index of text unit 1.

[0148] Step S1044, determining a speech attribute recognition result of the voiced speech segment based on the target tag, and determining a speech content recognition result of the voiced speech segment based on the index.

[0149] In some embodiments, the speech attribute recognition result of the voiced speech segment is determined by matching the target tag in a preset tag set, such as: target tag 24884 corresponds to "Chinese" in the language category tag group; target tag 25004 corresponds to "happy" in the language emotion tag group; target tag 24993 corresponds to "speech" in the acoustic event tag group; target tag 25017 corresponds to "no conversion" in the digital regularization tag group; at the same time, matching is performed in a preset text unit list according to the index to obtain the speech content recognition result, such as: index 12226 corresponds to the text unit "Hi".

[0150] Step S105, controlling the robot to execute a target behavior that matches the speech attribute recognition result and the speech content recognition result.

[0151] In some embodiments, when the language category attribute in the speech attribute recognition result is Chinese, other language types are discarded and the synthesized speech is output in Chinese.

[0152] When the speech emotion attribute in the speech attribute recognition result is determined, the determined emotion label is inserted into the prompt information of the pre-trained speech synthesis model, and the emotion parameters of the speech generated by the speech synthesis model are adjusted through the prompt information, such as "happy" or "sad", so that the robot performs speech synthesis through the corresponding emotion parameters.

[0153] When the acoustic event attribute in the speech attribute recognition result is determined, a corresponding response can be made directly. For example, if the acoustic event attribute is "singing", the robot will answer "You sing really well"; if the acoustic event attribute is "crying", the robot will answer "Don't cry, chat with me".

[0154] When the voice content recognition result is determined, the voice content recognition result is input into the pre-trained command recognition model to determine the command corresponding to the voice content recognition result; for example, when the voice content recognition result is an action command, such as "help me get an apple", the robot mobilizes the motor center to control the robotic arm to grab the apple; when the voice content recognition result is a walking command, such as "take me to the office", the robot mobilizes the navigation system to walk to the specified location according to the generated route.

[0155] The robot control method provided by the embodiment of the present application can be widely used in a variety of scenarios, such as: smart home, medical care and accompanying care, and human-machine collaboration. In the smart home scenario, the robot can be a sweeping robot, and the user controls the sweeping robot to perform corresponding operations through voice commands, such as "clean the kitchen". After collecting voice data, the sweeping robot recognizes the voice data, and controls the sweeping robot to walk to the kitchen and clean the kitchen according to the voice recognition results. In the medical and accompanying care scenario, the robot can be an intelligent accompanying robot, and the intelligent accompanying robot can communicate with the elderly. For example, if the elderly say "how is the weather today", the intelligent accompanying robot recognizes the voice data after collecting the voice data, obtains the weather information of the day according to the recognition results, and outputs the weather information in the form of voice. In the human-machine collaboration scenario, the robot can be an industrial robot, and the worker controls the industrial robot to perform corresponding operations through voice commands, such as "bring the wrench over here". After collecting the voice data, the industrial robot recognizes the voice data, and according to the voice recognition results, mobilizes the mechanical arm to pick up the wrench.

[0156] The robot control method provided in the embodiment of the present application can effectively filter out silent or invalid speech segments in continuous speech through speech state recognition, and only extract sounded speech segments, thereby reducing the interference of useless data; moreover, by introducing the attention mechanism, the speech feature vector can focus on the key feature part, thereby improving the expressiveness of the speech feature and improving the accuracy of speech recognition; in addition, the speech recognition results of the present application include speech attribute recognition results and speech content recognition results, realizing multi-level analysis of speech information, so that the robot can understand user needs more accurately; and, the robot behavior is controlled according to the speech attribute recognition results and the speech content recognition results, so that the robot can achieve accurate and personalized target behavior according to the speech input, thereby improving the robot's interactive ability.

[0157] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0158] An intelligent robot is a robot that integrates thinking, perception, and action in an organic and intelligent way. It not only has the ability of human-computer interaction and natural language, but can also interact with the environment and objects in real time through perception, cognition, and decision-making, assisting users in completing corresponding decision-making and action tasks.

[0159] First, the user issues a command to the robot through voice, and the voice recognition obtains the corresponding text (the above-mentioned voice content recognition result). The meaning of the text is analyzed by a large language model. The robot then decides whether it is a general question and answer, route navigation or task planning, etc., and finally executes the corresponding broadcast or action.

[0160] In order to improve the robot's interactive capabilities, speech recognition needs to meet the following capabilities:

[0161] 1) High-precision recognition; 2) Multi-language recognition, one model can automatically identify multiple languages; 3) High-speed reasoning, audio recognition reasoning takes very little time; 4) Emotion recognition, which can identify the emotions contained in speech; 5) Event detection, which supports the detection of multiple sound events such as music, applause, and laughter.

[0162] The robot control method provided in the embodiment of the present application can be divided into the following two parts:

[0163] 1. Voice segmentation:

[0164] 1) Pre-emphasize, frame and window the continuous input audio (the continuous speech collected above) to obtain a pre-processed speech frame sequence.

[0165] 2) Load the onnx model of SileroVAD (the pre-trained voice activity detection model mentioned above), and input the pre-processed voice frame sequence into the onnx model of SileroVAD. The input nodes of the onnx model include input, sr, h, and c; where sr is the audio sampling rate, the input dimension is {}, and 16000 or 8000 is supported; h is the intermediate state quantity 1, the input dimension is {2, batchSize, 64}, and the initial default is an all-zero matrix; C is the intermediate state quantity 2, the input dimension is {2, batchSize, 64}, and the initial default is an all-zero matrix; batchSize defaults to 1, representing single-channel processing, and can also be set to be greater than 1, representing multi-channel simultaneous processing.

[0166] 3) Through the onnx model, reasoning is performed, and the output nodes are output, hn, and cn; among them, output is the output result, the output dimension is {batchSize, 1}, and the speech probability value is obtained; hn is the output of the intermediate state quantity 1, the output dimension is {2, batchSize, 64}, and it is used as the h node input of the next frame; cn is the output of the intermediate state quantity 2, the output dimension is {2, batchSize, 64}, and it is used as the c node input of the next frame.

[0167] 4) Determine the voiced speech segment according to the speech probability value. For the specific determination method, see steps S201 to S206, which will not be described in detail in this application.

[0168] 2. Voice Recognition:

[0169] See also Figure 7 , Figure 7It is a speech recognition architecture diagram provided by an embodiment of the present application; feature encoding 701 is performed on the preset index data to obtain feature encoding data; feature extraction 702 is performed on the voiced speech segment to obtain a pre-processed sound feature vector; then the pre-processed sound feature vector and the encoded feature vector are feature concatenated to obtain a speech feature vector; then, the speech feature vector is input into the attention mechanism 703, and attention processing is performed to obtain the final speech attribute recognition result and speech content recognition result. The specific speech recognition process is referred to step S102 to step S104, and this application will not be repeated here.

[0170] The robot control method provided in the embodiment of the present application can improve the speech recognition process, add functions such as multi-language recognition, language switching, speech emotion recognition, and acoustic event detection, so that the robot can make more complete answers, appear more natural, real and emotional, and greatly enhance the robot's interactive ability.

[0171] Based on the robot control method described in the above embodiment, Figure 8 A structural block diagram of a robot control device 100 provided in an embodiment of the present application is shown. The robot control device may be a device in an electronic device (e.g., a robot). The robot control device may be implemented in software, which may be software in the form of programs and plug-ins, etc., including the following software modules: an identification module 101, a feature extraction module 102, an attention processing module 103, a determination module 104, and a control module 105. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented.

[0172] Among them, the recognition module 101 is used to perform speech state recognition on the speech frame sequence of the collected continuous speech to obtain the sound speech segment in the continuous speech; the feature extraction module 102 is used to extract features from the sound speech segment to obtain a speech feature vector; the attention processing module 103 is used to perform attention processing on the speech feature vector to obtain the attention feature vector of the sound speech segment; the determination module 104 is used to determine the speech attribute recognition result and the speech content recognition result of the sound speech segment based on the attention feature vector; the control module 105 is used to control the robot to perform the target behavior that matches the speech attribute recognition result and the speech content recognition result.

[0173] In some embodiments, the recognition module is also used to: perform a first preprocessing on the collected speech frame sequence of continuous speech to obtain a preprocessed speech frame sequence; call a pre-trained speech activity detection model to perform a first feature mapping on the preprocessed speech frame sequence to obtain a speech probability value for each speech frame in the preprocessed speech frame sequence; the speech probability value is used to characterize the probability of the presence of sound in the speech frame; and determine the voiced speech segment in the continuous speech based on the speech probability value of each speech frame.

[0174] In some embodiments, the recognition module is further used to: obtain a preset first speech probability threshold and a preset second speech probability threshold; wherein the second speech probability threshold is less than the first speech probability threshold; determine the speech frame whose speech probability value is greater than the first speech probability threshold as a voiced speech frame; in response to the first preceding frame of two adjacent speech frames being a silent speech frame, and the first succeeding frame of the two speech frames being the voiced speech frame, determine the first succeeding frame as a voiced speech start frame, and cache the speech frames starting from the voiced speech start frame; in the process of caching the speech frames, in response to When the second previous frame between two adjacent speech frames is the voiced speech frame, and the speech probability value of the second next frame between the two speech frames is less than the second speech probability threshold, the speech frame duration from the voiced speech start frame to the second next frame is obtained; in response to the speech frame duration being greater than a preset detection silence duration, the second previous frame is determined as a voiced speech end frame, and speech frames are stopped from being cached after the voiced speech end frame is cached; the voiced speech start frame, the voiced speech end frame, and the speech frames between the voiced speech start frame and the voiced speech end frame are determined as the voiced speech segment.

[0175] In some embodiments, the recognition module is further used to: in response to the speech frame duration being less than or equal to a preset detection silence duration, determine the second subsequent frame as the voiced speech frame, and continue to cache the second subsequent frame.

[0176] In some embodiments, the feature extraction module is also used to: preprocess the voiced speech segment to obtain preprocessed sound features; perform feature extraction on the preprocessed sound features to obtain a preprocessed sound feature vector; obtain preset index data and perform data encoding on the index data to obtain a coded feature vector; perform a first feature splicing on the preprocessed sound feature vector and the coded feature vector to obtain the speech feature vector.

[0177] In some embodiments, the second preprocessing includes a first preprocessing operation and a second preprocessing operation; the feature extraction module is further used to: perform the first preprocessing operation on the voiced speech segment to obtain preprocessing features; perform the second preprocessing operation on the preprocessing features to obtain the preprocessed sound features.

[0178] In some embodiments, the feature extraction module is also used to: perform frame processing on the voiced speech segment to obtain N analysis frames, where N is an integer greater than 1; perform normalization processing on the N analysis frames to obtain normalized features; perform fast Fourier transform on the normalized features to obtain an amplitude spectrum; perform Mel feature extraction on the amplitude spectrum to obtain a Mel spectrum feature; and determine the preprocessing feature based on the Mel spectrum feature.

[0179] In some embodiments, the feature extraction module is further used to: downsample the preprocessed features to obtain downsampled features; and normalize the downsampled features to obtain the preprocessed sound features.

[0180] In some embodiments, the determination module is also used to: decode the attention feature vector to obtain an attribute label probability value set and a content probability value set; each label probability value in the attribute label probability value set corresponds to a preset label in a preset label set; the label probability value is used to characterize the probability that the corresponding preset label matches the voiced speech segment; the preset label corresponding to the maximum label probability value in the attribute label probability value set is determined as the target label; based on the content probability value in the content probability value set, the index of each text unit in the text content is determined; based on the target label, the speech attribute recognition result of the voiced speech segment is determined, and based on the index, the speech content recognition result of the voiced speech segment is determined.

[0181] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0182] An embodiment of the present application provides an electronic device, which may be a robot. Fig. 9 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Fig. 9 As shown, the electronic device 130 includes: at least one processor 131 ( Fig. 9Only one is shown), a memory 132, and computer executable instructions 133 stored in the memory 132 and executable on at least one processor 131, the processor 131 executing the executable instructions 133 to implement the steps of any of the above-mentioned robot control method embodiments.

[0183] The electronic device may include but is not limited to a processor 131 and a memory 132. Those skilled in the art may understand that: Fig. 9 This is merely an example of the electronic device 130 and does not constitute a limitation on the electronic device 130 . The electronic device 130 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0184] The processor 131 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPG), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0185] In some embodiments, the memory 132 may be an internal storage unit of the electronic device 130, such as a hard disk or memory of the electronic device 130. In other embodiments, the memory 132 may also be an external storage device of the electronic device 130, such as a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, etc. equipped on the electronic device 130. Further, the memory 132 may also include both an internal storage unit of the electronic device 130 and an external storage device. The memory 132 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program. The memory 132 may also be used to temporarily store data that has been output or is to be output.

[0186] The present application embodiment provides a computer-readable storage medium in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the robot control method provided by the present application embodiment, for example, Figure 1The robot control method is shown.

[0187] The embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the robot control method described in the embodiment of the present application.

[0188] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0189] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0190] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0191] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0192] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A robot control method, characterized in that: The method comprises: Performing speech state recognition on the collected speech frame sequence of continuous speech to obtain the voiced speech segments in the continuous speech; Extracting features from the voiced speech segment to obtain a speech feature vector; Performing attention processing on the speech feature vector to obtain an attention feature vector of the voiced speech segment; Based on the attention feature vector, determining a speech attribute recognition result and a speech content recognition result of the voiced speech segment; The robot is controlled to execute a target behavior that matches the speech attribute recognition result and the speech content recognition result.

2. The method according to claim 1, characterized in that The step of performing speech state recognition on the collected speech frame sequence of continuous speech to obtain the voiced speech segment in the continuous speech includes: Performing a first preprocessing on the collected speech frame sequence of continuous speech to obtain a preprocessed speech frame sequence; Calling a pre-trained voice activity detection model, performing a first feature mapping on the pre-processed voice frame sequence, and obtaining a voice probability value of each voice frame in the pre-processed voice frame sequence; the voice probability value is used to characterize the probability of the presence of sound in the voice frame; Based on the speech probability value of each speech frame, a voiced speech segment in the continuous speech is determined.

3. The method according to claim 2, characterized in that The step of determining the voiced speech segment in the continuous speech based on the speech probability value of each speech frame comprises: Obtaining a preset first speech probability threshold and a preset second speech probability threshold; wherein the second speech probability threshold is less than the first speech probability threshold; Determine the speech frame whose speech probability value is greater than the first speech probability threshold as a voiced speech frame; In response to a first preceding frame of two adjacent speech frames being an unvoiced speech frame and a first succeeding frame of the two speech frames being a voiced speech frame, determining the first succeeding frame as a voiced speech start frame, and caching speech frames starting from the voiced speech start frame; In the process of caching speech frames, in response to the second previous frame of two adjacent speech frames being the voiced speech frame, and the speech probability value of the second subsequent frame of the two speech frames being less than the second speech probability threshold, obtaining the speech frame duration from the voiced speech start frame to the second subsequent frame; In response to the duration of the speech frame being greater than a preset detection silence duration, determining the second previous frame as a voiced speech termination frame, and stopping caching speech frames after caching the voiced speech termination frame; The voiced speech start frame, the voiced speech end frame, and the speech frames between the voiced speech start frame and the voiced speech end frame are determined as the voiced speech segment.

4. The method according to claim 3, characterized in that The method further comprises: In response to the voice frame duration being less than or equal to a preset detection silence duration, the second subsequent frame is determined as the voiced voice frame, and the second subsequent frame continues to be cached.

5. The method according to claim 1, characterized in that Performing feature extraction on the voiced speech segment to obtain a speech feature vector includes: Performing a second preprocessing on the voiced speech segment to obtain a preprocessed sound feature; Extracting the preprocessed sound features to obtain a preprocessed sound feature vector; Obtaining preset index data, and performing data encoding on the index data to obtain an encoded feature vector; A first feature concatenation is performed on the preprocessed sound feature vector and the encoded feature vector to obtain the speech feature vector.

6. The method according to claim 5, characterized in that The second preprocessing includes a first preprocessing operation and a second preprocessing operation; The preprocessing of the voiced speech segment to obtain preprocessed sound features includes: Performing the first preprocessing operation on the voiced speech segment to obtain a preprocessing feature; The second preprocessing operation is performed on the preprocessing feature to obtain the preprocessing sound feature.

7. The method according to claim 6, characterized in that The performing the first preprocessing operation on the voiced speech segment to obtain the preprocessing feature includes: Performing frame processing on the voiced speech segment to obtain N analysis frames, where N is an integer greater than 1; Performing normalization processing on the N analysis frames to obtain normalized features; Performing fast Fourier transform on the normalized features to obtain an amplitude spectrum; Performing Mel feature extraction on the amplitude spectrum to obtain Mel spectrum features; The preprocessing feature is determined based on the mel spectrum feature.

8. The method according to claim 7, characterized in that The performing the second preprocessing operation on the preprocessing feature to obtain the preprocessing sound feature includes: Downsampling the preprocessed features to obtain downsampled features; The down-sampled features are normalized to obtain the pre-processed sound features.

9. The method according to claim 1, characterized in that: The step of determining the speech attribute recognition result and the speech content recognition result of the voiced speech segment based on the attention feature vector includes: Decoding the attention feature vector to obtain an attribute label probability value set and a content probability value set; each label probability value in the attribute label probability value set corresponds to a preset label in a preset label set; the label probability value is used to represent the probability of the corresponding preset label matching the voiced speech segment; Determine the preset label corresponding to the maximum label probability value in the attribute label probability value set as the target label; Determining the index of each text unit in the text content based on the content probability values ​​in the content probability value set; A speech attribute recognition result of the voiced speech segment is determined based on the target tag, and a speech content recognition result of the voiced speech segment is determined based on the index.

10. A robot control device, characterized in that: include: A recognition module, used for performing speech state recognition on the speech frame sequence of the collected continuous speech to obtain the voiced speech segments in the continuous speech; A feature extraction module, used to extract features from the voiced speech segment to obtain a speech feature vector; An attention processing module, used for performing attention processing on the speech feature vector to obtain the attention feature vector of the voiced speech segment; A determination module, used to determine a speech attribute recognition result and a speech content recognition result of the voiced speech segment based on the attention feature vector; The control module is used to control the robot to execute a target behavior that matches the speech attribute recognition result and the speech content recognition result.

11. An electronic device, characterized in that: include: A memory for storing computer executable instructions; A processor, configured to implement the robot control method according to any one of claims 1 to 9 when executing the computer executable instructions stored in the memory.

12. A computer-readable storage medium, characterized in that: Computer executable instructions or computer programs are stored, and when the computer executable instructions or computer programs are executed by a processor, the robot control method according to any one of claims 1 to 9 is implemented.

Images