Teaching method, device and equipment for service robot training and storage medium

Abstract

The invention discloses a teaching method, device and equipment for service robot training and a storage medium, and relates to the technical field of service robots, the teaching method for service robot training is applied to a service robot system, and the method comprises the steps: collecting image data and voice data of a to-be-taught user; visual feature information is extracted from the image data, a visual feature vector is constructed based on the visual feature information, and the visual feature vector represents the facial emotion state of the to-be-taught user; voice feature information is extracted from the voice data, voice feature vectors are constructed based on the voice feature information, and the voice feature vectors represent voice personalized features and voice emotional states of the to-be-taught user; and predicting an emotional state type of the to-be-taught user based on the visual feature vector and the voice feature vector so as to execute a preset teaching strategy corresponding to the emotional state type. The teaching effect of the service robot can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of service robots, and in particular to a teaching method, device, equipment and storage medium for training service robots. Background Art

[0002] With the continuous development of artificial intelligence technology, the traditional teaching model is undergoing unprecedented changes. Among them, service robots used for teaching and training are the forefront of this change and are gradually becoming an important tool for auxiliary teaching.

[0003] In educational practice, students' emotional state is one of the key factors affecting learning outcomes. For example, students' positive emotional state can promote learning efficiency and learning outcomes to a certain extent. On the contrary, if students feel bored, frustrated or anxious during the learning process, their learning outcomes will also be negatively affected. However, most of the current service robots rely on preset programs and algorithms for interaction. These programs focus on the transfer of knowledge and the training of skills, ignoring the recognition and response to students' emotional states in the educational process. They are unable to adjust teaching strategies in time according to changes in students' emotional states to stimulate students' enthusiasm, resulting in problems that affect teaching outcomes.

[0004] Therefore, how to improve the teaching effect of service robots is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide a teaching method, device, equipment and storage medium for service robot training, aiming to improve the teaching effect of service robots.

[0006] To achieve the above objectives, the present application provides a teaching method for training a service robot, the teaching method for training a service robot is applied to a service robot system, and the teaching method for training a service robot includes:

[0007] Collect image data and voice data of the user to be taught;

[0008] Extracting visual feature information from the image data, and constructing a visual feature vector based on the visual feature information, wherein the visual feature vector represents the facial emotional state of the user to be taught;

[0009] Extracting speech feature information from the speech data, and constructing a speech feature vector based on the speech feature information, wherein the speech feature vector represents the speech individuality feature and speech emotional state of the user to be taught;

[0010] The emotional state type of the user to be taught is predicted based on the visual feature vector and the voice feature vector, so as to execute a preset teaching strategy corresponding to the emotional state type.

[0011] In one embodiment, the visual feature information includes a low-dimensional feature map and a high-dimensional feature map, and the step of extracting the visual feature information from the image data and constructing a visual feature vector based on the visual feature information includes:

[0012] Extracting first shallow features from the image data through a low-dimensional feature generation pyramid of a deep convolutional neural network, and generating the low-dimensional feature map based on the first shallow features, wherein the first shallow features represent local detail information in the image data;

[0013] Extracting second shallow features and deep features from the image data through the high-dimensional feature generation pyramid of the deep convolutional neural network, and combining the second shallow features and the deep features to generate the high-dimensional feature map, wherein the second shallow features represent local detail information in the image data, and the deep features represent global semantic information in the image data;

[0014] The low-dimensional feature map and the high-dimensional feature map are fused to obtain a visual comprehensive feature map, and the visual features in the visual comprehensive feature map are encoded into a visual feature vector.

[0015] In one embodiment, the voice feature information includes a first voice feature and a second voice feature, wherein the first voice feature is a spectral feature, an acoustic feature, and a waveform feature of the voice data, and the second voice feature is a biometric feature of the voice of the user to be taught.

[0016] In one embodiment, the speech feature vector includes an emotional feature vector and a personalized feature vector, and the step of extracting speech feature information from the speech data and constructing the speech feature vector based on the speech feature information includes:

[0017] Extracting the first speech feature from the speech data, and constructing the emotion feature vector based on the first speech feature, wherein the emotion feature vector represents the speech emotion state of the user to be taught;

[0018] The second speech feature is extracted from the speech data, and the personalized feature vector is constructed based on the second speech feature, wherein the personalized feature vector represents the personalized speech feature of the user to be taught.

[0019] In one embodiment, the step of extracting the first speech feature from the speech data and constructing the emotion feature vector based on the first speech feature includes:

[0020] Extracting features from the spectrogram of the speech data by a speech spectrum feature extractor to obtain a first convolution feature map, wherein the first convolution feature map represents the spectral features and the acoustic features;

[0021] Extracting features from the waveform data of the speech data by a speech waveform feature extractor to obtain a second convolution feature map, wherein the second convolution feature map represents the waveform features;

[0022] The emotion feature vector is constructed based on the first convolution feature map and the second convolution feature map.

[0023] In one embodiment, the step of constructing the emotion feature vector based on the first convolution feature map and the second convolution feature map includes:

[0024] Performing a fusion process on the first convolution feature map and the second convolution feature map to obtain a comprehensive speech feature map;

[0025] The speech features in the speech comprehensive feature graph are weighted and summed to obtain comprehensive speech features, and the comprehensive speech features are encoded into the emotion feature vector.

[0026] In one embodiment, the step of predicting the emotional state type of the user to be taught based on the visual feature vector and the voice feature vector to execute a preset teaching strategy corresponding to the emotional state type includes:

[0027] Using a multi-head attention mechanism to perform feature fusion on the visual feature vector and the speech feature vector to obtain a multimodal feature;

[0028] The multimodal features are input into a preset multimodal classifier to obtain the emotional state type of the user to be taught, so as to execute a preset teaching strategy corresponding to the emotional state type.

[0029] In addition, to achieve the above-mentioned purpose, the present application also provides a teaching device for training a service robot, wherein the teaching device for training a service robot is applied to a service robot system, and the teaching device for training a service robot comprises:

[0030] An acquisition module, used to acquire image data and voice data of the user to be taught;

[0031] A visual feature extraction module, used to extract visual feature information from the image data, and construct a visual feature vector based on the visual feature information, wherein the visual feature vector represents the facial emotional state of the user to be taught;

[0032] A speech feature extraction module, used to extract speech feature information from the speech data, and construct a speech feature vector based on the speech feature information, wherein the speech feature vector represents the speech individuality feature and speech emotional state of the user to be taught;

[0033] A prediction module is used to predict the emotional state type of the user to be taught based on the visual feature vector and the voice feature vector, so as to execute a preset teaching strategy corresponding to the emotional state type.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and the computer-readable storage medium stores a program for implementing a teaching method for service robot training. The program for implementing the teaching method for service robot training is executed by a processor to implement the steps of the teaching method for service robot training as described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the teaching method for service robot training as described above.

[0036] The present application provides a teaching method for service robot training. The present application is applied to a service robot system. The present application first collects image data and voice data of a user to be taught, extracts visual feature information from the image data, and constructs a visual feature vector representing the facial emotional state of the user to be taught based on the visual feature information. At the same time, voice feature information is extracted from the voice data, and a voice feature vector representing the personalized voice features and voice emotional state of the user to be taught is constructed based on the voice feature information. Finally, the emotional state type of the user to be taught is predicted based on the visual feature vector and the voice feature vector, so that the service robot can execute a preset teaching strategy corresponding to the current emotional state type.

[0037] In summary, compared with the traditional service robot teaching method that completely relies on preset programs and algorithms to perform teaching interactions, the service robot proposed in this application can extract features from the collected student image data and voice data, and construct a visual feature vector that represents the student's facial emotional state based on the extracted visual feature information, and construct a voice feature vector that represents the student's voice personalization characteristics and voice emotional state based on the extracted voice feature information. Finally, based on the visual feature vector and the voice feature vector, the student's emotional state type is predicted, and the teaching strategy is adjusted in time based on the student's current emotional state, thereby stimulating students' learning enthusiasm and improving the teaching effect of the service robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 This is a flow chart of the first embodiment of the teaching method for training a service robot according to the present application;

[0041] Figure 2 A schematic diagram of an emotional state recognition architecture involved in an embodiment of a teaching method for training a service robot of the present application;

[0042] Figure 3 This is a schematic diagram of the facial feature extraction module architecture involved in one embodiment of the teaching method for training a service robot of the present application;

[0043] Figure 4 This is a schematic diagram of the architecture of a speech feature extraction module involved in an embodiment of the teaching method for training a service robot of the present application;

[0044] Figure 5 This is a schematic diagram of the module structure of the teaching device for service robot training in this application;

[0045] Figure 6 This is a schematic diagram of the equipment structure of the hardware operating environment involved in the teaching method for service robot training in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solution of the present application is: collecting image data and voice data of the user to be taught; extracting visual feature information from the image data, and constructing a visual feature vector based on the visual feature information, wherein the visual feature vector represents the facial emotional state of the user to be taught; extracting voice feature information from the voice data, and constructing a voice feature vector based on the voice feature information, wherein the voice feature vector represents the personalized voice characteristics and voice emotional state of the user to be taught; predicting the emotional state type of the user to be taught based on the visual feature vector and the voice feature vector, so as to execute a preset teaching strategy corresponding to the emotional state type.

[0050] In educational practice, students' emotional state is one of the key factors affecting learning outcomes. For example, students' positive emotional state can promote learning efficiency and learning outcomes to a certain extent. On the contrary, if students feel bored, frustrated or anxious during the learning process, their learning outcomes will also be negatively affected. However, most of the current service robots rely on preset programs and algorithms for interaction. These programs focus on the transfer of knowledge and the training of skills, ignoring the recognition and response to students' emotional states in the educational process. They are unable to adjust teaching strategies in time according to changes in students' emotional states to stimulate students' enthusiasm, resulting in problems that affect teaching outcomes.

[0051] The service robot proposed in the present application can extract features from the collected image data and voice data of the students, and construct a visual feature vector representing the facial emotional state of the students based on the extracted visual feature information, and construct a voice feature vector representing the personalized characteristics of the students' voice and the voice emotional state based on the extracted voice feature information. Finally, the service robot can predict the type of the students' emotional state based on the visual feature vector and the voice feature vector, and timely adjust the teaching strategy based on the students' current emotional state, thereby stimulating the students' learning enthusiasm and improving the teaching effect of the service robot.

[0052] It should be noted that the execution subject of the method in each embodiment of the teaching method for training service robots of the present application may be a service robot system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a service robot capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the service robot system as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, this application proposes a teaching method for service robot training in the first embodiment, please refer to Figure 1 The service robot training teaching method includes steps S10 to S40:

[0054] Step S10, collecting image data and voice data of the user to be taught;

[0055] It should be noted that the service robot system of the present application includes a camera module and a microphone module. During the teaching process of the service robot, the image data of the student (i.e., the user to be taught) is collected through the camera module, and the voice data of the student is collected through the microphone module. It should be understood that the camera module can recognize the student's head, and after ensuring that the student's head is within the shooting range, the shooting operation is performed to obtain image data containing the student's head information. The service robot system can guide the student to answer questions by voice by executing a preset teaching interactive operation, thereby obtaining the student's voice signal through the microphone module and obtaining voice data.

[0056] Step S20, extracting visual feature information from the image data, and constructing a visual feature vector based on the visual feature information, wherein the visual feature vector represents the facial emotional state of the user to be taught;

[0057] It should be noted that the service robot system of the present application includes a facial feature extraction module, which is used to extract visual feature information from image data and also to encode the visual feature information into a low-dimensional feature vector, namely the above-mentioned visual feature vector.

[0058] The facial feature extraction module extracts visual feature information from the image data, and then constructs a visual feature vector based on the visual feature information, wherein the visual feature vector represents the emotional state of the face of the user to be taught, that is, the facial emotional state.

[0059] For example, the student's facial emotional state represented by the visual feature vector includes but is not limited to facial expression, eye gaze direction (e.g., whether looking at the teaching screen), and head posture, etc. The role of these low-dimensional feature vectors is to convert complex visual information into numerical representations that are convenient for subsequent model processing, thereby judging the student's emotional state.

[0060] Step S30, extracting speech feature information from the speech data, and constructing a speech feature vector based on the speech feature information, wherein the speech feature vector represents the speech individuality feature and speech emotional state of the user to be taught;

[0061] It should be noted that the service robot system of the present application also includes a speech feature extraction module, which is used to extract speech feature information from speech data and construct a speech feature vector based on the speech feature information to improve the accuracy of emotion state recognition.

[0062] The speech feature extraction module extracts speech feature information from the speech data, and constructs a speech feature vector based on the speech feature information. The speech feature vector represents the speech personalization characteristics and speech emotional state (i.e., the emotional state of the user's speech) of the user to be taught. The speech personalization characteristics can be understood as the different personality, timbre, intonation and other proprietary characteristics of different speakers in the conversation.

[0063] Step S40, predicting the emotional state type of the user to be taught based on the visual feature vector and the voice feature vector, so as to execute a preset teaching strategy corresponding to the emotional state type.

[0064] It should be noted that a coding-decoding network or classifier is pre-set to predict the type of students' emotional state based on the visual feature vector and the speech feature vector, and a teaching strategy corresponding to each type of emotional state is pre-set, wherein the emotional state types include but are not limited to positive, general and negative.

[0065] In this embodiment, the step S40 may include:

[0066] Step S401, using a multi-head attention mechanism to perform feature fusion on the visual feature vector and the speech feature vector to obtain a multimodal feature;

[0067] Step S402: input the multimodal features into a preset multimodal classifier to obtain the emotional state type of the user to be taught, so as to execute a preset teaching strategy corresponding to the emotional state type.

[0068] It should be noted that in order to fully capture the contextual information within the modality and the complementary information between modalities, the embodiment of the present application proposes a multi-source information fusion technology. Specifically, MDAN (Multi-Head Attention Network) is used to model the contextual dependency relationship between audio and visual modalities, and then MCAN (Multi-Head Cross-Modal Attention Network) is used to perform cross-modal modeling to extract complementary information between modalities.

[0069] The visual feature vector and the speech feature vector are fused by using a multi-head attention mechanism to obtain fused features (hereinafter referred to as multi-modal features for distinction). Specifically, the contextual information in the modality is captured by the multi-head attention network, that is, the dependency relationship between the visual information in the image modality and the speech information in the speech modality. Through the multi-head attention mechanism, the network can find the most relevant information for emotion recognition in the multi-modal features, and then the complementary information between the modalities is modeled by the multi-head cross-modal attention network. In other words, the visual features and speech features are fused to extract the complementary information between the two. Then the multi-modal features are input into a preset multi-modal classifier to obtain the emotional state type of the user to be taught, so that the service robot can execute the preset teaching strategy corresponding to the current emotional state type. Specifically, the fused multi-modal features are input into an encoding-decoding network or classifier to obtain the emotional state type of the student, that is, the preset multi-modal classifier is an encoding-decoding network or classifier.

[0070] For example, Figure 2 Shown is a schematic diagram of the emotion state recognition architecture. The service robot system includes a facial feature extraction module, a speech feature extraction module and an emotion state classification module. The facial feature extraction module is used to extract visual feature information from the student's image data, and construct a visual feature vector representing the student's facial emotion state based on the visual feature information; the speech feature extraction module is used to extract speech feature information from the student's speech data, and construct a speech feature vector representing the student's speech emotion state based on the speech feature information; the emotion state classification module is used to predict the student's emotion state category based on the feature vectors of the two modes.

[0071] The embodiment of the present application can extract features from the collected image data and voice data of the students, and construct a visual feature vector representing the student's facial emotional state based on the extracted visual feature information, and construct a voice feature vector representing the student's voice personalization characteristics and voice emotional state based on the extracted voice feature information. Finally, the student's emotional state type is predicted based on the visual feature vector and the voice feature vector, and the teaching strategy is adjusted in time based on the student's current emotional state, thereby stimulating the student's learning enthusiasm and further improving the teaching effect of the service robot.

[0072] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, the visual feature information includes a low-dimensional feature map and a high-dimensional feature map, and the step S20 may include:

[0073] Step S201, extracting first shallow features from the image data through a low-dimensional feature generation pyramid of a deep convolutional neural network, and generating the low-dimensional feature map based on the first shallow features, wherein the first shallow features represent local detail information in the image data;

[0074] It should be noted that the facial feature extraction module is a deep convolutional neural network model, including a low-dimensional feature generation pyramid, which is used to generate a low-dimensional feature map. The low-dimensional feature generation pyramid generates a low-dimensional feature map based on image data, which represents the local and basic visual features in the image data. It is extracted from the early convolutional layer of the deep network and usually includes low-level information such as edges, textures, corners, and brightness changes of the image data. This information is the basis for constructing higher-level semantic features.

[0075] The image data is input into a deep convolutional neural network model, and then shallow features (hereinafter referred to as first shallow features for distinction) are extracted from the image data through a low-dimensional feature generation pyramid of the deep convolutional neural network model, and then a low-dimensional feature map is generated based on the first shallow features, wherein the first shallow features represent local detail information in the image data.

[0076] Exemplarily, the first shallow features in the low-dimensional feature map include but are not limited to the edges and contours of objects or face areas in the image, texture information on the image surface (such as the texture of the facial skin), and brightness changes or geometric shapes in small areas (such as the contours of eyes, mouth, etc.).

[0077] Step S202, extracting second shallow features and deep features from the image data through a high-dimensional feature generation pyramid, and combining the second shallow features and the deep features to generate the high-dimensional feature map, wherein the second shallow features represent local detail information in the image data, and the deep features represent global semantic information in the image data;

[0078] It should be noted that the facial feature extraction module also includes a high-dimensional feature generation pyramid for generating a high-dimensional feature map that combines shallow features and deep features to obtain a more comprehensive representation of the image data. Exemplarily, in the facial feature extraction module, high-dimensional feature maps of different levels are generated by a ResNet50 deep network, including low-level and high-level details. That is, the deep convolutional neural network model in the embodiment of the present application can be a ResNet50 deep network model.

[0079] After the image data is input into the deep convolutional neural network model, shallow features (hereinafter referred to as the second shallow features for distinction) and deep features are extracted from the image data through the high-dimensional feature generation pyramid of the deep neural network model, and then the second shallow features and the deep features are combined to generate a high-dimensional feature map. It should be noted that the second shallow features represent the local detail information in the image data, and the deep features represent the global semantic information in the image data.

[0080] In a feasible implementation, the image data is input into the facial feature extraction module, and the first shallow features are extracted from the image data through the low-dimensional feature generation pyramid, and a low-dimensional feature map is generated; at the same time, the second shallow features are extracted from the first few layers of the network (the layers close to the input end) through the high-dimensional feature generation pyramid, including basic geometric shapes such as edges, corners and textures, and the second shallow features are used to capture local details in the image data, similar to the basic filters in image processing. For example, in an image data, the shallow features include but are not limited to basic visual information such as the edges of the eyes, the lines of the corners of the mouth and the facial contour; and the deep features are extracted from the last few layers of the network (the layers close to the output end) through the high-dimensional feature generation pyramid, such as the overall facial expression, head posture and gaze direction, that is, the deep features represent global and high-level semantic information. In other words, the high-dimensional feature map output by the high-dimensional feature generation pyramid can describe the local details of the image and also represent the global semantic information of the image.

[0081] Step S203, fusing the low-dimensional feature map and the high-dimensional feature map to obtain a visual comprehensive feature map, and encoding the visual features in the visual comprehensive feature map into a visual feature vector.

[0082] It should be noted that the deep convolutional neural network also includes a convolutional layer, which is used to fuse low-dimensional feature maps with high-dimensional feature maps, extract effective information therein, and establish more complex relationships between feature maps.

[0083] The low-dimensional feature map and the high-dimensional feature map are fused through the convolution layer to obtain a fused feature map (hereinafter referred to as a visual comprehensive feature map for distinction), and the visual features in the visual comprehensive feature map are encoded into a visual feature vector, wherein the visual feature vector is the final output of the facial feature extraction module. And, it should be understood that the visual features in the visual comprehensive feature map include the low-dimensional features in the low-dimensional feature map (i.e., the first shallow features) and the high-dimensional features in the high-dimensional feature map (i.e., the second shallow features and deep features).

[0084] For example, Figure 3The figure shows a schematic diagram of the facial feature extraction module architecture, which includes a low-dimensional feature generation pyramid, a high-dimensional feature generation pyramid and a convolution layer, wherein the low-dimensional feature generation pyramid is used to generate a low-dimensional feature map, the high-dimensional feature generation pyramid is used to generate a high-dimensional feature map, and the convolution layer is used to perform feature fusion processing on the low-dimensional feature map and the high-dimensional feature map to obtain a visual comprehensive feature map, and encode the visual features in the visual comprehensive feature map into a visual feature vector and output it.

[0085] In this way, the embodiment of the present application performs visual feature extraction on the collected image data through the facial feature extraction module, obtains low-dimensional features and high-dimensional features in the image data, and constructs feature vectors based on the extracted low-dimensional features and high-dimensional features, thereby facilitating the visual information that characterizes the user's emotional state to participate in the subsequent determination of the emotional state, and realizes the support of visual information for the recognition of the user's emotional state, thereby improving the accuracy of the recognition of the user's emotional state.

[0086] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, the step S30 may include:

[0087] The voice feature information includes a first voice feature and a second voice feature, the first voice feature is a spectrum feature, an acoustic feature and a waveform feature of the voice data, and the second voice feature is a biometric feature of the voice of the user to be taught.

[0088] It should be noted that the voice feature information extracted from the voice data includes a first voice feature and a second voice feature, wherein the first voice feature is the spectral feature, acoustic feature and waveform feature of the voice data, and the second voice feature is the biometric feature of the voice of the user to be taught. Specifically, the spectral feature includes but is not limited to the frequency components of the voice data and the pattern of frequency changes over time. For example, the frequency of angry voice is higher, while the frequency of sad voice is lower and the rhythm is slower; the acoustic feature includes but is not limited to the pattern of pitch and volume changes over time, such as pauses in the voice and the ups and downs of the tone, which can reflect the speaker's emotions; the waveform feature includes but is not limited to the amplitude, peak value and energy distribution in the waveform, etc., which can reflect the strength of the speech and the intensity of the emotion. The biometric features of the user's voice include but are not limited to the speaker's timbre, accent, pitch and personalized speaking style.

[0089] In this embodiment, the speech feature vector includes an emotional feature vector and a personalized feature vector, and step S30 may include:

[0090] Step S301, extracting the first voice feature from the voice data, and constructing the emotion feature vector based on the first voice feature, wherein the emotion feature vector represents the voice emotion state of the user to be taught;

[0091] Step S302: extracting the second speech feature from the speech data, and constructing the personalized feature vector based on the second speech feature, wherein the personalized feature vector represents the personalized speech feature of the user to be taught.

[0092] It should be noted that the speech feature vector constructed based on speech feature information includes an emotional feature vector and a personalized feature vector, wherein the emotional feature vector represents the speech emotional state of the user to be taught, and the personalized feature vector represents the personalized speech features of the user to be taught.

[0093] A first voice feature is extracted from the voice data, and an emotional feature vector is constructed based on the first voice feature; at the same time, a second voice feature is extracted from the voice data, and a personalized feature vector is constructed based on the second voice feature. It should be understood that the personalized feature vector represents the personalized feature of the speaker and is a low-dimensional feature vector.

[0094] In this embodiment, the step S301 may include:

[0095] Step A10, extracting features from the spectrogram of the speech data by a speech spectrum feature extractor to obtain a first convolution feature map, wherein the first convolution feature map represents the spectral features and the acoustic features;

[0096] It should be noted that the speech feature extraction module includes a speech spectral feature extractor, which is used to extract features from the speech time-domain converted spectrogram through a ResNet neural network, capturing spectrum and acoustic information, that is, capturing spectrum features and acoustic features.

[0097] The speech spectrum feature extractor is used to extract features from the spectrum graph of the speech data, thereby obtaining a convolution feature graph (hereinafter referred to as the first convolution feature graph for distinction) that characterizes the spectral features and acoustic features of the speech data.

[0098] In one feasible implementation, the speech spectral feature extractor converts speech data into a spectrogram in the time domain, and then processes the spectrogram through the ResNet neural network built into the speech spectral feature extractor to extract the spectral features and acoustic features of the speech, and outputs a convolutional feature map that reflects the patterns of different frequency bands and time changes in the spectrogram. It should be noted that the output convolutional feature map is usually a feature map of multiple channels, each channel representing a different spectral feature.

[0099] Step A20, extracting features from the waveform data of the speech data by a speech waveform feature extractor to obtain a second convolution feature map, wherein the second convolution feature map represents the waveform features;

[0100] It should be noted that the speech feature extraction module also includes a speech waveform feature extractor, which is used to process the waveform data of the speech data through multi-layer convolution to capture the waveform features.

[0101] The waveform data of the speech data is subjected to feature extraction by a speech spectrum feature extractor to obtain a convolution feature map (hereinafter referred to as the second convolution feature map for distinction) representing the waveform features of the speech data.

[0102] In one feasible implementation, the speech waveform feature extractor directly performs convolution processing on the original waveform data of the speech data to extract waveform features such as amplitude changes and waveform energy in the waveform. It can gradually capture important waveform features of different sizes and durations through multi-layer convolution. The output result of the speech waveform feature extractor is also a feature map of multiple channels, representing the patterns of different time and amplitude changes in the waveform.

[0103] Step A30, constructing the emotion feature vector based on the first convolution feature map and the second convolution feature map.

[0104] An emotion feature vector is constructed based on the speech features in the first convolution feature map and the second convolution feature map.

[0105] In a specific implementation, the step A30 may include:

[0106] Step A301, fusing the first convolution feature map and the second convolution feature map to obtain a comprehensive speech feature map;

[0107] The first convolution feature map and the second convolution feature map are fused to obtain a fused feature map (hereinafter referred to as a speech comprehensive feature map for distinction).

[0108] In this embodiment, the spectral features and acoustic features in the first convolutional feature map and the waveform features in the second convolutional feature map are cross-fused to achieve the combination of different modal information at the feature map level. The information combination is achieved by including but not limited to feature concatenation, element-by-element addition, etc. Only by mixing the features in the two feature maps, the model can learn richer information from multiple signal sources.

[0109] Step A302, performing weighted sum processing on the speech features in the speech comprehensive feature graph to obtain comprehensive speech features, and encoding the comprehensive speech features into the emotion feature vector.

[0110] It should be noted that the speech features in the comprehensive speech feature map include the spectrum features and acoustic features in the first convolution feature map, and the waveform features in the second convolution feature map.

[0111] The speech features in the speech comprehensive feature graph are weighted and summed to obtain the added speech features (hereinafter referred to as comprehensive speech features for distinction), and then the comprehensive speech features are encoded into an emotion feature vector, which is the final output result of the speech feature extraction module.

[0112] Weighted summation means that after fusing the convolutional feature maps of the two modalities, different weights are assigned to the features of each modality and they are combined into the final output. This process can be seen as further weighting the results of the cross-fusion, so that the model can focus on the features that are more important for emotion recognition.

[0113] For example, Figure 4 This is a schematic diagram of the speech feature extraction module architecture. The input data of the speech feature extraction module is speech data. The speech spectrogram and speech waveform data can be determined based on the speech data. Then, the speech spectrum feature extractor extracts spectral features and acoustic features from the spectrogram and generates a first convolution feature map. The speech waveform feature extractor extracts waveform features from the waveform data and generates a second convolution feature map. Specifically, after multiple layers of convolution, pooling, normalization and flattening operations, important features in the speech are gradually extracted and converted into low-dimensional vector representations that are easy to classify. The final output is the extracted low-dimensional feature vector related to speech emotion (i.e., the above-mentioned speech feature vector), which can be used for further processing of the emotion recognition model.

[0114] In this way, the embodiment of the present application extracts spectral features, acoustic features, waveform features and biometric features from the student's speech data through a speech feature extraction module, and constructs an emotional feature vector characterizing the emotional state of the user's speech and a personalized feature vector characterizing the personalized features of the user's speech based on the extracted speech features, so that the speech information characterizing the user's emotional state can participate in the subsequent determination of the emotional state, thereby realizing the support of speech information for the recognition of the user's emotional state, thereby improving the accuracy of the recognition of the user's emotional state.

[0115] The present application also provides a teaching device for training a service robot. Figure 5 The service robot training teaching device is applied to a service robot system, and the service robot training teaching device comprises:

[0116] The acquisition module 10 is used to acquire image data and voice data of the user to be taught;

[0117] A visual feature extraction module 20, for extracting visual feature information from the image data, and constructing a visual feature vector based on the visual feature information, wherein the visual feature vector represents the facial emotional state of the user to be taught;

[0118] A speech feature extraction module 30, for extracting speech feature information from the speech data, and constructing a speech feature vector based on the speech feature information, wherein the speech feature vector represents the speech individuality feature and speech emotional state of the user to be taught;

[0119] The prediction module 40 is used to predict the emotional state type of the user to be taught based on the visual feature vector and the voice feature vector, so as to execute a preset teaching strategy corresponding to the emotional state type.

[0120] Optionally, the visual feature information includes a low-dimensional feature map and a high-dimensional feature map, and the visual feature extraction module 20 is further used for:

[0121] Extracting first shallow features from the image data through a low-dimensional feature generation pyramid of a deep convolutional neural network, and generating the low-dimensional feature map based on the first shallow features, wherein the first shallow features represent local detail information in the image data;

[0122] Extracting second shallow features and deep features from the image data through the high-dimensional feature generation pyramid of the deep convolutional neural network, and combining the second shallow features and the deep features to generate the high-dimensional feature map, wherein the second shallow features represent local detail information in the image data, and the deep features represent global semantic information in the image data;

[0123] The low-dimensional feature map and the high-dimensional feature map are fused to obtain a visual comprehensive feature map, and the visual features in the visual comprehensive feature map are encoded into a visual feature vector.

[0124] Optionally, the voice feature information includes a first voice feature and a second voice feature, the first voice feature is a spectral feature, an acoustic feature and a waveform feature of the voice data, and the second voice feature is a biometric feature of the voice of the user to be taught.

[0125] Optionally, the speech feature vector includes an emotional feature vector and a personalized feature vector, and the speech feature extraction module 30 is further used for:

[0126] Extracting the first speech feature from the speech data, and constructing the emotion feature vector based on the first speech feature, wherein the emotion feature vector represents the speech emotion state of the user to be taught;

[0127] The second speech feature is extracted from the speech data, and the personalized feature vector is constructed based on the second speech feature, wherein the personalized feature vector represents the personalized speech feature of the user to be taught.

[0128] Optionally, the speech feature extraction module 30 is further used for:

[0129] Extracting features from the spectrogram of the speech data by a speech spectrum feature extractor to obtain a first convolution feature map, wherein the first convolution feature map represents the spectral features and the acoustic features;

[0130] Extracting features from the waveform data of the speech data by a speech waveform feature extractor to obtain a second convolution feature map, wherein the second convolution feature map represents the waveform features;

[0131] The emotion feature vector is constructed based on the first convolution feature map and the second convolution feature map.

[0132] Optionally, the speech feature extraction module 30 is further used for:

[0133] Performing a fusion process on the first convolution feature map and the second convolution feature map to obtain a comprehensive speech feature map;

[0134] The speech features in the speech comprehensive feature graph are weighted and summed to obtain comprehensive speech features, and the comprehensive speech features are encoded into the emotion feature vector.

[0135] Optionally, the prediction module 40 is further configured to:

[0136] Using a multi-head attention mechanism to perform feature fusion on the visual feature vector and the speech feature vector to obtain a multimodal feature;

[0137] The multimodal features are input into a preset multimodal classifier to obtain the emotional state type of the user to be taught, so as to execute a preset teaching strategy corresponding to the emotional state type.

[0138] The teaching device for training a service robot provided in the embodiment of the present application adopts the teaching method for training a service robot in the above embodiment, and can solve the technical problem of how to improve the teaching effect of a service robot. Compared with the prior art, the beneficial effects of the teaching device for training a service robot provided in the embodiment of the present application are the same as the beneficial effects of the teaching method for training a service robot provided in the above embodiment, and the other technical features of the teaching device for training a service robot are the same as the features disclosed in the above embodiment method, and will not be repeated here.

[0139] The present application provides a service robot, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the teaching method for service robot training in the above-mentioned embodiment one.

[0140] Reference below Figure 6 , which shows a schematic structural diagram of a service robot suitable for implementing an embodiment of the present application. Figure 6 The service robot shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Figure 6 As shown, the service robot may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read On ly Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the service robot are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the service robot to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a service robot with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0142] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0143] The service robot provided by the present application adopts the teaching method for training the service robot in the above embodiment, which can solve the technical problem of how to improve the teaching effect of the service robot. Compared with the prior art, the beneficial effects of the service robot provided by the present application are the same as the beneficial effects of the teaching method for training the service robot provided by the above embodiment, and the other technical features of the service robot are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0144] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0146] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the teaching method for training a service robot in the above-mentioned embodiment.

[0147] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM: Erasable Programmab le Read On ly Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read On ly Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0148] The computer-readable storage medium may be included in the service robot; or it may exist independently without being assembled into the service robot.

[0149] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the service robot, the service robot: collects image data and voice data of the user to be taught; extracts visual feature information from the image data, and constructs a visual feature vector based on the visual feature information, wherein the visual feature vector represents the facial emotional state of the user to be taught; extracts voice feature information from the voice data, and constructs a voice feature vector based on the voice feature information, wherein the voice feature vector represents the personalized voice characteristics and voice emotional state of the user to be taught; predicts the emotional state type of the user to be taught based on the visual feature vector and the voice feature vector, so as to execute a preset teaching strategy corresponding to the emotional state type.

[0150] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0151] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0153] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned teaching method for training a service robot, and can solve the technical problem of how to improve the teaching effect of a service robot. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the teaching method for training a service robot provided by the above-mentioned embodiment, and will not be elaborated here.

[0154] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the teaching method for training a service robot as described above.

[0155] The computer program product provided in this application can improve the teaching effect of the service robot. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the teaching method for service robot training provided in the above embodiment, which will not be repeated here.

[0156] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A teaching method for training a service robot, characterized in that: The service robot training teaching method is applied to a service robot system, and the service robot training teaching method includes: Collect image data and voice data of the user to be taught; Extracting visual feature information from the image data, and constructing a visual feature vector based on the visual feature information, wherein the visual feature vector represents the facial emotional state of the user to be taught; Extracting speech feature information from the speech data, and constructing a speech feature vector based on the speech feature information, wherein the speech feature vector represents the speech individuality feature and speech emotional state of the user to be taught; The emotional state type of the user to be taught is predicted based on the visual feature vector and the voice feature vector, so as to execute a preset teaching strategy corresponding to the emotional state type.

2. The method according to claim 1, characterized in that The visual feature information includes a low-dimensional feature map and a high-dimensional feature map. The step of extracting the visual feature information from the image data and constructing a visual feature vector based on the visual feature information includes: Extracting first shallow features from the image data through a low-dimensional feature generation pyramid of a deep convolutional neural network, and generating the low-dimensional feature map based on the first shallow features, wherein the first shallow features represent local detail information in the image data; Extracting second shallow features and deep features from the image data through the high-dimensional feature generation pyramid of the deep convolutional neural network, and combining the second shallow features and the deep features to generate the high-dimensional feature map, wherein the second shallow features represent local detail information in the image data, and the deep features represent global semantic information in the image data; The low-dimensional feature map and the high-dimensional feature map are fused to obtain a visual comprehensive feature map, and the visual features in the visual comprehensive feature map are encoded into a visual feature vector.

3. The method according to claim 1, characterized in that The voice feature information includes a first voice feature and a second voice feature, the first voice feature is a spectrum feature, an acoustic feature and a waveform feature of the voice data, and the second voice feature is a biometric feature of the voice of the user to be taught.

4. The method according to claim 3, characterized in that The speech feature vector includes an emotional feature vector and a personalized feature vector. The step of extracting speech feature information from the speech data and constructing the speech feature vector based on the speech feature information includes: Extracting the first speech feature from the speech data, and constructing the emotion feature vector based on the first speech feature, wherein the emotion feature vector represents the speech emotion state of the user to be taught; The second speech feature is extracted from the speech data, and the personalized feature vector is constructed based on the second speech feature, wherein the personalized feature vector represents the personalized speech feature of the user to be taught.

5. The method according to claim 4, characterized in that The step of extracting the first speech feature from the speech data and constructing the emotion feature vector based on the first speech feature comprises: Extracting features from the spectrogram of the speech data by a speech spectrum feature extractor to obtain a first convolution feature map, wherein the first convolution feature map represents the spectral features and the acoustic features; Extracting features from the waveform data of the speech data by a speech waveform feature extractor to obtain a second convolution feature map, wherein the second convolution feature map represents the waveform features; The emotion feature vector is constructed based on the first convolution feature map and the second convolution feature map.

6. The method according to claim 5, characterized in that The step of constructing the emotion feature vector based on the first convolution feature map and the second convolution feature map comprises: Performing a fusion process on the first convolution feature map and the second convolution feature map to obtain a comprehensive speech feature map; The speech features in the speech comprehensive feature graph are weighted and summed to obtain comprehensive speech features, and the comprehensive speech features are encoded into the emotion feature vector.

7. The method according to any one of claims 1 to 6, characterized in that The step of predicting the emotional state type of the user to be taught based on the visual feature vector and the voice feature vector to execute a preset teaching strategy corresponding to the emotional state type includes: Using a multi-head attention mechanism to perform feature fusion on the visual feature vector and the speech feature vector to obtain a multimodal feature; The multimodal features are input into a preset multimodal classifier to obtain the emotional state type of the user to be taught, so as to execute a preset teaching strategy corresponding to the emotional state type.

8. A teaching device for training a service robot, characterized in that: The service robot training teaching device is applied to a service robot system, and the service robot training teaching device comprises: An acquisition module, used to acquire image data and voice data of the user to be taught; A visual feature extraction module, used to extract visual feature information from the image data, and construct a visual feature vector based on the visual feature information, wherein the visual feature vector represents the facial emotional state of the user to be taught; A speech feature extraction module, used to extract speech feature information from the speech data, and construct a speech feature vector based on the speech feature information, wherein the speech feature vector represents the speech individuality feature and speech emotional state of the user to be taught; A prediction module is used to predict the emotional state type of the user to be taught based on the visual feature vector and the voice feature vector, so as to execute a preset teaching strategy corresponding to the emotional state type.

9. A service robot, characterized in that: The service robot comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the teaching method for training a service robot as claimed in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the teaching method for training a service robot as described in any one of claims 1 to 7 are implemented.

Images