Home-based care robot intelligent interaction method based on artificial intelligence

Abstract

The invention discloses a home-based care robot intelligent interaction method based on artificial intelligence, and relates to the technical field of man-machine interaction, and the method comprises the steps: collecting a voice signal, a color image sequence and environment data obtained by a home-based care robot, inputting the voice signal, the color image sequence and the environment data into a multi-modal coding network through feature extraction to generate a situation representation vector, and forming a situation representation sequence; inputting the situation representation sequence into an intention and interaction state network, outputting an intention label and a parameter vector through an intention classification head and a parameter regression head, outputting an emotion label and a posture label through a state discrimination head, and combining the emotion label and the posture label into an interaction state label; the specific intention label, the parameter vector, the interaction state label and the current situation representation vector are input into a strategy generation network to generate a strategy primitive sequence, the home-based care robot executes interaction service operation according to the strategy primitive sequence, an interaction comfort evaluation value is obtained in the interaction process, and subjective satisfaction is obtained at the end of interaction. And safety monitoring and personalized interaction are both considered.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to an intelligent interaction method for a home care robot based on artificial intelligence. BACKGROUND

[0002] With the aggravation of population aging and the promotion of smart care for the elderly, home care robots have gradually become important interactive terminals for undertaking care and escort functions in family scenes. Existing home care robots usually integrate microphone arrays, cameras and environmental sensors, combine voice recognition, natural language processing, expression recognition and posture analysis technologies, obtain voice instructions, emotional states and basic behavior information of the elderly, and then drive voice broadcasting, screen display, indoor movement and home appliance control through preset rules, dialogue management or learning decision methods to realize home care services such as environmental regulation, medication and work and rest reminders, question and answer companionship and abnormal contact, thereby improving the life convenience and safety of the elderly in the family environment to a certain extent.

[0003] In the existing conventional technology, home care robots mostly focus on semantic intent recognition, and only use interactive states such as emotions and postures as additional references. There is a lack of linkage modeling between intent understanding and safety risk perception, and emotional pacification. At the same time, the interactive strategy usually relies on general scripts or static configurations, and it is difficult to adapt to individual preferences using interactive comfort and subjective satisfaction. The adaptive adjustment capability for different old people's speech speed, confirmation frequency and close-up method is limited. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides an intelligent interaction method for a home care robot based on artificial intelligence to solve the problem that intent understanding and interactive state evaluation are fragmented and interactive strategies are difficult to adaptively optimize according to interactive comfort and individual preferences.

[0006] To solve the above technical problems, the present application provides the following technical solutions: The present application provides an intelligent interaction method for a home care robot based on artificial intelligence, which comprises, Collecting voice signals, color image sequences and environmental data obtained by the home care robot, inputting the features into a multi-modal coding network to generate a context representation vector, and forming a context representation sequence; Inputting the context representation sequence into an intent and interactive state network, outputting an intent label and a parameter vector through an intent classification head and a parameter regression head, outputting an emotional label and a posture label through a state discrimination head, and combining the emotional label and the posture label into an interactive state label; The explicit intention label, parameter vector, interaction state label and current context representation vector are input into the strategy generation network to generate a strategy primitive sequence, and the home care robot executes the interaction service operation according to the strategy primitive sequence, obtains the interaction comfort evaluation value in the interaction process, and obtains the subjective satisfaction degree at the end of the interaction; The interaction comfort evaluation value, subjective satisfaction degree, context representation sequence, intention label, interaction state label and strategy primitive sequence are stored in an archive record, the individual preference vector is calculated according to the archive record when the robot is charging, and the training data is constructed according to the archive record and the individual preference vector, and the intention and interaction state network and the strategy generation network are updated in small steps.

[0007] As a preferred scheme of the home care robot intelligent interaction method based on artificial intelligence, wherein: the context representation vector generated by the feature extraction input multi-modal encoding network comprises, The time axis of the interaction process is divided into continuous time slices according to a fixed time length, the preprocessed voice signal in each time slice is divided into voice frames, the feature vector related to the mel spectrum is calculated for each voice frame, and the feature vectors of all voice frames in the time slice are normalized and statistically summarized to obtain the voice feature. Each color image belonging to the current time slice in each time slice is processed in turn, face detection and key point positioning are performed on each color image, representative images are selected at uniform time intervals in all color images contained in the current time slice, and visual feature vectors are extracted, all visual feature vectors in the current time slice are normalized and averaged to obtain the image feature of the current time slice. The environment data collected in each time slice is uniformly processed, and the average value and the change amplitude of each type of environment data are calculated in the time slice to form the environment feature of the time slice.

[0008] As a preferred scheme of the home care robot intelligent interaction method based on artificial intelligence, wherein: the context representation vector generated by the feature extraction input multi-modal encoding network further comprises, The voice feature, image feature and environment feature of the same time slice are spliced in order to obtain the multi-modal feature input vector of the current time slice, and the multi-modal feature input vector is input into the trained multi-modal encoding network to output the context representation vector of the current time slice. The context representation vectors of each time slice generated in time sequence are arranged in order from the start of the interaction to the current time, to obtain the context representation sequence.

[0009] As a preferred scheme of the intelligent interaction method of the home-based aged care robot based on artificial intelligence, wherein: the sequence of context representation is input into the intention and interaction state network, and an intention label and a parameter vector are output through an intention classification head and a parameter regression head, and the specific steps are, The most recent continuous context representation vectors are selected in chronological order to form an input segment, and the input segment is sent to the time sequence encoding part of the intention and interaction state network, and the time sequence encoding part performs multi-layer nonlinear transformation and context association calculation on each context representation vector in the input segment in chronological order to obtain an encoding result. The encoding result is sent to the intention classification head, and the intention classification head outputs a probability distribution corresponding to a preset intention category, from which the intention category with the maximum probability is selected as the intention label of the current round of interaction, and the same encoding result is sent to the parameter regression head, and the parameter regression head outputs a parameter vector corresponding to the intention label.

[0010] As a preferred scheme of the intelligent interaction method of the home-based aged care robot based on artificial intelligence, wherein: the state discrimination head outputs an emotion label and a posture label, and the emotion label and the posture label are combined into an interaction state label, and the specific steps are, The encoding result is sent to the state discrimination head, and the state discrimination head classifies the emotion state and the posture state respectively, and the emotion classification output obtains an emotion label, and the posture classification output obtains a posture label. The emotion label and the posture label are spliced to form an interaction state label.

[0011] As a preferred scheme of the intelligent interaction method of the home-based aged care robot based on artificial intelligence, wherein: the explicit intention label, the parameter vector, the interaction state label and the current context representation vector are input into the strategy generation network to generate a strategy primitive sequence, and the specific steps are, The explicit intention label, the parameter vector, the interaction state label and the current context representation vector are spliced into a decision input vector, the decision input vector is sent to the input end of the strategy generation network, the decision input vector is nonlinearly transformed and time correlation modeled inside, and the strategy primitives are sequentially output at the output end. Each strategy primitive is composed of a primitive type and a primitive parameter, and all strategy primitives are arranged in output order to form a strategy primitive sequence of the current round of interaction.

[0012] As a preferred scheme of the intelligent interaction method of the home-based aged care robot based on artificial intelligence, wherein: the home-based aged care robot performs an interaction service operation according to the strategy primitive sequence, and the specific steps are, The home care robot sequentially executes the interactive service operation in the order of the strategy primitive sequence, and when the strategy primitive is inquiry, confirmation, explanation and pacification, corresponding voice is broadcast through a loudspeaker and text and expression animation are synchronously displayed on a display screen; When the strategy primitive is navigation movement, a route is planned according to the target room and the navigation stop position in the primitive parameter, and the movement speed is dynamically adjusted in combination with the posture label in the interactive state label; When the strategy primitive is Internet of Things control, the on-off state of a household device is controlled according to the primitive parameter; When the strategy primitive is to dial a family phone, whether to initiate a help call is judged according to the emotion label in the interactive state label, and when help is needed, a call is initiated to a family number and the current call state is prompted on the display screen.

[0013] As a preferred scheme of the intelligent interaction method of the home care robot based on artificial intelligence, in the method, an interactive comfort degree evaluation value is obtained during an interaction process, and a subjective satisfaction degree is obtained at the end of the interaction, and the specific steps are, The interaction process is periodically sampled, speech understanding degree estimation, expression comfort degree estimation and posture stability degree estimation are respectively calculated at each sampling time, and are normalized into dimensionless values, the product of the speech understanding degree estimation, the expression comfort degree estimation and the posture stability degree estimation is taken as a cubic root at each sampling time, an instantaneous interaction evaluation value at the sampling time is obtained, and an arithmetic mean of all the instantaneous interaction evaluation values during a current round of interaction is calculated as an interactive comfort degree evaluation value of the current round of interaction; After the strategy primitive sequence is completely executed, the home care robot displays a satisfaction degree icon and synchronously broadcasts voice to guide the old person to make a subjective evaluation by touch and voice, and the subjective satisfaction degree is obtained.

[0014] As a preferred scheme of the intelligent interaction method of the home care robot based on artificial intelligence, in the method, an individual preference vector is counted according to an archive record when the robot is charging, and the specific steps are, When the home care robot enters a charging state and does not execute an interactive service operation, the archive records of the same old person are selected, and the strategy primitive sequence and the evaluation result in the archive records are statistically analyzed to obtain an individual preference vector of the old person.

[0015] As a preferred scheme of the intelligent interaction method of the home care robot based on artificial intelligence, in the method, the intention and interactive state network and the strategy generation network are updated in small steps, including, The training data is constructed according to the archive record, and the context representation sequence, the intention label, the parameter vector, the interaction state label, the strategy primitive sequence and the individual preference vector constitute a training sample, and the training sample is divided into preferred samples and ordinary samples according to the interaction comfort and subjective satisfaction; When the intention and interaction state network is updated, the context representation sequence and the individual preference vector are spliced and input, and the intention label and the interaction state label are used as supervision to fine-tune the output terminal parameters; When the strategy generation network is updated, the intention label, the parameter vector, the interaction state label, the current context representation vector and the individual preference vector are spliced as decision input, the preferred sample training round is more than the ordinary sample, and the new parameters are updated and the old parameters are retained.

[0016] The application has the beneficial effects that: the intention label and the parameter vector are output through the intention classification head and the parameter regression head, the emotion label and the posture label are output through the state discrimination head and combined into the interaction state label, the intention demand and the emotion posture of the old people are described in the unified context, used for joint decision of the strategy generation network, the individual preference vector is counted according to the archive record when the robot is charged, and the intention and interaction state network and the strategy generation network are updated in small steps, the individual preference model is formed according to the historical interaction, and the beneficial effects of considering safety monitoring and personalized interaction experience are achieved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Fig. 1 The flowchart of the intelligent interaction method of the home-based elderly care robot based on artificial intelligence.

[0019] Fig. 2 The flowchart of the multi-modal feature extraction and context representation vector generation.

[0020] Fig. 3 The flowchart of the intention and interaction state identification.

[0021] Fig. 4 The flowchart of the archive record and network update. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the specific embodiments of the application will be described in detail below with reference to the drawings of the specification.

[0023] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0024] Second, the "one embodiment" or "an embodiment" as used herein means a specific implementation that can include features, structures, or characteristics that are not included in all embodiments of the present application. The appearances of "in one embodiment" or "in an embodiment" in various places in this specification do not necessarily all refer to the same embodiment, nor are they necessarily mutually exclusive.

[0025] Reference Figs. 1-4 For one embodiment of the present application, the embodiment provides an artificial intelligence-based smart interaction method for a home care robot, comprising the following steps: S1, collect the voice signal, color image sequence and environment data obtained by the home care robot, input the feature extraction into the multi-modal coding network to generate the context representation vector, and form the context representation sequence.

[0026] A microphone array and a camera are fixedly installed on the home care robot, the microphone array is directed towards the main activity area of the elderly in the room, the field of view of the camera covers the upper body area of the elderly, and environment acquisition devices for measuring illumination, temperature, humidity, position and orientation are configured.

[0027] The time axis of the interaction process is divided into consecutive time slices according to a fixed time length (for example, 1 second), and an overlap of half the fixed time length is set between adjacent time slices. Each time slice contains voice signals, color images and environment data, and the voice signals, color images and environment data are respectively classified into corresponding time slice numbers through time stamps.

[0028] The voice signal in each time slice is preprocessed, and the mute endpoint detection, noise suppression and pre-emphasis processing are sequentially performed. The preprocessed voice signal is divided into voice frames in the time slice. The feature vector related to the mel spectrum is calculated for each voice frame, and the feature vectors of all voice frames in the time slice are normalized and statistically summarized to obtain the voice feature.

[0029] The calculation of the feature vector related to the mel spectrum refers to the frequency domain transformation of each voice frame to obtain the energy distribution of the voice frame at different frequencies. Through the mel scale close to human ear perception, the frequency axis is divided into auditory frequency bands, and the energy size in each auditory frequency band is calculated. The energy values of all auditory frequency bands of the voice frame are arranged in order to form a string of numbers, which is the feature vector related to the mel spectrum.

[0030] The color images within each time slice are processed sequentially. Face detection and key point localization are performed on each color image. The color image region containing the elderly person's face and upper body is cropped and uniformly scaled to the same size. Representative images are selected from all color images in the current time slice at uniform time intervals. Visual feature vectors are extracted from each representative image. All visual feature vectors are normalized and averaged within the current time slice to obtain the image features of the current time slice.

[0031] The environmental data collected in each time slice is processed uniformly. The light data, temperature data, humidity data, and the position and orientation data of the home-based elderly care robot in the time slice are arranged in chronological order. The average value and change range of each type of environmental data are calculated in the time slice to form the environmental features corresponding to the time slice. The environmental features, voice features, and image features are aligned under the same time slice number.

[0032] The speech features, image features, and environmental features of the same time slice are concatenated in sequence to obtain the multimodal feature input vector of the current time slice. The multimodal feature input vector is fed into a trained multimodal coding network to output a contextual representation vector for the current time slice. The contextual representation vector centrally represents the speech content, facial expressions, postures, and environmental states related to human-computer interaction within the current time slice.

[0033] During the training phase, the speech features, image features, and environmental features of each time slice are sequentially concatenated into a multimodal feature input vector, which is then fed into the input of the multimodal coding network. Internally, the multimodal coding network performs intrachannel processing on the speech features, image features, and environmental features and performs joint modeling through a cross-attention structure, generating a corresponding contextual representation vector at the output. During training, a reconstruction branch and a prediction branch are connected after the multimodal coding network. The reconstruction branch reconstructs the speech features, image features, and environmental features of the current time slice based on the contextual representation vector, while the prediction branch estimates the multimodal features of subsequent time slices based on the contextual representation vector of the current time slice. By comparing the error between the reconstruction result and the original multimodal features, as well as the error between the prediction result and the actual subsequent multimodal features, the internal parameters of the multimodal coding network are automatically adjusted.

[0034] The multimodal coding network (MCN) is repeatedly input, reconstructed, and predicted on the training dataset, and its parameters are continuously updated until the reconstruction error and prediction error on the validation data remain stable within a preset training cycle. At this point, the MCN is considered to have the ability to express the context for intelligent interaction of home-based elderly care robots. The parameters of the MCN are then solidified, and the solidified MCN is deployed inside the home-based elderly care robot. This allows the home-based elderly care robot to generate a context representation vector by performing a single forward computation on the collected speech features, image features, and environmental features during actual operation.

[0035] The context representation vectors of each time slice generated in chronological order are arranged in order from the start of the interaction to the current time according to the time slice number, to obtain the context representation sequence, which is then stored inside the home-based elderly care robot.

[0036] S2. Input the context representation sequence into the intent and interaction state network. The intent label and parameter vector are output through the intent classification head and parameter regression head. The emotion label and posture label are output through the state discrimination head. The emotion label and posture label are combined into the interaction state label.

[0037] From the context representation sequence, the most recent consecutive context representation vectors are selected in chronological order to form an input segment. This input segment is then fed into the temporal encoding part of the intent and interaction state network. The temporal encoding part performs multi-layer nonlinear transformations and context association calculations on each context representation vector in the input segment in chronological order to obtain an encoding result containing information about the entire context change. This encoding result is then fed into the intent classification head, which outputs a probability distribution corresponding to a preset intent category. The intent category with the highest probability is selected as the intent label for the current round of interaction. Simultaneously, the same encoding result is fed into the parameter regression head, which outputs a parameter vector corresponding to the intent label. This allows the intent label and parameter vector to jointly determine the target task for the current round of interaction.

[0038] Preset intent categories refer to several pre-defined intents that home-based elderly care robots need to recognize, such as environmental adjustment, medication reminders, companionship conversations, object retrieval assistance, requesting help, and daily greetings.

[0039] The encoding results of the timing encoding part are sent to the state discrimination head. The state discrimination head classifies the elderly's emotional state and posture state respectively. The emotional classification outputs an emotional label, and the posture classification outputs a posture label. The emotional label is used to represent the elderly's current emotional state such as stable, happy, tense, sad or angry. The posture label is used to represent the elderly's current state of sitting stability, standing stability, lying stability or suspected fall. The emotional label and posture label are concatenated to form an interaction state label that reflects the overall safety status and emotional state of the current round of interaction.

[0040] During the training phase of the intent and interaction state network, training data is constructed using stored context representation sequences and corresponding intent labels, parameter vectors, emotion labels, and posture labels. Each training context representation sequence is fed into the temporal encoding part, and the intent classification head, parameter regression head, and state discrimination head output prediction results respectively. The errors between the predicted intent label and the real intent label, the predicted parameter vector and the real parameter vector, and the predicted emotion label and posture label and the real emotion label and posture label are used as training losses. The internal parameters of the intent and interaction state network are automatically adjusted based on the training loss. After a preset number of training rounds, reference values ​​for distinguishing between explicit intent labels and uncertain intent labels are obtained by statistically analyzing the probability distribution output by the intent classification head on the validation dataset. The adjusted intent and interaction state network parameters and reference values ​​are then deployed into the home-based elderly care robot.

[0041] The maximum probability value in the probability distribution output by the intent classification head is read and compared with the reference value obtained from the validation dataset during the training phase. When the maximum probability value is less than the reference value, the current intent label is considered uncertain. At this time, the home-based elderly care robot broadcasts clarification statements through the speaker and camera according to the preset clarification script and facial animation, guiding the elderly to confirm their real needs with simple sentences. At the same time, new voice signals, color images and environmental data are collected. After feature extraction and multimodal encoding, a new context representation vector is generated. The new context representation vector is appended to the end of the context representation sequence. The most recent continuous context representation vector is selected from the updated context representation sequence to form an input segment and sent to the intent and interaction state network for inference.

[0042] When the maximum probability value output by the intent classification head is not less than the reference value and the emotion label and posture label output by the state discrimination head remain stable for multiple consecutive times, the intent label obtained at this time is determined as a clear intent label, the output of the parameter regression head obtained at this time is determined as a clear parameter vector, the interaction state label obtained by combining the emotion label and posture label at this time is determined as a clear interaction state label, and the context representation vector corresponding to the current round of interaction is selected from the context representation sequence as the current context representation vector.

[0043] S3. Input the explicit intent label, parameter vector, interaction state label and current context representation vector into the strategy generation network to generate a strategy primitive sequence. The home-based elderly care robot executes interactive service operations according to the strategy primitive sequence, obtains the interaction comfort evaluation value during the interaction process, and obtains the subjective satisfaction at the end of the interaction.

[0044] The explicit intent label, parameter vector, interaction state label, and current context representation vector are concatenated into a decision input vector. This decision input vector is then fed into the input of the policy generation network. Internally, the decision input vector undergoes nonlinear transformation and temporal correlation modeling. At the output, policy primitives are output sequentially. Each policy primitive includes a primitive type and primitive parameters corresponding to that type. All policy primitives are arranged in the output order to form the policy primitive sequence for the current round of interaction.

[0045] The primitive type refers to one of the following: inquiry primitive, confirmation primitive, explanation primitive, reassurance primitive, voice broadcast primitive, screen display primitive, navigation and movement primitive, Internet of Things control primitive, and dialing family members' phone primitive.

[0046] Primitive parameters refer to information used to indicate the target room, target object, voice content template number, navigation stop location, and IoT device identifier.

[0047] During the training of the policy generation network, the network is instructed to predict the entire policy primitive sequence in sequence when given a decision input vector. By comparing the difference between the predicted policy primitive sequence and the training output policy primitive sequence, the network's internal parameters are automatically adjusted. This allows the network to gradually learn to generate policy primitive sequences that are more acceptable to the elderly and also ensure safety under different intent labels, parameter vectors, interaction state labels, and different context representation vectors. After training, the obtained policy generation network parameters are solidified and deployed into the home-based elderly care robot.

[0048] The home-based elderly care robot executes interactive service operations sequentially according to the order in the strategy primitive sequence. When the strategy primitive is an inquiry primitive, confirmation primitive, explanation primitive, or reassurance primitive, the robot controls the speaker to generate corresponding voice broadcasts according to the voice content template, and simultaneously displays text and facial animations matching the voice content on the display screen. When the strategy primitive is a navigation and movement primitive, the robot invokes existing navigation capabilities to drive the chassis to move along the planned route based on the target room and navigation stop position in the primitive parameters, and adjusts the movement speed and the proximity distance to the elderly based on the posture labels in the interaction status labels during the movement. When the strategy primitive is an IoT control primitive, the robot controls the on / off status of lights, TVs, or home appliances through the IoT interface based on the target object and IoT device identifier in the primitive parameters. When the strategy primitive is a call to family members primitive, the robot determines whether it needs to make a call to family members based on the emotion labels in the interaction status labels. If a call to family members is needed, the robot initiates a call to the preset family member number, and simultaneously displays the current call status on the display screen.

[0049] The interaction process is sampled, and at each sampling time, the speech comprehension estimation, facial comfort estimation, and posture stability estimation are obtained. The speech comprehension estimation is obtained by analyzing the semantic similarity between the elderly's response and the most recent speech broadcast within the sampling period, as well as the proportion of the elderly using affirmative, negative, and repetitive expressions, and is normalized to a dimensionless value. The facial comfort estimation is obtained by analyzing the elderly's facial expressions in the color images captured by the camera, and gives the comfort level value in the expression categories such as calm, smiling, and frowning, and is normalized to a dimensionless value. The posture stability estimation is obtained by analyzing posture labels and the relative distance between the home care robot and the elderly, and maps sitting stability, standing stability, lying stability, and suspected fall status to dimensionless values.

[0050] At each sampling time, the cube root of the product of the speech comprehension estimation, facial expression comfort estimation, and posture stability estimation at the sampling time is used as the instantaneous interaction evaluation value at the sampling time. During the duration of the current round of interaction, the arithmetic mean of the instantaneous interaction evaluation values ​​obtained at all sampling times is calculated, and the average result is used as the interaction comfort evaluation value for the current round of interaction.

[0051] After all the policy primitives are executed, the home-based elderly care robot displays multiple satisfaction level icons on the screen and simultaneously provides voice prompts for the elderly to evaluate. The robot guides the elderly to select one of "satisfied," "neutral," or "dissatisfied" via the touchscreen, or to give a brief voice response indicating their level of satisfaction. The elderly's selection or voice response is mapped to their subjective satisfaction level, and the interaction comfort rating and subjective satisfaction level corresponding to the current round of interaction are stored as the evaluation results of the current round of interaction.

[0052] S4. Store the interaction comfort evaluation value, subjective satisfaction, context representation sequence, intent label, interaction state label and policy primitive sequence into the archive record. When the robot is charging, count the individual preference vector according to the archive record. Construct training data based on the archive record and individual preference vector, and update the intent and interaction state network and policy generation network in small steps.

[0053] After each round of interaction ends and the interaction comfort evaluation value and subjective satisfaction value for the current round of interaction are obtained, the context representation sequence, explicit intent label, explicit parameter vector, explicit interaction state label, actual executed strategy primitive sequence, interaction comfort evaluation value and subjective satisfaction value for that round of interaction are written into the archive record.

[0054] When the home-based elderly care robot is in charging mode and not performing interactive service operations, archived records belonging to the same elderly person are selected. Statistical analysis is performed on the strategy primitive sequences and evaluation results in the archived records to obtain the elderly person's individual preference vector. The individual preference vector includes components such as speech rate preference, confirmation frequency preference, navigation dwell position preference, and IoT control preference. Speech rate preference is determined by statistically analyzing the correspondence between speech rate configurations under different voice content templates and subjective satisfaction. Confirmation frequency preference is determined by statistically analyzing the correspondence between different combinations of question-type and confirmation-type primitives and subjective satisfaction. Navigation dwell position preference is determined by statistically analyzing the correspondence between different dwell positions in navigation movement primitives and interaction comfort evaluation values. IoT control preference is determined by statistically analyzing the correspondence between the target object of IoT control primitives and subjective satisfaction.

[0055] Training data is constructed based on the same archived record. The training samples are composed of the context representation sequence, explicit intent label, explicit parameter vector, explicit interaction state label, actual executed policy primitive sequence and corresponding individual preference vector in the archived record. The context representation vector corresponding to the current round of interaction is selected from the context representation sequence as the current context representation vector. The training samples with an interaction comfort evaluation value not less than the average of all interaction comfort evaluation values ​​and a subjective satisfaction level of "satisfied" or "average" are marked as preferred samples, and the remaining training samples are marked as ordinary samples.

[0056] When updating the intent and interaction state network with small steps, the context representation sequence and individual preference vector in the training samples are concatenated and input into the intent and interaction state network. The intent labels and interaction state labels recorded in the training samples are used as supervision signals. The parameters near the output end are slightly adjusted according to the difference between the output of the intent and interaction state network and the supervision signals.

[0057] When updating the policy generation network in small steps, the explicit intent label, parameter vector, interaction state label, current context representation vector, and individual preference vector in the training samples are concatenated into a decision input vector and input into the policy generation network. The actual policy primitive sequences recorded in the training samples are used as supervision signals. Multiple rounds of training are performed on the preferred samples, and a few rounds of training are performed on the ordinary samples. Under the same intent label, interaction state label, and context representation vector, the policy generation network gradually tends to generate policy primitive sequences that are easier to obtain high interaction comfort evaluation values ​​and subjective satisfaction in the historical records.

[0058] The updated intent and interaction state network and policy generation network parameters are solidified, while the parameters before the update are retained as backups. When the interaction comfort evaluation value is found to be declining for a long period of time during operation, the backup parameters are switched back. In the subsequent actual interaction process, the newly generated context representation sequence, intent label, interaction state label, policy primitive sequence, interaction comfort evaluation value and subjective satisfaction continue to be added to the archived record.

[0059] In summary, this invention achieves the ability to characterize the intentions, needs, emotions, and postures of the elderly within a unified context by outputting intention labels and parameter vectors through an intention classification head and a parameter regression head, and outputting emotion labels and posture labels through a state discrimination head and combining them into an interaction state label. This allows for joint decision-making by the policy generation network. By statistically analyzing individual preference vectors based on archived records during robot charging and updating the intention and interaction state network and the policy generation network in small steps, an individual preference model is formed based on historical interactions, achieving the beneficial effect of balancing safety monitoring and personalized interaction experience.

[0060] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An intelligent interaction method for home-based elderly care robots based on artificial intelligence, characterized in that: include, Voice signals, color image sequences, and environmental data obtained by home-based elderly care robots are collected, and after feature extraction, they are input into a multimodal coding network to generate context representation vectors and form context representation sequences. The context representation sequence is input into the intent and interaction state network. The intent label and parameter vector are output through the intent classification head and parameter regression head. The emotion label and posture label are output through the state discrimination head. The emotion label and posture label are combined into the interaction state label. The system inputs a clear intent label, parameter vector, interaction state label, and current context representation vector into the strategy generation network to generate a sequence of strategy primitives. The home-based elderly care robot executes interactive service operations according to the sequence of strategy primitives, obtains an interaction comfort evaluation value during the interaction process, and obtains subjective satisfaction at the end of the interaction. The interaction comfort evaluation value, subjective satisfaction, context representation sequence, intent label, interaction state label and policy primitive sequence are stored in the archive record. When the robot is charging, the individual preference vector is counted according to the archive record. Training data is constructed based on the archive record and the individual preference vector, and the intent and interaction state network and policy generation network are updated in small steps.

2. The intelligent interaction method for home-based elderly care robots based on artificial intelligence as described in claim 1, characterized in that: The context representation vector generated by the feature extraction input multimodal coding network includes, The timeline of the interaction process is divided into continuous time slices with a fixed duration. The speech signal in each time slice is preprocessed. The preprocessed speech signal is divided into speech frames within the time slice. Mel-spectrum correlation feature vectors are calculated for each speech frame. The feature vectors of all speech frames are normalized and statistically summarized within the time slice to obtain speech features. The color images belonging to the current time slice are processed sequentially within each time slice. Face detection and key point localization are performed on each color image. Representative images are selected from all color images contained in the current time slice at uniform time intervals and visual feature vectors are extracted. All visual feature vectors are normalized and averaged within the current time slice to obtain the image features of the current time slice. The environmental data collected in each time slice is processed uniformly, and the average value and change range of each type of environmental data are calculated within the time slice to form the environmental characteristics of the time slice.

3. The intelligent interaction method for home-based elderly care robots based on artificial intelligence as described in claim 2, characterized in that: The feature-extracted input multimodal coding network generates a contextual representation vector, which also includes: The speech features, image features, and environmental features of the same time slice are concatenated in sequence to obtain the multimodal feature input vector of the current time slice. The multimodal feature input vector is then fed into a trained multimodal coding network to output a contextual representation vector for the current time slice. The context representation vectors of each time slice generated in chronological order are arranged in order from the start of the interaction to the current moment to obtain the context representation sequence.

4. The intelligent interaction method for home-based elderly care robots based on artificial intelligence as described in claim 3, characterized in that: The specific steps are as follows: inputting the context representation sequence into the intent and interaction state network, and outputting intent labels and parameter vectors through the intent classification head and parameter regression head. The input segment is composed of the most recent consecutive context representation vectors in chronological order. The input segment is then fed into the temporal coding part of the intent and interaction state network. The temporal coding part performs multi-layer nonlinear transformation and context association calculation on each context representation vector in the input segment in chronological order to obtain the coding result. The encoding result is sent to the intent classification head, which outputs the probability distribution of the corresponding preset intent category. The intent category with the highest probability is selected as the intent label for the current round of interaction. At the same time, the same encoding result is sent to the parameter regression head, which outputs the parameter vector corresponding to the intent label.

5. The intelligent interaction method for home-based elderly care robots based on artificial intelligence as described in claim 4, characterized in that: The process involves outputting emotion and posture labels via the state discrimination head, and combining these labels into an interaction state label. The specific steps are as follows: The encoded result is fed into the state discrimination head, which classifies the emotional state and the posture state respectively. The emotional classification outputs an emotional label, and the posture classification outputs a posture label. Emotion tags and posture tags are combined to form interactive status tags.

6. The intelligent interaction method for home-based elderly care robots based on artificial intelligence as described in claim 5, characterized in that: The specific steps for inputting explicit intent labels, parameter vectors, interaction state labels, and current context representation vectors into the policy generation network to generate a sequence of policy primitives are as follows: The explicit intent label, parameter vector, interaction state label, and current context representation vector are concatenated into a decision input vector. This decision input vector is then fed into the input of the policy generation network. Internally, the decision input vector undergoes nonlinear transformation and temporal correlation modeling. At the output end, policy primitives are output sequentially. Each policy primitive consists of a primitive type and primitive parameters. All policy primitives are arranged in the output order to form the policy primitive sequence for the current round of interaction.

7. The intelligent interaction method for home-based elderly care robots based on artificial intelligence as described in claim 6, characterized in that: The home-based elderly care robot executes interactive service operations according to a sequence of policy primitives. The specific steps are as follows: The home-based elderly care robot executes interactive service operations in the order of the strategy primitives. When the strategy primitives are to ask, confirm, explain, and comfort, the robot will broadcast the corresponding voice through the speaker and simultaneously display text and facial animation on the screen. When the strategy primitive is navigation movement, the route is planned according to the target room and navigation stop position in the primitive parameters, and the movement speed is dynamically adjusted in combination with the posture label in the interaction state label. When the policy primitive is IoT control, the on / off state of home appliances is controlled according to the primitive parameters. When the strategy primitive is to call a family member, the system determines whether to initiate a help call based on the emotion tag in the interaction status label. If help is needed, the system initiates a call to the family member's number and displays the current call status on the screen.

8. The intelligent interaction method for home-based elderly care robots based on artificial intelligence as described in claim 7, characterized in that: The specific steps for obtaining an interaction comfort evaluation value during the interaction process and obtaining subjective satisfaction at the end of the interaction are as follows: The interaction process is periodically sampled. At each sampling time, the speech intelligibility estimate, facial expression comfort estimate, and posture stability estimate are calculated and normalized to dimensionless values. At each sampling time, the cube root of the product of the speech intelligibility estimate, facial expression comfort estimate, and posture stability estimate is taken to obtain the instantaneous interaction evaluation value at the sampling time. During the duration of the current round of interaction, the arithmetic mean of all instantaneous interaction evaluation values ​​is calculated as the interaction comfort evaluation value of the current round of interaction. After all the policy primitives have been executed, the home-based elderly care robot guides the elderly to make subjective evaluations by displaying satisfaction icons and simultaneously broadcasting voice messages, thereby obtaining subjective satisfaction.

9. The intelligent interaction method for home-based elderly care robots based on artificial intelligence as described in claim 8, characterized in that: The specific steps for calculating individual preference vectors based on archived records during robot charging are as follows: When the home-based elderly care robot is in charging mode and not performing interactive service operations, the archived records belonging to the same elderly person are selected, and the strategy primitive sequence and evaluation results in the archived records are statistically analyzed to obtain the elderly person's individual preference vector.

10. The intelligent interaction method for home-based elderly care robots based on artificial intelligence as described in claim 9, characterized in that: The step-by-step update of the intent and interaction state network and the policy generation network includes, Training data is constructed based on archived records. The training samples consist of context representation sequences, intent labels, parameter vectors, interaction state labels, policy primitive sequences, and individual preference vectors. The training samples are divided into preferred samples and ordinary samples based on interaction comfort and subjective satisfaction. When updating the intent and interaction state network, the context representation sequence and individual preference vector are concatenated as input, and the output parameters are fine-tuned with intent labels and interaction state labels as supervision. When updating the policy generation network, the intent label, parameter vector, interaction state label, current context representation vector, and individual preference vector are concatenated as the decision input. The selected samples are trained more times than the ordinary samples. After the update, the new parameters are fixed and the old parameters are retained.

Images