Multimodal human-computer interaction methods and devices for the elderly
By acquiring behavioral data from the elderly to assess their sensory and emotional states and dynamically adjusting multimodal interaction methods, the problem of cognitive load caused by multimodal parallel output is solved, achieving an efficient and personalized interactive experience for the elderly.
Patent Information
- Application Number
- CN202610982624.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-02
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, multimodal parallel output increases the cognitive load on the elderly, resulting in low interaction success rates and emotional acceptance. Manual settings are also inflexible and have high barriers to entry, making it difficult to adapt to fluctuations in sensory abilities.
By acquiring behavioral data of interactive objects, assessing their sensory abilities and emotional states, dynamically adjusting the combination of interaction modalities, selecting primary and secondary modalities and configuring parameters, and adapting the interaction data to reduce the operational threshold.
It improves the success rate of interaction and emotional acceptance, reduces the complexity of operation, adapts to the sensory abilities and emotional changes of the elderly, and provides a personalized interactive experience.
Smart Images

Figure CN123114246A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal human-computer interaction method and device for the elderly. Background Technology
[0002] During interactions with users, companion robots utilize multimodal interaction schemes, such as multimodal parallel output, which simultaneously outputs data in multiple modalities, including voice, on-screen text, and animated facial expressions. Alternatively, users can manually set the modality of the companion robot's output for each interaction, ensuring that the robot outputs data according to that modality each time.
[0003] The above-mentioned solutions have the following drawbacks when targeting human-computer interaction scenarios for the elderly: First, the multimodal parallel output method increases cognitive load. The elderly have limited cognitive resources, and information overload actually reduces comprehension efficiency, resulting in a lower success rate and lower emotional acceptance. Second, manual settings are difficult and inflexible. The elderly have difficulty operating the settings menu, and their sensory abilities fluctuate continuously. Fixed-setting modalities are difficult to adapt to fluctuating sensory abilities, leading to a lower success rate and lower emotional acceptance. Summary of the Invention
[0004] This invention provides a multimodal human-computer interaction method and device for the elderly, which solves the defects of existing technologies that increase cognitive load due to multiple channels running in parallel or that lead to low interaction success rate due to high manual thresholds. It achieves improved interaction success rate and emotional acceptance while reducing the operation threshold of the interaction object.
[0005] This invention provides a multimodal human-computer interaction method for the elderly, comprising the following steps: Obtain behavioral data of the interacting object; The sensory ability assessment results and emotional state of the interactive object are obtained based on behavioral data. The sensory ability assessment results are the result of assessing at least one of hearing, vision, touch ability and language expression ability. Based on the sensory ability assessment results and emotional state, a predictive modality group is obtained, which includes at least one interaction modality; Based on at least one interaction mode in the predicted mode group, the companion robot is controlled to output interaction data with the interaction object.
[0006] According to the present invention, a multimodal human-computer interaction method for the elderly is provided, which controls a companion robot to output interaction data with the interaction object based on at least one interaction mode in a predicted mode group, including: Choose one interaction mode from the various interaction modes as the primary mode and the rest as auxiliary modes; Generate the first interaction data of the main modality, and generate the second interaction data of the auxiliary modality based on the first interaction data of the main modality; Control the companion robot to output the first and second interaction data.
[0007] According to the present invention, a multimodal human-computer interaction method for the elderly is provided, which controls a companion robot to output first interaction data and second interaction data, including: Obtain the first parameter configuration information of the primary mode and the second parameter configuration information of the auxiliary mode; Based on the first parameter configuration information, adjust the interaction parameters of the first interactive data; and based on the second parameter configuration information, adjust the interaction parameters of the second interactive data.
[0008] According to the present invention, a multimodal human-computer interaction method for the elderly generates first interaction data of the main modality, including: Determine if there is any sub-interaction data to be output; If it does not exist, generate initial interaction data based on historical dialogue data; If the number of key points in the initial interaction data exceeds the preset number, the initial interaction data is split into multiple sub-interaction data. The first interaction data is obtained based on one of the sub-interaction data, and each of the sub-interaction data that was not selected among the multiple sub-interaction data is taken as the sub-interaction data to be output.
[0009] According to the present invention, a multimodal human-computer interaction method for the elderly, if the modality of the first interaction data or the second interaction data is voice, controls a companion robot to output the first interaction data and the second interaction data, including: Dialect adaptation processing is performed on the speech modal of the first and second interactive data to obtain the target interactive data; The companion robot is controlled to output target interaction data, as well as interaction data from the first and second interaction data that have not been processed for dialect adaptation.
[0010] According to the present invention, a multimodal human-computer interaction method for the elderly is provided, which obtains a predictive modality set based on sensory ability assessment results and emotional state, including: The sensory ability assessment results and emotional state are processed using an emotion-modality preference association model to obtain a predictive modality group; Among them, the sentiment and modality preference association model is obtained based on the interaction results of interactive objects in the historical process.
[0011] According to the multimodal human-computer interaction method for the elderly provided by the present invention, after controlling a companion robot to output interaction data with the interaction object based on at least one interaction mode in a predicted mode group, the method further includes: Obtain the interaction results of the interactive object regarding the interaction data. The interaction results include the interaction completion rate, abandonment rate, and / or repeat request frequency. The emotion-modality preference association model was adjusted based on sensory ability assessment results, emotional state, predicted modality groups, and interaction results.
[0012] According to the present invention, a multimodal human-computer interaction method for the elderly obtains sensory ability assessment results of the interactive object based on behavioral data, including: The response data of the interactive voice output by the companion robot in the behavioral data is evaluated to obtain the hearing assessment results; The visual assessment results are obtained by evaluating the response data of the interactive screen output by the companion robot in the behavioral data. The response data of the interactive screen output by the companion robot in the behavioral data is evaluated to obtain the touch control capability evaluation results; The speech output by the interactive objects in the behavioral data is evaluated to obtain the language expression ability assessment results; The sensory ability assessment results are obtained by combining the hearing assessment results, vision assessment results, touch ability assessment results, and language expression ability assessment results.
[0013] According to the present invention, a multimodal human-computer interaction method for the elderly is provided, in which a companion robot is equipped with several sensors to acquire behavioral data of the interaction object, including: Acquire data collected by each sensor; Based on the data collected by each sensor, the behavioral data of the interactive object is obtained.
[0014] The present invention also provides a multimodal human-computer interaction device for the elderly, comprising the following modules: The behavior data acquisition module is used to acquire the behavior data of the interactive objects; The sensory ability assessment and emotional state confirmation module is used to obtain the sensory ability assessment results and emotional state of the interactive object based on behavioral data. The sensory ability assessment results are the results of assessing at least one of hearing, vision, touch ability and language expression ability. A modality determination module is used to obtain a predictive modality set based on sensory ability assessment results and emotional state, wherein the predictive modality set includes at least one interaction modality; The interaction output module is used to control the companion robot to output interaction data with the interaction object based on at least one interaction mode in the prediction mode group.
[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described multimodal human-computer interaction methods for the elderly.
[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal human-computer interaction methods for the elderly described above.
[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described multimodal human-computer interaction methods for the elderly.
[0018] This invention provides a multimodal human-computer interaction method and device for the elderly. It determines the sensory abilities and emotional states of the interactive object based on the behavioral data of the interactive object, and then adaptively determines the interaction mode suitable for the current sensory abilities and emotions of the interactive object. The interactive data is then output according to the determined interaction mode. Compared with multi-channel parallelism which increases cognitive load or manual settings, this invention improves the success rate of interaction and emotional acceptance by reducing the operation threshold of the interactive object. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0020] Figure 1 This is one of the flowcharts of the multimodal human-computer interaction method for the elderly provided by the present invention.
[0021] Figure 2 This is the second flowchart of the multimodal human-computer interaction method for the elderly provided by the present invention.
[0022] Figure 3 This is the third flowchart of the multimodal human-computer interaction method for the elderly provided by this invention.
[0023] Figure 4 This is a schematic diagram of the structure of the multimodal human-computer interaction device for the elderly provided by the present invention.
[0024] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0026] The following is combined with Figures 1 to 3 This invention describes a multimodal human-computer interaction method for the elderly. This method can be applied to multimodal human-computer interaction devices, companion robots, and other electronic devices, or to servers and other devices that communicate with companion robots. Figure 1 This is one of the flowcharts illustrating the multimodal human-computer interaction method for the elderly provided by this invention, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain the behavior data of the interactive object.
[0027] The interaction object can be someone who interacts with the companion robot, such as an elderly person.
[0028] Behavioral data refers to various types of data generated by interactive objects during their interaction with the external environment or devices, which can reflect their physiological, behavioral, and emotional states.
[0029] Behavioral data can be acquired in various ways. For example, it can be collected through multi-source sensors mounted on a companion robot. These sensors may include, but are not limited to, at least one of the following: cameras for capturing facial expressions and body movements, microphone arrays for collecting voice signals and ambient sounds, touchscreens for recording screen clicks and swipes, and distance sensors for measuring the distance between the interacting object and the device. By continuously collecting raw data through these sensors, and after preliminary processing, structured behavioral data is formed, providing a data foundation for subsequent sensory ability assessment and emotional state recognition. For example, a camera can capture the squinting motion of the interacting object when looking at the screen, a microphone can capture the voice of the interacting object requesting repeated commands, and a touchscreen can record the precise location and time taken for the interacting object to click a button.
[0030] Step 102: Obtain the sensory ability assessment results and emotional state of the interactive object based on the behavioral data.
[0031] The sensory abilities that this embodiment focuses on include, but are not limited to, at least one of the following: hearing, vision, touch control, and language expression. The sensory ability assessment result is the result of evaluating at least one of hearing, vision, touch control, and language expression.
[0032] Specifically, the following methods can be used to dynamically assess an interactive object's hearing: analyzing behavioral data such as response delay to different volume levels of speech and the frequency of requests to repeat "please say it again." Visual acuity can be assessed by tracking behavioral data such as the distance the interactive object views the screen, whether they squint, and the speed at which they read text. Touchscreen capabilities can be evaluated by analyzing the accuracy of touchscreen clicks (whether the target is hit), the stability of swipe paths, and the time required to complete specific operations (such as dragging a slider). Finally, speech data can be analyzed to assess language expression ability, such as clarity of pronunciation, presence of a strong regional accent, and fluency of speech.
[0033] Emotional state refers to the current emotion or mood of an interacting object, such as joy, sadness, anxiety, or calmness. Methods for obtaining emotional state can include: analyzing facial images in behavioral data to extract facial expression features; analyzing speech signals to extract speech features such as pitch, speech rate, and energy; and analyzing the frequency and content of the interacting object's behavior to extract interaction frequency features. Based on facial expression features, speech features, and interaction frequency features, the current emotional state of the interacting object is obtained. For example, an emotional state recognition module can be pre-built, using facial images, speech signals, and interaction frequency as inputs. By extracting facial expression features, speech features, and interaction frequency features, the module can obtain the current emotional state of the interacting object.
[0034] Step 103: Based on the sensory ability assessment results and emotional state, a predictive modality group is obtained.
[0035] The prediction modality set includes at least one interaction modality. That is, the prediction modality set can contain one or more interaction modalities. An interaction modality refers to the way information is presented, such as speech, text, images, animation, etc.
[0036] Step 103 can be considered as the process of achieving personalized and emotionally adapted interaction modalities. This predictive modality set is dynamically changing and can reflect the current needs of the interaction object in real time. For example, when it is assessed that the interaction object has poor eyesight and is emotionally depressed, it is appropriate to output a slow and gentle voice and an interactive interface with low-saturation warm colors. Therefore, the predictive modality set can include voice + animation.
[0037] Step 104: Based on at least one interaction mode in the predicted mode group, control the companion robot to output interaction data with the interaction object.
[0038] Interactive data refers to the information that the companion robot actually outputs to the interacting object. For example, if the predictive modality set is "voice + image," then for a query like "What will the weather be like tomorrow?", the companion robot will not only announce "It will be colder tomorrow, with light rain" via voice, but can also simultaneously display a temperature curve and suggested clothing images on the screen. In this way, the abstract modality set can be transformed into concrete interactive output.
[0039] In step 104, the companion robot can first generate interaction data for each interaction mode, and then output the interaction data.
[0040] In the above scheme, the sensory abilities and emotional states of the interactive object are determined based on the behavioral data of the interactive object. Then, an interaction mode suitable for the current sensory abilities and emotions of the interactive object is adaptively determined based on the sensory abilities and emotional states. Interaction data is then output based on the determined interaction mode. Compared with multi-channel parallelism which increases cognitive load or manual settings, this invention improves the success rate of interaction and emotional acceptance by reducing the operation threshold of the interactive object.
[0041] In some embodiments, the companion robot is equipped with several sensors to acquire behavioral data of the interactive object, including: acquiring data collected by each sensor; and obtaining behavioral data of the interactive object based on the data collected by each sensor.
[0042] Several sensors may include image sensors (such as color depth sensors, which simultaneously capture facial images of the interactive object and the distance between the interactive object and the display screen), microphone arrays for capturing voice signals and ambient sounds, and touchscreens for recording screen clicks and swipes.
[0043] One way to obtain the behavioral data of the interactive object based on the data collected by each sensor is to either directly use the data collected by each sensor as the behavioral data of the interactive object, or to obtain the behavioral data of the interactive object after denoising the data collected by some of the sensors. For example, multiple facial images collected by an image sensor can be filtered to obtain the final facial image. The filtering criteria could be image clarity, facial region integrity, etc.
[0044] In this embodiment, behavioral data is acquired through sensors, eliminating the need for manual settings of the interactive object and lowering the operational threshold.
[0045] In some embodiments, the above-described method of obtaining a predicted modality group based on sensory ability assessment results and emotional state may include: processing the sensory ability assessment results and emotional state using an emotion-modality preference association model to obtain a predicted modality group; wherein the emotion-modality preference association model is obtained based on the interaction results of the interactive objects in a historical process.
[0046] The emotion-modal preference association model can be a mapping matrix or a neural network. The input to the model is the sensory ability assessment result and the emotional state. These results can be vectors. The model outputs a predicted modality set through internal computational logic. This set can be represented by the acceptance weights of each interaction modality (e.g., speech, text, image, animation). Modalities with acceptance weights higher than a preset weight are added to the predicted modality set. Higher weights indicate a greater preference for or easier acceptance of the interaction object in the current state. For example, if the input to the model is "{vision: 0.3, hearing: 0.8, emotion: depressed}", the model might output "{speech: 0.9, text: 0.4, animation: 0.1}", indicating that speech is the modality with the highest acceptance of the current interaction object.
[0047] Alternatively, in some application scenarios, the input to the emotion and modality preference association model includes emotional state, sensory ability assessment results, and historical preference records, and the output is the acceptance weight of each modality (speech / text / image / animation). The historical preference records can be the mapping relationship between multiple sets of sensory ability assessment results and emotional states and the predicted modality groups, statistically analyzed based on historical interaction processes.
[0048] Considering that the preferences of interactive objects may not be static, the sentiment and modality preference association model can be dynamically adjusted continuously during the interaction process. In the initial stage, the sentiment and modality preference association model can be built based on general public data. During the interaction process, the interactive behaviors and results of the interactive objects can be continuously collected to construct new samples and labels, iteratively updating the sentiment and modality preference association model to make it increasingly consistent with the usage preferences of specific interactive objects. The training samples consist of sentiment states, sensory ability assessment results, and predicted modality groups from historical interaction data. The training labels are the interactive object's acceptance of each modality (inferred through behavioral indicators such as response rate and completion rate). The mean squared error (MSE) loss function is used.
[0049] Interaction results can characterize the current sensory ability assessment results and the fit of the predicted modality group in the emotional state. For example, interaction results may include interaction completion rate, abandonment rate, and / or frequency of repeat requests.
[0050] For example, the system can record the interaction object's responses to predicted modalities under different emotional and sensory states, such as the smoothness of the interaction, the efficiency of task completion, and whether it actively switches to or ignores a certain modality. This interaction data is used to continuously train and fine-tune the model, enabling it to increasingly accurately grasp the individual preferences of the interaction object. For instance, the model can learn that an interaction object prefers a combination of voice and facial animation when happy, while preferring to see concise, large text and confirmation buttons when anxious.
[0051] By introducing a learnable emotion and modality preference association model, modality selection no longer depends on fixed rules, but has the ability to learn and adapt, and can evolve dynamically and in a personalized way, providing a tailored interactive experience for each interactive object.
[0052] In some embodiments, after controlling the companion robot to output interaction data with the interaction object according to at least one interaction modality in the predicted modality group, the method further includes: obtaining the interaction result of the interaction object regarding the interaction data, the interaction result including interaction completion rate, abandonment rate and / or repeat request frequency; and adjusting the emotion and modality preference association model according to the sensory ability assessment result, emotional state, predicted modality group and interaction result.
[0053] The interaction completion rate can be considered as the proportion of times an interactive task initiated by a companion robot is successfully completed within a specified time, out of the total number of times the interactive task is initiated. For example, based on the mapping relationship between the same sensory ability assessment results and the predicted modality group determined by emotional state, the ratio between the number of completed interactive tasks initiated according to this mapping relationship and the total number of interactive tasks initiated according to this mapping relationship within a certain period is calculated. If the robot initiates 10 reminder tasks according to this mapping relationship within the statistical period and completes them 8 times, then the interaction completion rate is 80%.
[0054] Abandonment rate can be considered as the proportion of times an interactive object actively interrupts, remains unresponsive for an extended period, or explicitly refuses to continue interacting during the interaction process, out of the total number of interactions. For example, regarding the mapping relationship between the same sensory ability assessment results and the predicted modality group determined by the same emotional state, it can be the ratio between the number of abandoned interactive tasks initiated based on this mapping relationship within a certain time period and the total number of interactive tasks initiated based on this mapping relationship.
[0055] Repeat request frequency can be considered as the average number of times an interactive object repeatedly issues requests such as "please say it again" or "I didn't understand" within the same interactive task. For example, regarding the mapping relationship between the same sensory ability assessment results and the predicted modality group determined by the emotional state, it is the average number of repeat requests for each interactive task initiated based on this mapping relationship within a certain time period.
[0056] The above-mentioned method of adjusting the emotion and modality preference association model based on sensory ability assessment results, emotional state, predicted modality group, and interaction results can be as follows: using sensory ability assessment results, emotional state, and predicted modality group as samples, determining the acceptability of each interaction modality in the predicted modality group based on the interaction results, and then determining the label corresponding to the sample based on each acceptability.
[0057] In this embodiment, by continuously updating the emotion and modality preference association model based on the feedback from the interactive object, the emotion and modality preference association model can provide a customized interactive experience for each interactive object.
[0058] In some embodiments, obtaining sensory ability assessment results of interactive objects based on behavioral data includes: The response data of the companion robot's interactive voice output from the behavioral data is evaluated to obtain hearing assessment results. This response data may include, but is not limited to, response delays and / or request repetition frequencies for different voice volumes. Hearing assessment results are obtained based on the response delays and / or request repetition frequencies for each voice volume. Furthermore, hearing sensitivity curves for different voice volumes can be constructed based on the hearing assessment results.
[0059] The visual acuity assessment results are obtained by evaluating the response data of the interactive screen output by the companion robot in the behavioral data. The response data may include, but is not limited to, the viewing distance, squinting frequency, and / or reading speed. The visual acuity assessment results may be the reading difficulty for different text sizes, or they may be visual acuity assessment values.
[0060] The response data of the interactive screen output by the companion robot in the behavioral data is evaluated to obtain the touch capability evaluation result. The response data of the interactive screen output by the companion robot may include, but is not limited to, click accuracy, swipe stability and / or operation completion time. The evaluation result of touch capability can be the touch difficulty for interactive components of different sizes or display positions.
[0061] The speech output by the interactive objects in the behavioral data is evaluated to obtain the language expression ability assessment results; the language expression ability assessment results include the recognition of the current speech clarity, dialect concentration, and expression coherence.
[0062] Sensory ability assessment results are obtained by combining hearing assessment results, visual assessment results, touch ability assessment results, and language expression ability assessment results. For example, the sensory ability assessment results are obtained by combining hearing assessment results, visual assessment results, touch ability assessment results, and language expression ability assessment results.
[0063] In some embodiments, step 104 described above may include, for example: Figure 2 The following steps are shown: Step 201: Select one interaction mode from the various interaction modes as the primary mode and the remaining interaction modes as auxiliary modes.
[0064] A predictive modality group may contain multiple interaction modalities. To prioritize information delivery and prevent information overload, a primary modality (or primary channel) can be determined based on one or more preset rules. All other interaction modalities in the predictive modality group besides the primary modality are auxiliary modalities (or auxiliary channels). For example, the primary modality can be selected based on the modality with the highest acceptance rate among current interaction objects.
[0065] For example, based on sensory ability assessments, speech might be chosen as the primary modality when the user has poor eyesight, while large-font text might be selected as the primary modality when hearing is impaired by a noisy environment. The primary modality can carry core information. The remaining interaction modalities serve as auxiliary modalities, providing supplementary, explanatory, or enhancing information.
[0066] Step 202: Generate the first interaction data of the main modality, and generate the second interaction data of the auxiliary modality based on the first interaction data of the main modality.
[0067] After determining the primary and secondary modalities, the core information of the primary modality, namely the first interaction data, is generated first. Then, based on the content of the first interaction data, the second interaction data of the secondary modality is generated.
[0068] This design ensures a clear hierarchy of information delivery, preventing information overload. For example, the primary modality (voice) outputs the first interactive data as "the temperature will drop by 5 degrees Celsius tomorrow," while the secondary modality (image) generates the second interactive data based on this core information: a clear temperature trend chart and an image providing clothing suggestions as supplementary information. It is evident that the secondary modality's interactive data is generated based on the primary modality's interactive data, rather than being independent of the primary modality's interactive data and unrelated to the first interactive data.
[0069] In some embodiments, the method of generating the first interaction data of the main modality described above may include, for example: Figure 3 The following steps are shown: Step 301: Determine whether there is any sub-interaction data to be output.
[0070] Before generating new initial interaction data, it is possible to check whether there are any unoutputted sub-interaction data from historical interactions that have been split and are yet to be output. Specifically, a queue is maintained to store unoutputted sub-interaction data that has been split due to excessive information volume. Step 301 can be implemented by first checking whether this queue is empty before generating new interaction data.
[0071] If it does not exist, proceed to step 302; if it exists, proceed to step 305.
[0072] Step 302: Generate initial interaction data based on historical dialogue data.
[0073] If the output queue is empty, it indicates the start of a new interaction. A large language model or generative model can be used to generate complete, unsplit initial interaction data based on the input of the current interaction and the historical dialogue context. Historical dialogue data refers to dialogue data over a specific period, such as a day, a month, or a year. The initial interaction data can be in the form of text or an image sequence; this example uses text as the initial interaction data. If the interaction is initiated by the companion robot, the initial interaction data can be directly generated using historical dialogue data.
[0074] Step 303: If the number of key points in the initial interaction data exceeds the preset number, the initial interaction data is split into multiple sub-interaction data.
[0075] The key points in the initial interaction data can be determined by splitting the initial interaction data into multiple segments, then clustering these segments, and using the number of clusters as the number of key points. The preset number can be determined based on the interaction object's responses in historical dialogue data. For example, if it is determined that the interaction object can answer n questions in a single interaction, the preset number can be less than or equal to n. For instance, the preset number could be 3.
[0076] Step 304: Obtain the first interaction data based on one of the sub-interaction data, and take each of the unselected sub-interaction data from the multiple sub-interaction data as the sub-interaction data to be output.
[0077] The method for obtaining the first interaction data from one of the sub-interaction data can be as follows: select the first sub-interaction data from the multiple sub-interaction data sets as the first interaction data to be output this time. Store the remaining sub-interaction data in the output queue, to be output in subsequent interaction rounds. This achieves the decomposition of complex information into a multi-round, progressive presentation.
[0078] Alternatively, consider that the modality of the selected sub-interaction data may not be consistent with the main modality of this round of interaction. The way to obtain the first interaction data based on one of the sub-interaction data is to convert the sub-interaction data into the presentation form corresponding to the main modality to obtain the first interaction data. If the modality of the selected sub-interaction data is consistent with the main modality of this round of interaction, then the selected sub-interaction data can be directly used as the first interaction data.
[0079] Step 305: Select one of the several sub-interaction data to be output as candidate interaction data.
[0080] Step 305 can be implemented by selecting the first one as the candidate interaction data according to the order of the sub-interaction data to be output in the output queue.
[0081] Step 306: If the interaction modality of the candidate interaction data is consistent with the main modality, the candidate interaction data is used as the first interaction data; if the interaction modality of the candidate interaction data is inconsistent with the main modality, the candidate interaction data is converted into the presentation format corresponding to the main modality to obtain the first interaction data.
[0082] In this embodiment, the cognitive load control mechanism for information output described above can present complex problems or information in stages, effectively reducing the cognitive pressure on the interactive object caused by excessive information volume at one time. This conforms to the information processing characteristics of the interactive object and can ensure the effective transmission and absorption of information.
[0083] Step 203: Control the companion robot to output the first interaction data and the second interaction data.
[0084] The control robot will generate first and second interaction data and present them to the interaction object simultaneously or in a certain sequence through corresponding output devices (such as speakers and displays).
[0085] In this embodiment, by dividing and adapting the primary and secondary channels, information can be effectively organized and presented, enabling the interactive object to first capture the core information and at the same time deepen its understanding through auxiliary information. This reduces the cognitive load caused by the parallel output of multi-channel information, making the interaction process clearer and more orderly.
[0086] In some embodiments, considering that fixed interaction parameters are not conducive to information transmission during the interaction process, the interaction parameters of the interaction data can be adjusted in this solution. Step 203 above may include the following steps: First, obtain the first parameter configuration information of the primary mode and the second parameter configuration information of the auxiliary mode.
[0087] The primary modality's first parameter configuration information and the secondary modality's second parameter configuration information are related to the sensory ability assessment results and emotional state of the interactive object. For example, based on the interactive object's sensory ability assessment results and emotional state, the parameter configuration information for the primary and secondary modalities in this interaction is determined. For example, interaction parameters for the speech modality may include, but are not limited to, at least one of the following: speech rate, volume, and pitch; interaction parameters for the text modality may include, but are not limited to, at least one of the following: font size, font type, and color contrast; interaction parameters for the animation modality may include playback rate. Interaction parameters for the image modality may include, but are not limited to, at least one of the following: image hue and contrast.
[0088] For example, if the interactive object is assessed to have poor hearing, the first parameter configuration information will instruct to increase the volume and slow down the speech rate (e.g., adjust the speech rate within ±30% of the standard speech rate). If the interactive object is assessed to have weakened vision, the second parameter configuration information will instruct to use a larger font size and higher color contrast.
[0089] Then, based on the first parameter configuration information, the interaction parameters of the first interactive data are adjusted; and based on the second parameter configuration information, the interaction parameters of the second interactive data are adjusted.
[0090] In this embodiment, not only was a modality suitable for the current state of the interactive object selected, but the presentation details of each modality were further optimized, achieving deep adaptation to the specific physiological decay of the interactive object. This allows the information to be received by the interactive object to the greatest extent possible, greatly improving the effectiveness and comfort of the interaction.
[0091] In some embodiments, considering that interactive objects from different ethnic groups or regions may have accents or dialects, this embodiment may choose to convert the output speech into colloquial or dialectal forms to improve the interactive object's understanding and acceptance of the speech output by the companion robot.
[0092] That is, if the modality of the first or second interactive data is speech, step 203 above may include the following steps: performing dialect adaptation processing on the interactive data in the first and second interactive data that is speech to obtain target interactive data; controlling the companion robot to output the target interactive data, as well as the interactive data in the first and second interactive data that has not undergone dialect adaptation processing.
[0093] For example, based on the assessment results of the interactive object's language expression ability (e.g., recognizing that the interactive object has a strong regional dialect), or based on the language habits used by the interactive object in historical interactions, the output content in standard Mandarin can be adapted to a dialect or colloquial expression familiar to the interactive object. For instance, a speech synthesis model with a built-in common dialect of the interactive object can adapt "Set an alarm for you at 7 a.m. tomorrow" to a more colloquial "Okay, I've set an alarm for you at 7 a.m. tomorrow." The speech data generated after this processing is the target interactive data.
[0094] Ultimately, the companion robot will output target interaction data (voice) that has been processed to adapt to the local dialect, as well as other non-voice modal interaction data (such as screen text, images, etc.) that have not been processed in this way.
[0095] In this embodiment, by introducing dialects and colloquialisms, the inclusivity with the language habits of the interactive objects can be greatly enhanced, the communication barriers caused by language differences can be reduced, the interaction process can be made more intimate and natural, and the acceptance and willingness of the interactive objects can be further improved.
[0096] The multimodal human-computer interaction method for the elderly provided by this invention can perceive the sensory abilities and emotional states of the interaction object in real time and dynamically adjust the interaction mode without requiring any complex settings from the interaction object, thus lowering the threshold for use. At the same time, it takes into account the physiological abilities and psychological feelings of the interaction object, making human-computer interaction more natural, efficient and full of humanistic care, and improving the accuracy of information understanding and interaction satisfaction of the interaction object.
[0097] The following describes the multimodal human-computer interaction device for the elderly provided by the present invention. The multimodal human-computer interaction device for the elderly described below can be referred to in correspondence with the multimodal human-computer interaction method for the elderly described above. For example... Figure 4 As shown, the multimodal human-computer interaction device 400 for the elderly includes the following modules: Behavior data acquisition module 401 is used to acquire behavior data of interactive objects; The sensory ability assessment and emotional state confirmation module 402 is used to obtain the sensory ability assessment results and emotional state of the interactive object based on behavioral data. The sensory ability assessment results are the results of assessing at least one of hearing, vision, touch ability and language expression ability. The modality determination module 403 is used to obtain a predicted modality group based on the sensory ability assessment results and emotional state, wherein the predicted modality group includes at least one interaction modality; The interaction output module 404 is used to control the companion robot to output interaction data with the interaction object based on at least one interaction mode in the prediction mode group.
[0098] According to the present invention, a multimodal human-computer interaction device 400 for the elderly includes an interaction output module 404 that controls a companion robot to output interaction data with the interaction object based on at least one interaction mode in a predicted mode group, including: Choose one interaction mode from the various interaction modes as the primary mode and the rest as auxiliary modes; Generate the first interaction data of the main modality, and generate the second interaction data of the auxiliary modality based on the first interaction data of the main modality; Control the companion robot to output the first and second interaction data.
[0099] According to the present invention, a multimodal human-computer interaction device 400 for the elderly includes an interaction output module 404 that controls a companion robot to output first interaction data and second interaction data, comprising: Obtain the first parameter configuration information of the primary mode and the second parameter configuration information of the auxiliary mode; Based on the first parameter configuration information, adjust the interaction parameters of the first interactive data; and based on the second parameter configuration information, adjust the interaction parameters of the second interactive data.
[0100] According to the present invention, a multimodal human-computer interaction device 400 for the elderly includes an interaction output module 404 that generates first interaction data for the main modality, comprising: Determine if there is any sub-interaction data to be output; If it does not exist, generate initial interaction data based on historical dialogue data; If the number of key points in the initial interaction data exceeds the preset number, the initial interaction data is split into multiple sub-interaction data. The first interaction data is obtained based on one of the sub-interaction data, and each of the sub-interaction data that was not selected among the multiple sub-interaction data is taken as the sub-interaction data to be output.
[0101] According to the present invention, a multimodal human-computer interaction device 400 for the elderly, if the modality of the first interaction data or the second interaction data is voice, the interaction output module 404 controls the companion robot to output the first interaction data and the second interaction data, including: Dialect adaptation processing is performed on the speech modal of the first and second interactive data to obtain the target interactive data; The companion robot is controlled to output target interaction data, as well as interaction data from the first and second interaction data that have not been processed for dialect adaptation.
[0102] According to the present invention, a multimodal human-computer interaction device 400 for the elderly includes a modality determination module 403 that obtains a predicted modality set based on sensory ability assessment results and emotional state, comprising: The sensory ability assessment results and emotional state are processed using an emotion-modality preference association model to obtain a predictive modality group; Among them, the sentiment and modality preference association model is obtained based on the interaction results of interactive objects in the historical process.
[0103] According to the multimodal human-computer interaction device 400 for the elderly provided by the present invention, after controlling the companion robot to output interaction data with the interaction object based on at least one interaction mode in the predicted mode group, the mode determination module 403 is further configured to: Obtain the interaction results of the interactive object regarding the interaction data. The interaction results include the interaction completion rate, abandonment rate, and / or repeat request frequency. The emotion-modality preference association model was adjusted based on sensory ability assessment results, emotional state, predicted modality groups, and interaction results.
[0104] According to the present invention, a multimodal human-computer interaction device 400 for the elderly includes a sensory ability assessment and emotional state confirmation module 402 that obtains the sensory ability assessment results of the interactive object based on behavioral data, including: The response data of the interactive voice output by the companion robot in the behavioral data is evaluated to obtain the hearing assessment results; The visual assessment results are obtained by evaluating the response data of the interactive screen output by the companion robot in the behavioral data. The response data of the interactive screen output by the companion robot in the behavioral data is evaluated to obtain the touch control capability evaluation results; The speech output by the interactive objects in the behavioral data is evaluated to obtain the language expression ability assessment results; The sensory ability assessment results are obtained by combining the hearing assessment results, vision assessment results, touch ability assessment results, and language expression ability assessment results.
[0105] According to the present invention, a multimodal human-computer interaction device 400 for the elderly includes a behavior data acquisition module 401. The companion robot is equipped with several sensors to acquire behavior data of the interacting object, including: Acquire data collected by each sensor; Based on the data collected by each sensor, the behavioral data of the interactive object is obtained.
[0106] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a multimodal human-computer interaction method for the elderly. The method includes: acquiring behavioral data of the interaction object; acquiring sensory ability assessment results and emotional state of the interaction object based on the behavioral data, wherein the sensory ability assessment results are the result of assessing at least one of hearing, vision, touch ability, and language expression ability; obtaining a predictive modality group based on the sensory ability assessment results and emotional state, wherein the predictive modality group includes at least one interaction modality; and controlling the companion robot to output interaction data with the interaction object according to at least one interaction modality in the predictive modality group.
[0107] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0108] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal human-computer interaction method for the elderly provided by the above methods. The method includes: acquiring behavioral data of the interaction object; acquiring sensory ability assessment results and emotional state of the interaction object based on the behavioral data, wherein the sensory ability assessment results are the result of assessing at least one of hearing, vision, touch ability, and language expression ability; obtaining a predictive modality group based on the sensory ability assessment results and emotional state, wherein the predictive modality group includes at least one interaction modality; and controlling a companion robot to output interaction data with the interaction object according to at least one interaction modality in the predictive modality group.
[0109] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal human-computer interaction method for the elderly provided by the methods described above. The method includes: acquiring behavioral data of the interaction object; acquiring sensory ability assessment results and emotional state of the interaction object based on the behavioral data, wherein the sensory ability assessment results are the result of assessing at least one of hearing, vision, touch control ability, and language expression ability; obtaining a predictive modality set based on the sensory ability assessment results and emotional state, wherein the predictive modality set includes at least one interaction modality; and controlling a companion robot to output interaction data with the interaction object based on at least one interaction modality in the predictive modality set.
[0110] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multimodal human-computer interaction method for the elderly, characterized in that, include: Obtain behavioral data of the interacting object; The sensory ability assessment results and emotional state of the interactive object are obtained based on the behavioral data. The sensory ability assessment results are the result of assessing at least one of hearing, vision, touch ability and language expression ability. Based on the sensory ability assessment results and the emotional state, a predictive modality group is obtained, which includes at least one interaction modality; Based on at least one interaction mode in the predicted mode group, the companion robot is controlled to output interaction data with the interaction object.
2. The multimodal human-computer interaction method for the elderly according to claim 1, characterized in that, The step of controlling the companion robot to output interaction data with the interaction object based on at least one interaction mode in the predicted mode group includes: Select one interaction mode from the aforementioned interaction modes as the primary mode, and the remaining interaction modes as auxiliary modes; Generate first interaction data for the primary modality, and generate second interaction data for the auxiliary modality based on the first interaction data for the primary modality; The companion robot is controlled to output the first interaction data and the second interaction data.
3. The multimodal human-computer interaction method for the elderly according to claim 2, characterized in that, Controlling the companion robot to output the first interaction data and the second interaction data includes: Obtain the first parameter configuration information of the primary mode and the second parameter configuration information of the auxiliary mode; Based on the first parameter configuration information, adjust the interaction parameters of the first interactive data; and based on the second parameter configuration information, adjust the interaction parameters of the second interactive data.
4. The multimodal human-computer interaction method for the elderly according to claim 2, characterized in that, Generating the first interaction data of the main modality includes: Determine if there is any sub-interaction data to be output; If it does not exist, generate initial interaction data based on historical dialogue data; If the number of key points in the initial interaction data exceeds a preset number, the initial interaction data is split into multiple sub-interaction data. The first interaction data is obtained based on one of the sub-interaction data, and each of the sub-interaction data that was not selected among the multiple sub-interaction data is taken as the sub-interaction data to be output.
5. The multimodal human-computer interaction method for the elderly according to claim 2, characterized in that, If the modality of the first interaction data or the second interaction data is voice, control the companion robot to output the first interaction data and the second interaction data, including: Dialect adaptation processing is performed on the speech modal of the first and second interactive data to obtain the target interactive data; The companion robot is controlled to output the target interaction data, as well as the interaction data in the first interaction data and the second interaction data that has not been processed for dialect adaptation.
6. The multimodal human-computer interaction method for the elderly according to any one of claims 1 to 5, characterized in that, Based on the sensory ability assessment results and the emotional state, a predictive modality set is obtained, including: The sensory ability assessment results and the emotional state are processed using an emotion and modality preference association model to obtain the predicted modality group; The emotion and modality preference association model is obtained based on the interaction results of the interactive objects in the historical process.
7. The multimodal human-computer interaction method for the elderly according to claim 6, characterized in that, After controlling the companion robot to output interaction data with the interaction object based on at least one interaction mode in the predicted mode group, the method further includes: Obtain the interaction results of the interaction object with respect to the interaction data, wherein the interaction results include the interaction completion rate, abandonment rate, and / or the frequency of repeated requests; The emotion-modal preference association model is adjusted based on the sensory ability assessment results, the emotional state, the predicted modality group, and the interaction results.
8. The multimodal human-computer interaction method for the elderly according to any one of claims 1 to 5, characterized in that, The sensory ability assessment results of the interactive object are obtained based on the behavioral data, including: The response data of the interactive voice output by the companion robot in the behavioral data is evaluated to obtain the hearing assessment result; The response data of the interactive screen output by the companion robot in the behavioral data is evaluated to obtain the vision assessment result; The response data of the interactive screen output by the companion robot in the behavioral data is evaluated to obtain the touch control capability evaluation result; The speech output by the interactive object in the behavioral data is evaluated to obtain the language expression ability evaluation result; The sensory ability assessment result is obtained by combining the hearing assessment result, the vision assessment result, the touch ability assessment result, and the language expression ability assessment result.
9. The multimodal human-computer interaction method for the elderly according to claim 8, characterized in that, The companion robot is equipped with several sensors to acquire behavioral data of the interacting object, including: Acquire the data collected by each of the aforementioned sensors; The behavioral data of the interactive object is obtained based on the data collected by each of the sensors.
10. A multimodal human-computer interaction device for the elderly, characterized in that, include: The behavior data acquisition module is used to acquire the behavior data of the interactive objects; The sensory ability assessment and emotional state confirmation module is used to obtain the sensory ability assessment results and emotional state of the interactive object based on the behavioral data. The sensory ability assessment results are the results of assessing at least one of hearing, vision, touch ability and language expression ability. A modality determination module is used to obtain a predicted modality group based on the sensory ability assessment results and the emotional state, wherein the predicted modality group includes at least one interaction modality; An interaction output module is used to control the companion robot to output interaction data with the interaction object based on at least one interaction mode in the predicted mode group.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal human-computer interaction method for the elderly as described in any one of claims 1 to 9.
12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal human-computer interaction method for the elderly as described in any one of claims 1 to 9.
13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal human-computer interaction method for the elderly as described in any one of claims 1 to 9.
