A Smart Toy Control Method Based on Multimodal Perception

By dynamically analyzing the activity index and feature sequence of multimodal signals and adaptively updating the modal confidence and decision weights, the problem of interaction accuracy and fun caused by fixed multimodal fusion weights in intelligent toy control systems is solved. Personalized strategy optimization and emotional feedback are realized, thus improving the interaction effect.

CN123116137APending Publication Date: 2026-11-17YANGZHOU XINHONG GIFT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611280354.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-22
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing intelligent toy control systems struggle to dynamically adjust multimodal fusion weights based on the user's real-time status, resulting in reduced interaction accuracy and fun.

Method used

By acquiring multimodal signals from smart toys, analyzing the activity indicators and feature sequences of each modality, dynamically updating modality confidence and decision weights, and calculating the degree of emotion improvement using changes in emotion vectors in historical interactions, the system can automatically select the strategies that best enhance users' positive emotions.

Benefits of technology

It improves the accuracy and naturalness of multimodal interaction, enhancing the mood-enhancing effects and sustained appeal of smart toys.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116137A_ABST
    Figure CN123116137A_ABST
Patent Text Reader

Abstract

This invention provides a multimodal perception-based intelligent toy control method, belonging to the field of intelligent toy control technology. The method includes: synchronously acquiring multimodal signals during interaction, and parsing frame-by-frame to obtain the activity index and feature sequence of each modality; updating the confidence level of each modality based on the activity index, feature sequence, and preset threshold to determine the current confidence level of each modality and performing normalization processing to obtain the decision weight of each modality; selecting candidate combination strategies based on the decision weights, calculating the emotion improvement degree using the dot product of the difference in emotion vectors before and after execution in historical interactions and the positive emotion baseline vector, using the product of the emotion improvement degree and the strategy weight as the evaluation value, and selecting the strategy with the highest evaluation value to convert into a multimodal output control command. This invention solves the problems of fixed-weight fusion and lack of emotion effect evaluation in strategy selection, improving the accuracy and emotion improvement capability of intelligent toy interaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent toy control, and in particular to an intelligent toy control method based on multimodal perception. Background Technology

[0002] With the development of sensor technology and artificial intelligence, smart toys are gradually acquiring multimodal perception capabilities such as voice recognition, touch sensing, and visual tracking, enabling them to interact with users in a wider range of ways. Most current smart toy control systems adopt multimodal fusion methods with preset rules or fixed weights, directly mapping the collected voice, touch, and visual signals to fixed actions or feedback, such as making a corresponding action when a touch is detected.

[0003] However, during the interaction process, the user's attention state, willingness to interact, and sensory activity level are dynamically changing. Some modalities may experience reduced input quality due to user fatigue, attention shift, or environmental interference. Existing methods are difficult to adaptively adjust the reliability of different modalities according to real-time conditions, and fixed fusion weights reduce the accuracy and fun of multimodal interaction. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent toy control method based on multimodal perception to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for controlling intelligent toys based on multimodal perception includes the following steps:

[0007] The system acquires multimodal signals of the smart toy during the current interaction period, and performs frame-by-frame analysis of the multimodal signals to extract the activity index and feature sequence of each modality.

[0008] The activity index of each modality is compared with the corresponding preset threshold. The confidence of each modality is updated based on the comparison results and feature sequences to determine the current confidence of each modality and perform normalization processing to obtain the decision weight of each modality.

[0009] Based on the decision weights of each modality, candidate combination strategies that meet the matching conditions are selected from the pre-set combination strategy library and their strategy weights are determined. For each candidate combination strategy, the emotion vectors detected before and after execution in past interactions are obtained, the difference between the two emotion vectors is calculated, and the dot product operation is performed on the difference with the preset positive emotion baseline vector to obtain the emotion improvement degree.

[0010] The product of the emotion improvement of each candidate combination strategy and its corresponding strategy weight is used as the evaluation value of the candidate combination strategy. The candidate combination strategy with the largest evaluation value is selected as the target combination strategy. The target combination strategy is converted into multimodal output parameters and control instructions are generated.

[0011] Furthermore, the multimodal signal includes at least a speech signal from the speech modality, a touch signal from the touch modality, and a visual signal from the visual modality. The activity index of the speech modality is the short-time energy mean of the speech signal, the activity index of the touch modality is the pressure peak of the touch signal, and the activity index of the visual modality is the inter-frame difference energy of the visual signal. Each frame element in the feature sequence of the speech modality includes at least two evaluation dimensions: frame-by-frame fundamental frequency and frame-by-frame short-time energy. Each frame element in the feature sequence of the touch modality includes at least two evaluation dimensions: frame-by-frame touch pressure and frame-by-frame contact area ratio. Each frame element in the feature sequence of the visual modality includes at least three evaluation dimensions: frame-by-frame head yaw angle, frame-by-frame angle between the line of sight and the center of the toy, and frame-by-frame proportion of the face in the field of vision.

[0012] Furthermore, obtaining the decision weights for each modality includes the following steps:

[0013] Set an initial confidence level for each modality. The current confidence level of any modality is its confidence level in the current interaction period, and the initial value of the current confidence level of that modality is equal to the initial confidence level.

[0014] For any modality, its activity index is compared with the corresponding preset threshold. If its activity index is greater than the corresponding preset threshold, the modality is marked as a high-activity modality; otherwise, the modality is marked as a normal modality.

[0015] For any modality, its feature sequence is divided into a first subsequence and a second subsequence according to a preset ratio in chronological order. When the feature sequence of the modality contains multiple evaluation dimensions, the first and second subsequence dispersions are calculated for each evaluation dimension. The arithmetic mean of the first subsequence dispersions of all evaluation dimensions of the modality is then used as the first subsequence dispersion of the modality. The arithmetic mean of the second subsequence dispersions of all evaluation dimensions of the modality is then used as the second subsequence dispersion of the modality. In a single evaluation dimension, the absolute values ​​of the scalar differences of the element values ​​of the evaluation dimension in adjacent frames are summed. The sum of the absolute values ​​of the first subsequences is divided by the number of frames in the first subsequence to obtain the first subsequence dispersion. The sum of the absolute values ​​of the second subsequences is divided by the number of frames in the second subsequence to obtain the second subsequence dispersion.

[0016] During the current interaction period, if a mode is a highly active mode and the dispersion of the latter part of the mode is less than that of the former part, then the current confidence of the mode is set to the value of its confidence in the previous interaction period after decay calculation; otherwise, the current confidence of the mode is set to its confidence in the previous interaction period.

[0017] The ratio of the current confidence level of each modality to the sum of the current confidence levels of all modalities is used as the decision weight of each modality.

[0018] Furthermore, the logic for determining the preset threshold is as follows: extract multiple consecutive historical interaction periods from the historical time period, obtain the activity index of each modality in each historical interaction period, arrange the activity indexes from smallest to largest to form an activity index sequence, and use the upper quartile of the activity index sequence as the preset threshold of the corresponding activity index.

[0019] Furthermore, for any modality, the confidence level in the previous interaction period is decayed to determine its current confidence level. The logic is as follows:

[0020] Multiple consecutive historical interaction periods are extracted from the historical period. The activity index of the modality in each historical interaction period is obtained and the average value is calculated. If the average value is greater than zero, the ratio of the activity index of the modality in the current interaction period to the average value is taken as the relative activity level of the modality. Otherwise, the relative activity level of the modality is set to 1.

[0021] The ratio of the dispersion of the first segment to the dispersion of the second segment of the mode is taken as the dispersion change ratio of the mode.

[0022] The relative activity level of the mode is multiplied by the dispersion change ratio to obtain its comprehensive fatigue index. The confidence level of the mode in the previous interaction period is divided by the comprehensive fatigue index to obtain the current confidence level of the mode.

[0023] Furthermore, for any modality, when determining the current confidence level based on the confidence level during the previous interaction period, the following steps are also included:

[0024] For any modality, retrieve the dormancy time of that modality in multiple consecutive historical interaction periods, wherein the dormancy time is defined as the number of interaction periods in which the activity index of that modality is continuously lower than or equal to the corresponding preset threshold, starting from the first normal modality interaction period after the current confidence of that modality first falls below the initial confidence.

[0025] Arrange the periods of silence in order of length to form a period of silence time series, and calculate the mean of the period of silence time series;

[0026] If the current confidence level of the current mode is lower than the initial confidence level, and the quiescent time of the current mode is greater than or equal to the mean of the quiescent time series, then the current confidence level of the current mode is adjusted according to... The function gradually recovers from the current value to the initial value with the interaction period as the step size; otherwise, the current confidence of the modality is set to its confidence in the previous interaction period.

[0027] Furthermore, the matching condition is that the sum of the decision weights of each modality involved in the candidate combination strategy is greater than or equal to a preset matching threshold, wherein the matching threshold is the product of the sum of the decision weights of each modality and a first attenuation factor, and the first attenuation factor is a positive number less than 1.

[0028] Furthermore, the combined strategy library contains a variety of preset strategies composed of different modal response modes. Each candidate combined strategy is associated with a corresponding strategy weight. For any candidate combined strategy, its strategy weight is the geometric mean of the decision weights of each modality involved.

[0029] Furthermore, when the past interaction records of a candidate combination strategy are empty, the strategy weights associated with the candidate combination strategy and the decision weights of each modality involved in the candidate combination strategy are extracted. The standard deviation of the decision weights of each modality involved in the candidate combination strategy is calculated. If the standard deviation is greater than zero, the product of the standard deviation and the strategy weight is used as the emotional improvement degree of the candidate combination strategy. If the standard deviation is equal to zero, the product of the strategy weight and the second decay factor is directly used as the emotional improvement degree of the candidate combination strategy. The second decay factor is a positive number less than 1.

[0030] Furthermore, the emotion vector includes a voice emotion value, a touch response value, and a visual attention value. The voice emotion value is obtained by extracting the fundamental frequency mean, average speech rate, and short-time energy envelope from the voice signal, normalizing them, and then calculating the arithmetic mean. The touch response value is obtained by extracting the average pressure, touch frequency, and effective contact area percentage from the touch signal, normalizing them, and then calculating the arithmetic mean. The visual attention value is obtained by extracting the head orientation yaw angle mean, gaze duration percentage, and body horizontal distance from the visual signal, normalizing them, and then calculating the arithmetic mean.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] By dynamically analyzing the activity index and feature sequence of multimodal signals, the confidence of each modality is adaptively updated and decision weights are generated, avoiding the interaction inaccuracy problem caused by fixed weight fusion. At the same time, the degree of emotion improvement is calculated by using the change of emotion vector in historical interactions, and the target strategy that can improve the user's positive emotions is automatically selected from the candidate combination strategies by combining strategy weights. This can enhance the accuracy and naturalness of multimodal interaction, realize personalized strategy optimization based on emotion feedback, and significantly improve the emotion improvement effect and sustained appeal of smart toys. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of a smart toy control method based on multimodal perception according to the present invention;

[0035] Figure 2 This is a schematic diagram illustrating the process of obtaining decision weights for each modality in this invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] The following is combined Figures 1-2 This invention describes an intelligent toy control method based on multimodal perception.

[0038] Example:

[0039] Please see Figure 1 and Figure 2 This invention provides a smart toy control method based on multimodal perception, comprising the following steps:

[0040] Step S100: Obtain the multimodal signals of the smart toy during the current interaction period, and analyze the multimodal signals frame by frame to extract the activity index and feature sequence of each modality.

[0041] Specifically, the multimodal signal includes at least a speech signal from the speech modality, a touch signal from the touch modality, and a visual signal from the visual modality. The activity index of the speech modality is the short-time energy average of the speech signal, the activity index of the touch modality is the pressure peak value of the touch signal, and the activity index of the visual modality is the inter-frame differential energy of the visual signal.

[0042] The formula for calculating the short-time energy of a single frame is as follows:

[0043]

[0044] in, The short-time energy value of the speech signal in the nth frame is the higher the overall energy of the speech signal in that frame, that is, the louder the sound or the more obvious the speech activity. The amplitude of the m-th sampling point in the n-th frame of the audio signal is denoted as N. The larger the amplitude, the higher the sound pressure level picked up by the microphone at that moment, and the louder the user's voice. N is the number of sampling points corresponding to the frame length.

[0045] After the frame-by-frame calculation is completed, the short-time energy of each frame is accumulated and then divided by the total number of frames in the current interaction period. The result is the average short-time energy.

[0046] The touch signal collected by the touch sensor array during the current interaction period is acquired. The touch sensor array consists of multiple pressure sensing units. Each sensing unit outputs a pressure value at a fixed sampling frequency. After analog-to-digital conversion, the processor caches all pressure sampling values ​​according to the interaction period. The processor iterates through all sampling data of all pressure sensing units during the current interaction period and selects the maximum value by comparing them one by one. This maximum value is the pressure peak value of the touch signal.

[0047] The visual signal captured by the camera during the current interaction period is acquired. The visual signal is a continuous sequence of image frames. After analog-to-digital conversion, it is cached by the digital signal processor according to the interaction period. The processor performs inter-frame difference operation on each pair of adjacent frames in the cached image sequence. That is, the gray value of each pixel in the next frame is subtracted from the gray value of the corresponding pixel in the previous frame and the absolute value is taken to obtain the difference image of each pair of adjacent frames. Then, the sum of the gray values ​​of all pixels in all difference images is calculated and divided by the total number of pixels. The average value obtained is the inter-frame difference energy of the visual signal.

[0048] Specifically, each frame element in the feature sequence of the speech modality includes at least two evaluation dimensions: frame-by-frame fundamental frequency and frame-by-frame short-time energy. Each frame element in the feature sequence of the touch modality includes at least two evaluation dimensions: frame-by-frame touch pressure and frame-by-frame contact area ratio. Each frame element in the feature sequence of the visual modality includes at least three evaluation dimensions: frame-by-frame head yaw angle, frame-by-frame angle between the line of sight and the center of the toy, and frame-by-frame proportion of the face in the field of vision.

[0049] The audio signal captured by the microphone during the current interaction period is acquired. After analog-to-digital conversion, the digital signal processor divides the audio signal into frames with a frame length of 40 milliseconds and a frame shift of 20 milliseconds. The fundamental frequency of each frame of audio signal is extracted using the autocorrelation function method. The processor calculates the autocorrelation function sequence of the current frame, searches for the position of the maximum peak of the autocorrelation function except for zero delay, and converts the delay corresponding to the position into the pitch period. The reciprocal of the pitch period is the fundamental frequency of the frame.

[0050] The expression for the single-frame autocorrelation function is:

[0051]

[0052] in, It is a similarity measure between the nth frame of speech signal and its own signal after a delay of τ sampling points. The larger the similarity measure, the higher the degree of overlap between the speech signal and its own delayed waveform under the delay amount τ, that is, the stronger the periodicity or self-similarity of the signal at the delay interval. This represents the delay amount.

[0053] It should also be noted that the lower limit of the delay range for the autocorrelation function search corresponds to a fundamental frequency of approximately 50Hz, and the upper limit corresponds to a fundamental frequency of approximately 400Hz, covering the general fundamental frequency range of human speech. If the current frame is an unvoiced frame, the autocorrelation function has no obvious peak value. In this case, the fundamental frequency of the frame can be set to zero or the fundamental frequency value of the previous frame can be used.

[0054] The processor divides the current interaction period into several touch frames with a touch frame length of 30 to 50 milliseconds. It calculates the arithmetic mean of the pressure values ​​of all pressure sensing units in each touch frame to obtain the frame-by-frame touch pressure. At the same time, it counts the number of sensing units whose pressure values ​​exceed the contact determination threshold in each touch frame. It divides this number by the total number of sensing units to obtain the frame-by-frame contact area percentage of the touch frame. The contact determination threshold is selected within 3% to 5% of the maximum range of the pressure sensor.

[0055] For each frame of the image, the facial landmark detection model is called to output the two-dimensional coordinates of the facial landmarks in the image coordinate system, as well as the coordinates of the center of the pupils of both eyes.

[0056] First, the tip of the nose, the center of the left eye, and the center of the right eye are selected as reference key points. Using the image coordinates of these three points and a preset standard 3D human head model, the head posture rotation vector of the current frame is obtained by solving the perspective n-point localization problem. The rotation vector is converted into Euler angles, and the yaw angle component around the vertical axis is extracted as the head yaw angle for each frame. A head yaw angle of zero degrees indicates that the user is facing the toy, a positive value indicates that the user is turning their head to the left, and a negative value indicates that the user is turning their head to the right.

[0057] Secondly, the midpoint of the line of sight is calculated based on the coordinates of the center of the pupils of both eyes. Taking the midpoint of the line of sight as the starting point and the ray along the head posture direction as the estimated line of sight direction, the angle between the estimated line of sight direction and the direction from the midpoint of the line of sight to the optical center of the camera, i.e. the center of the toy, is calculated. The angle between the line of sight and the center of the toy is obtained frame by frame. The smaller the angle between the line of sight and the center of the toy, the more focused the user's attention is on the toy.

[0058] Next, the ratio of the face bounding box area to the total area of ​​the current frame image is calculated to obtain the proportion of the face in the field of view for each frame. The proportion of the face in the field of view reflects the relative distance between the user and the toy. The larger the proportion, the closer the user is to the toy.

[0059] It should also be noted that: the facial landmark detection model can use a pre-trained convolutional neural network model or a cascaded regression model; in solving the perspective n-point localization problem, the intrinsic parameter matrix of the camera needs to be known, and the intrinsic parameter matrix is ​​obtained in advance through the chessboard calibration method.

[0060] Step S200: The activity index of each modality is compared with the corresponding preset threshold. The confidence of each modality is updated based on the comparison results and feature sequences to determine the current confidence of each modality and perform normalization processing to obtain the decision weight of each modality.

[0061] Specifically, obtaining the decision weights for each modality includes the following steps:

[0062] Step S210: Set the initial confidence level for each modality. The current confidence level of any modality is its confidence level in the current interaction period, and the initial value of the current confidence level of the modality is equal to the initial confidence level.

[0063] After the toy is first started or after a factory reset, the processor reads the preset initial confidence parameters from the non-volatile memory and writes the same initial confidence value for the voice modality, touch modality and visual modality. The initial confidence can be set to 0.7 to 0.9, and can be adjusted according to the behavioral characteristics of the toy's target user group.

[0064] Step S220: For any modality, compare its activity index with the corresponding preset threshold. If its activity index is greater than the corresponding preset threshold, then mark the modality as a high-activity modality; otherwise, mark the modality as a normal modality.

[0065] The logic for determining the preset threshold is as follows: extract multiple consecutive historical interaction periods from the historical time period, obtain the activity index of each modality in each historical interaction period, arrange the activity indexes from smallest to largest to form an activity index sequence, and use the upper quartile of the activity index sequence as the preset threshold of the corresponding activity index.

[0066] By comparing the activity index with a preset threshold, the modality is divided into a highly active modality and a normal modality, which makes it easier to identify the modality with the most prominent intensity in the current interaction.

[0067] Step S230: For each modality, the feature sequence corresponding to the modality is divided into a front subsequence and a back subsequence in a preset ratio of 1:1 according to the time sequence. When the feature sequence of the modality contains multiple evaluation dimensions, the front-segment dispersion and the back-segment dispersion are calculated for each evaluation dimension. Then, the arithmetic mean of the front-segment dispersion of all evaluation dimensions of the modality is taken as the front-segment dispersion of the modality. The arithmetic mean of the back-segment dispersion of all evaluation dimensions of the modality is taken as the back-segment dispersion of the modality. In a single evaluation dimension, the absolute values ​​of the scalar differences of the element values ​​of the evaluation dimension in adjacent frames are summed. The sum of the absolute values ​​of the front subsequences is divided by the number of frames in the front subsequence to obtain the front-segment dispersion. The sum of the absolute values ​​of the back subsequences is divided by the number of frames in the back subsequence to obtain the back-segment dispersion.

[0068] It should be noted that when the total number of frames in the feature sequence of this modality is odd, the frame located in the middle position is included in the first subsequence; when the feature sequence of this modality is a single evaluation dimension, the first and second segment dispersions are obtained directly according to the calculation method of a single evaluation dimension.

[0069] By comparing whether the activity index exceeds a preset threshold, it is identified that the user is using the modality intensively. However, high intensity does not directly equate to high-quality interaction. If the visual modality is judged to be very reliable and given high weight solely based on this, the toy may respond too actively to the user's blank stare, which may disturb the user and create negative feedback. Therefore, dispersion is introduced for comprehensive judgment.

[0070] Step S240: During the current interaction period, if a mode is a highly active mode and the dispersion of the latter part of the mode is less than the dispersion of the former part, then the current confidence of the mode is set to the value of its confidence in the previous interaction period after decay calculation; otherwise, the current confidence of the mode is set to its confidence in the previous interaction period.

[0071] When the modality is highly active and the dispersion of the latter part is smaller than that of the former, it indicates that the user's behavioral performance begins to decline after continuous high-intensity interaction. Although the input signal is strong, it lacks variation and richness, reflecting user fatigue, inattention, or decreased interest. If the modality signal is adopted with the original confidence level, it will lead to a decrease in the overall interaction experience. Therefore, the current confidence level of the modality is attenuated to reduce its proportion in the subsequent decision weight calculation, so that the system's control output relies more on other more reliable modalities, thereby maintaining the accuracy of multimodal fusion and the naturalness of the interaction.

[0072] When the mode is a highly active mode and the dispersion of the latter part is greater than or equal to the dispersion of the former part, it indicates that the user's engagement in the interaction of this mode is still high, the signal quality is good, and there are no obvious signs of fatigue or interest decay. Since the increase in dispersion can be due to both positive situations (e.g., the user comes out of a daze and starts to interact actively, making the interaction richer) and negative situations (e.g., the user moves or presses randomly due to irritation; or environmental noise or interference from others causes signal confusion), if the signal confusion is rashly judged as an improvement in interaction quality and rewarded, it will lead to a decrease in the system's robustness to interference and make incorrect responses. Therefore, the current confidence level should be kept unchanged.

[0073] When the modality is a normal modality and the dispersion of the latter part is less than that of the former part, it indicates that the user's interaction intensity in this modality is at a normal or low level. In this case, keeping the current confidence level unchanged can avoid misjudging and inappropriately punishing the user's normal low-intensity interaction behavior.

[0074] When the modality is the normal modality and the dispersion of the latter part is greater than or equal to that of the former part, it indicates that the user is in the transition period of attention regression, or unconscious, low-intensity habitual actions, or the intervention of external momentary interference. At this time, the current confidence level is kept unchanged to avoid misjudging unconscious random fluctuations or external interference as reliable interaction intentions.

[0075] For any modality, the logic for determining its current confidence level by calculating the decay of its confidence level in the previous interaction period is as follows:

[0076] Step S241: Extract multiple consecutive historical interaction periods from the historical time period, obtain the activity index of the modality in each historical interaction period and calculate the average value. If the average value is greater than zero, the ratio of the activity index of the modality in the current interaction period to the average value is taken as the relative activity level of the modality. Otherwise, the relative activity level of the modality is set to 1.

[0077] The historical average of the activity metric needs to reflect the user's recent normal level of interaction. The further back in time the historical data is from the current interaction, the more likely it is to lose its reference value due to user growth, changes in emotional state, or changes in the usage environment. Therefore, we need to trace back from the current interaction time and extract several consecutive historical interaction time periods that are immediately adjacent to the current interaction time.

[0078] If the average value is zero, it means that the modality was previously in a completely unused dormant state. Therefore, the relative activity level is directly set to 1.

[0079] The relative level of activity directly reflects the intensity of user interaction during the current interaction period. The deviation from its own historical average performance ensures that the criteria for judging high intensity vary from person to person. The same toy can make completely different and appropriate fatigue judgments based on their respective baselines when facing a user who is usually very active and a user who is usually very quiet.

[0080] Step S242: The ratio of the dispersion of the first segment to the dispersion of the second segment of the mode is taken as the dispersion change ratio of the mode.

[0081] The dispersion change ratio reflects the changing trend of the feature sequence within the current interaction period. The smaller the dispersion change ratio, the smaller the difference between the dispersion of the first segment and the dispersion of the second segment. The larger the dispersion change ratio, the larger the dispersion of the first segment relative to the dispersion of the second segment.

[0082] When the dispersion change ratio of a modality is greater than 1, that is, when the dispersion of the latter part is less than that of the former part, it indicates that the variation amplitude of the feature sequence in the latter part is significantly narrower than that in the former part. The interactive behavior tends to become simple and stable from rich and varied. Under the premise of a highly active modality, the user still uses the modality in a relatively large and rich way in the first half, but in the second half, the behavioral variation weakens and the input tends to be repetitive or fixed. This is a typical manifestation of fatigue or interest decay: the user is still interacting, but the quality of input has decreased. Although the strength of the output signal is still there, the information content and variability have been reduced. Therefore, a dispersion change ratio greater than 1 provides trend evidence for fatigue judgment. Without this evidence, even if the activity level is very high, it can only indicate that the user's interaction intensity is high, but it cannot indicate that the user is fatigued.

[0083] When the dispersion change ratio of this mode is less than 1, it indicates that the dispersion of the latter part is greater than that of the former part, and the change of the feature sequence in the latter half is actually intensified. At this time, there are two possibilities: one is that the user interaction is upgraded and the behavior becomes richer and more engaged, such as changing from a single gaze to saccades or turning the head, and changing from a flat tone to a high-pitched and low-pitched tone; the other is that the signal becomes chaotic due to the user's emotional fluctuations or environmental interference, such as the intervention of external noise or the user's restlessness and fidgeting. Regardless of which one it is, the system lacks a reliable basis to judge it as fatigue. Therefore, attenuation calculation is not performed in this case.

[0084] When the dispersion change ratio of this modality is equal to 1, it means that the dispersion of the first segment is equal to that of the second segment, and the variation amplitude of the feature sequence remains consistent throughout the entire interaction period. There is neither monotonicity nor enrichment. In this case, the user behavior pattern is stable. If this modality is a highly active modality, it means that the user has been interacting with a high intensity and a relatively stable variation amplitude. This is closer to a continuous and healthy state of engagement than a state of fatigue. The characteristic of fatigue is that the behavior pattern slides from change to no change, rather than maintaining the same level of change. Therefore, attenuation calculation is not performed at this time.

[0085] Step S243: Multiply the relative activity level of the mode by the dispersion change ratio to obtain its comprehensive fatigue index, and divide the confidence level of the mode in the previous interaction period by the comprehensive fatigue index to obtain the current confidence level of the mode.

[0086] Since the mode is a highly active mode during attenuation calculation, and the dispersion of the latter part of the mode is less than that of the former part, the dispersion change ratio of this mode is greater than 1, and the relative activity level of this mode is usually greater than 1. The comprehensive fatigue index reflects the combined amplification effect of the dispersion change ratio and the relative activity level. If either the dispersion change ratio or the relative activity level is close to 1, then the comprehensive fatigue index is mainly determined by the other. The increase in the comprehensive fatigue index is limited, and the attenuation should also be limited. Only when the relative activity level is significantly greater than 1 and the dispersion change ratio is significantly greater than 1 can the comprehensive fatigue index be significantly amplified, the attenuation increase, and the current confidence level of this mode decrease significantly.

[0087] Step S250: The ratio of the current confidence level of each modality to the sum of the current confidence levels of all modalities is used as the decision weight of each modality.

[0088] The sum of the decision weights for all modalities is 1, and the decision weights can accurately reflect the relative credibility of each modality.

[0089] Since confidence decay calculation is an immediate negative evaluation of modal interaction fatigue, but this penalty is not permanent, the modality should be given the opportunity to re-engage in decision-making after the user stops using the modality intensively for a period of time. Therefore, when determining the current confidence level for any modality based on its confidence level in the previous interaction period, the following steps are also included:

[0090] Step S260: For any modality, retrieve the dormancy time of the modality in multiple consecutive historical interaction periods, wherein the dormancy time is defined as the number of interaction periods in which the activity index of the modality is continuously lower than or equal to the corresponding preset threshold, starting from the first normal modality interaction period after the current confidence of the modality first falls below the initial confidence.

[0091] Since the statistics on inactivity time need to reflect the true inactivity characteristics of the modality in recent interactions, the further back the historical inactivity data is from the current time, the more likely it is to lose its reference value due to changes in user behavior or usage environment. Therefore, we trace back from the current interaction period and extract several consecutive historical interaction periods that are immediately adjacent to the current period.

[0092] Only when the modality transitions to the normal modality and the activity index falls below the preset threshold for the first time is it considered that the modality has been released from high-intensity use, and the dormancy period begins to be counted. For modalities that are already in dormancy statistics, check whether the activity index of the modality in the current period is still lower than or equal to the preset threshold. If so, the dormancy time is incremented by 1. If not, the dormancy time count is interrupted and the count will restart when the modality enters the normal modality again. If the current confidence level of the modality has not fallen below the initial confidence level, the dormancy time count will not be started.

[0093] Step S270: Arrange the silence times in order of magnitude to form a silence time series, and calculate the mean of the silence time series.

[0094] The processor maintains a historical dormancy time record table for this mode in memory. Each time a dormancy period ends, that is, after the mode becomes a highly active mode again or the confidence level returns to the initial value, the system appends the total duration of the dormancy period to the record table. The record table only retains the dormancy time values ​​of the most recent odd number of times, and old records that exceed the retention window are discarded.

[0095] At the moment when it is necessary to determine whether to initiate recovery, the processor retrieves all historical dormant time values ​​from the modality record table, arranges them according to their numerical values ​​to form a dormant time series, and calculates the arithmetic mean of the dormant time series.

[0096] Step S280: If the current confidence level of the current mode is lower than the initial confidence level, and the quiescent time of the current mode is greater than or equal to the mean of the quiescent time series, then the current confidence level of the current mode is adjusted according to... The function gradually recovers from the current value to the initial value with the interaction period as the step size; otherwise, the current confidence of the modality is set to its confidence in the previous interaction period.

[0097] During each interaction period after recovery startup, the processor calculates the corresponding recovery value based on the current recovery step, updates the current confidence level to this value, and determines whether the initial confidence level has been reached or exceeded. If it has been reached, the recovery stops and the current confidence level is set to the initial confidence level.

[0098] The S-shaped curve characteristic of the function determines the three-stage rhythm of the recovery process. In the early stage of recovery, growth is slow, and the system's renewed trust in the mode is tentative. The increase in its decision weight is limited and will not have a significant impact on the current multimodal fusion pattern. In the middle stage of recovery, as the dormant period continues to verify, the system's confidence in the mode has been fully restored, and the confidence level recovers at a faster pace. In the late stage of recovery, growth slows down again when approaching the initial confidence level, ensuring that the final convergence is stable and that overshoot does not occur.

[0099] It should also be noted that: The slope parameter of the function controls the recovery speed. The larger the slope, the shorter the recovery process and the less interaction time is required for the modality to return from the decay state to the initial confidence level. This parameter is preset according to the target interaction rhythm of the toy and is usually set to make the recovery process span several to a dozen interaction periods to match the attention switching cycle in children's interaction.

[0100] Step S300: Based on the decision weights of each modality, select candidate combination strategies that meet the matching conditions from the preset combination strategy library and determine their strategy weights. For each candidate combination strategy, obtain the emotion vectors detected before and after execution in past interactions, calculate the difference between the two emotion vectors, and perform a dot product operation on the difference with the preset positive emotion benchmark vector to obtain the emotion improvement degree.

[0101] Examples of the combined strategy library are shown in Table 1 below:

[0102] Table 1. Preset Strategies in the Combination Strategy Library

[0103] Strategy Number Strategy Name Input modes involved Output mode combination Output parameter description S01 Pure voice encouragement voice Hearing Play the pre-stored encouraging voice clip "You're awesome!" at the system default medium volume. S02 Pure tactile feedback touch Tactile movement The torso motor is driven to perform a short vibration with an amplitude of 20% of the maximum stroke. S03 Pure visual appeal Visual Visual lighting The eye LEDs feature a warm yellow breathing light effect with a two-second cycle. S04 Voice and nod Voice, vision Auditory and tactile movements Playing a pre-stored questioning voice clip, "Hmm?", synchronously drives the head motor to perform a downward and then return motion with an amplitude of 15% of the maximum travel. S05 Voice and Lights Voice, vision Auditory and visual lighting Playing a pre-stored laughter clip simultaneously causes the ear-shaped decorative light strip to flash three times in a fast green mode. S06 Touch and light Touch, vision Tactile motion, visual lighting The torso motor drives a left-right sway with an amplitude of 30% of the maximum stroke, simultaneously driving the eye lights to change in a blue gradient pattern. S07 Voice and touch soothing Voice, Touch Auditory and tactile movements Playing a pre-stored, gentle humming track, simultaneously drives the back vibration motor to vibrate continuously for 3 seconds at a low duty cycle. S08 Multimodal celebration Voice, touch, vision Hearing, tactile movement, visual light Playing a pre-stored celebratory music clip, the head motors perform a nodding motion twice, simultaneously driving the full-body decorative light strips to flash in a rainbow-colored rotating pattern. S09 Default Combination Strategy No specific dependencies Hearing, vision, light, touch, movement Playing a raised-pitched questioning voice clip "Hmm?", the eye lights switch to a soft breathing light mode, and the head motor performs a slight head tilt before returning to its original position.

[0104] It should be noted that if no candidate combination strategy meets the matching conditions, the preset default combination strategy will be determined as the target combination strategy.

[0105] It should also be noted that the strategy weights of the default combination strategy are set to fixed values ​​and are not associated with the decision weights of any modality. The default combination strategy does not participate in the historical accumulation and evaluation of emotion improvement. After it is executed as the target combination strategy, the corresponding emotion vector difference will still be stored in the historical interaction record for use in subsequent strategy selection.

[0106] The matching condition is that the sum of the decision weights of each modality involved in the candidate combination strategy is greater than or equal to a preset matching threshold, wherein the matching threshold is the product of the sum of the decision weights of each modality and a first attenuation factor, and the first attenuation factor is a positive number less than 1.

[0107] It should be noted that the larger the first decay factor, the fewer the candidate strategies there are. In extreme cases, no strategy may meet the conditions, and the default combination strategy must be used as a fallback, which makes the interaction monotonous. The smaller the first decay factor, the more candidate strategies there are, making the screening mechanism ineffective and increasing the burden of subsequent evaluation value calculation. In order to balance the response success rate and response accuracy, it is usually taken as 0.5 to 0.8.

[0108] The combined strategy library contains a variety of preset strategies composed of different modal response modes. Each candidate combined strategy is associated with a corresponding strategy weight. For any candidate combined strategy, its strategy weight is the geometric mean of the decision weights of each modality involved.

[0109] First, the sum of the decision weights of each modality is used to quickly eliminate strategies that rely on input modalities with too little overall influence, ensuring that the strategies entering the candidate list have sufficient signal support at the perceptual level. Then, the geometric mean of the decision weights of each modality is used for comprehensive evaluation. Its function is to measure the degree of balance between the modalities relied upon by each strategy among the candidate strategies that have passed the screening. Then, this balance index is multiplied by the degree of emotion improvement to select the strategy with the highest comprehensive score.

[0110] When the past interaction records of a candidate combination strategy are empty, extract the strategy weights associated with the candidate combination strategy and the decision weights of each modality involved in the candidate combination strategy. Calculate the standard deviation of the decision weights of each modality involved in the candidate combination strategy. If the standard deviation is greater than zero, the product of the standard deviation and the strategy weight is taken as the emotional improvement degree of the candidate combination strategy. If the standard deviation is equal to zero, the product of the strategy weight and the second decay factor is taken as the emotional improvement degree of the candidate combination strategy. The second decay factor is a positive number less than 1.

[0111] A standard deviation greater than zero indicates that at least one mode in the candidate strategy combination has a significantly higher decision weight than the other modes. The larger the standard deviation, the more significant the difference between the decision weights of each mode, and the more prominent the dominant mode. Multiplying the standard deviation by the strategy weight makes the evaluation value more favorable to candidate strategies combination dominated by clearly high-weight modes.

[0112] A standard deviation of zero indicates that the decision weights of each modality involved in the candidate strategy combination are completely equal. Since the dominant modality cannot be identified, a conservative method is adopted to use the product of the strategy weight and the second decay factor as the degree of emotion improvement. The second decay factor can be set to 0.2 to 0.4.

[0113] The emotion vector includes a voice emotion value, a touch response value, and a visual attention value. The voice emotion value is obtained by extracting the fundamental frequency mean, average speech rate, and short-time energy envelope from the voice signal, normalizing them, and then calculating the arithmetic mean. The touch response value is obtained by extracting the average pressure, touch frequency, and effective contact area percentage from the touch signal, normalizing them, and then calculating the arithmetic mean. The visual attention value is obtained by extracting the head orientation yaw angle mean, gaze duration percentage, and body horizontal distance from the visual signal, normalizing them, and then calculating the arithmetic mean.

[0114] It should be noted that the normalization method used in the calculation of voice emotion value, touch response value, and visual attention value is min-max normalization.

[0115] The three components of the positive emotion baseline vector correspond to the voice emotion value, touch response value, and visual attention value in the emotion vector, respectively. Each component of the positive emotion baseline vector takes a value between 0 and 1, which indicates the direction of contribution and reference strength of the component when judging positive emotion changes. When the dot product of the difference between the emotion vector before and after the strategy is executed and the positive emotion baseline vector is positive and the larger the value, it means that the direction of the emotion change is more consistent with the preset positive direction and the emotion improvement effect is more obvious. When the dot product is negative, it means that the direction of the emotion change is opposite to the positive direction and the strategy may have produced a negative effect.

[0116] The specific values ​​of the three components of the positive emotion baseline vector are set based on the representativeness and reliability of each modality's expression of positive emotions. The vocal emotion value component is usually set to the highest value because pitch increase, speech rate increase, and energy enhancement in speech are the most direct and stable external manifestations of children's excitement and pleasure, and have the strongest indicative ability of positive emotions. The touch response value component is usually set to the second highest value; changes in touch initiative and pressure can also reflect willingness to participate, but its reliability is slightly lower than that of vocal emotion due to user habits and sensor sensitivity. The visual attention value component is usually set to a lower value because changes in gaze and head orientation can indicate active attention, blank staring, or passive gazing; a single-direction indicator has limited discriminative power for positive emotions. Therefore, the positive emotion baseline vector can be set to... .

[0117] Step S400: The product of the emotion improvement degree of each candidate combination strategy and its corresponding strategy weight is used as the evaluation value of the candidate combination strategy. The candidate combination strategy with the largest evaluation value is selected as the target combination strategy. The target combination strategy is converted into multimodal output parameters and control instructions are generated.

[0118] The evaluation value is calculated, and the comparator selects the strategy corresponding to the maximum evaluation value as the target combination strategy. The processor maps it to specific output parameters such as audio segment number, motor action amplitude and duration, light color and flashing mode, and generates control commands which are sent to the speaker, motor driver and LED driver units via I2C bus or PWM.

[0119] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0121] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for controlling intelligent toys based on multimodal perception, characterized in that, Includes the following steps: The system acquires multimodal signals of the smart toy during the current interaction period, and performs frame-by-frame analysis of the multimodal signals to extract the activity index and feature sequence of each modality. The activity index of each modality is compared with the corresponding preset threshold. The confidence of each modality is updated based on the comparison results and feature sequences to determine the current confidence of each modality and perform normalization processing to obtain the decision weight of each modality. Based on the decision weights of each modality, candidate combination strategies that meet the matching conditions are selected from the pre-set combination strategy library and their strategy weights are determined. For each candidate combination strategy, the emotion vectors detected before and after execution in past interactions are obtained, the difference between the two emotion vectors is calculated, and the dot product operation is performed on the difference with the preset positive emotion baseline vector to obtain the emotion improvement degree. The product of the emotion improvement of each candidate combination strategy and its corresponding strategy weight is used as the evaluation value of the candidate combination strategy. The candidate combination strategy with the largest evaluation value is selected as the target combination strategy. The target combination strategy is converted into multimodal output parameters and control instructions are generated.

2. The intelligent toy control method based on multimodal perception according to claim 1, characterized in that, The multimodal signals include at least speech signals from the speech modality, touch signals from the touch modality, and visual signals from the visual modality. The activity index of the speech modality is the short-time average energy of the speech signal, the activity index of the touch modality is the pressure peak of the touch signal, and the activity index of the visual modality is the inter-frame difference energy of the visual signal. Each frame element in the feature sequence of the speech modality includes at least two evaluation dimensions: frame-by-frame fundamental frequency and frame-by-frame short-time energy. Each frame element in the feature sequence of the touch modality includes at least two evaluation dimensions: frame-by-frame touch pressure and frame-by-frame contact area percentage. Each frame element in the feature sequence of the visual modality includes at least three evaluation dimensions: frame-by-frame head yaw angle, frame-by-frame angle between the line of sight and the center of the toy, and frame-by-frame proportion of the face in the field of vision.

3. The intelligent toy control method based on multimodal perception according to claim 2, characterized in that, Obtaining the decision weights for each modality includes the following steps: Set an initial confidence level for each modality. The current confidence level of any modality is its confidence level in the current interaction period, and the initial value of the current confidence level of that modality is equal to the initial confidence level. For any modality, its activity index is compared with the corresponding preset threshold. If its activity index is greater than the corresponding preset threshold, the modality is marked as a high-activity modality; otherwise, the modality is marked as a normal modality. For any modality, its feature sequence is divided into a first subsequence and a second subsequence according to a preset ratio in chronological order. When the feature sequence of the modality contains multiple evaluation dimensions, the first and second subsequence dispersions are calculated for each evaluation dimension. The arithmetic mean of the first subsequence dispersions of all evaluation dimensions of the modality is then used as the first subsequence dispersion of the modality. The arithmetic mean of the second subsequence dispersions of all evaluation dimensions of the modality is then used as the second subsequence dispersion of the modality. In a single evaluation dimension, the absolute values ​​of the scalar differences of the element values ​​of the evaluation dimension in adjacent frames are summed. The sum of the absolute values ​​of the first subsequences is divided by the number of frames in the first subsequence to obtain the first subsequence dispersion. The sum of the absolute values ​​of the second subsequences is divided by the number of frames in the second subsequence to obtain the second subsequence dispersion. During the current interaction period, if a mode is a highly active mode and the dispersion of the latter part of the mode is less than that of the former part, then the current confidence of the mode is set to the value of its confidence in the previous interaction period after decay calculation; otherwise, the current confidence of the mode is set to its confidence in the previous interaction period. The ratio of the current confidence level of each modality to the sum of the current confidence levels of all modalities is used as the decision weight of each modality.

4. The intelligent toy control method based on multimodal perception according to claim 3, characterized in that, The logic for determining the preset threshold is as follows: extract multiple consecutive historical interaction periods from the historical time period, obtain the activity index of each modality in each historical interaction period, arrange the activity indexes from smallest to largest to form an activity index sequence, and use the upper quartile of the activity index sequence as the preset threshold of the corresponding activity index.

5. The intelligent toy control method based on multimodal perception according to claim 3, characterized in that, For any modality, the logic for determining its current confidence level by calculating the decay of its confidence level in the previous interaction period is as follows: Multiple consecutive historical interaction periods are extracted from the historical period. The activity index of the modality in each historical interaction period is obtained and the average value is calculated. If the average value is greater than zero, the ratio of the activity index of the modality in the current interaction period to the average value is taken as the relative activity level of the modality. Otherwise, the relative activity level of the modality is set to 1. The ratio of the dispersion of the first segment to the dispersion of the second segment of the mode is taken as the dispersion change ratio of the mode. The relative activity level of the mode is multiplied by the dispersion change ratio to obtain its comprehensive fatigue index. The confidence level of the mode in the previous interaction period is divided by the comprehensive fatigue index to obtain the current confidence level of the mode.

6. The intelligent toy control method based on multimodal perception according to claim 3, characterized in that, For any modality, when determining the current confidence level based on the confidence level during the previous interaction period, the following steps are also included: For any modality, retrieve the dormancy time of that modality in multiple consecutive historical interaction periods, wherein the dormancy time is defined as the number of interaction periods in which the activity index of that modality is continuously lower than or equal to the corresponding preset threshold, starting from the first normal modality interaction period after the current confidence of that modality first falls below the initial confidence. Arrange the periods of silence in order of length to form a period of silence time series, and calculate the mean of the period of silence time series; If the current confidence level of the current mode is lower than the initial confidence level, and the quiescent time of the current mode is greater than or equal to the mean of the quiescent time series, then the current confidence level of the current mode is adjusted according to... The function gradually recovers from the current value to the initial value with the interaction period as the step size; otherwise, the current confidence of the modality is set to its confidence in the previous interaction period.

7. The intelligent toy control method based on multimodal perception according to claim 1, characterized in that, The matching condition is that the sum of the decision weights of each modality involved in the candidate combination strategy is greater than or equal to a preset matching threshold, wherein the matching threshold is the product of the sum of the decision weights of each modality and a first attenuation factor, and the first attenuation factor is a positive number less than 1.

8. The intelligent toy control method based on multimodal perception according to claim 1, characterized in that, The combined strategy library contains a variety of preset strategies composed of different modal response modes. Each candidate combined strategy is associated with a corresponding strategy weight. For any candidate combined strategy, its strategy weight is the geometric mean of the decision weights of each modality involved.

9. The intelligent toy control method based on multimodal perception according to claim 1, characterized in that, When the past interaction records of a candidate combination strategy are empty, extract the strategy weights associated with the candidate combination strategy and the decision weights of each modality involved in the candidate combination strategy. Calculate the standard deviation of the decision weights of each modality involved in the candidate combination strategy. If the standard deviation is greater than zero, the product of the standard deviation and the strategy weight is taken as the emotional improvement degree of the candidate combination strategy. If the standard deviation is equal to zero, the product of the strategy weight and the second decay factor is taken as the emotional improvement degree of the candidate combination strategy. The second decay factor is a positive number less than 1.

10. The intelligent toy control method based on multimodal perception according to claim 1, characterized in that, The emotion vector includes a voice emotion value, a touch response value, and a visual attention value. The voice emotion value is obtained by extracting the fundamental frequency mean, average speech rate, and short-time energy envelope from the voice signal, normalizing them, and then calculating the arithmetic mean. The touch response value is obtained by extracting the average pressure, touch frequency, and effective contact area percentage from the touch signal, normalizing them, and then calculating the arithmetic mean. The visual attention value is obtained by extracting the head orientation yaw angle mean, gaze duration percentage, and body horizontal distance from the visual signal, normalizing them, and then calculating the arithmetic mean.