Robot autonomous humming method and device, electronic equipment and medium

Abstract

The present disclosure provides a robot autonomous humming method and device, electronic equipment and medium, which can be applied to the technical fields of intelligent robots, human-computer interaction, emotional computing, audio processing and the like. The method comprises: evaluating a humming allowable state of a robot based on object state information of a target object, task state information of the robot and interaction state information between the target object and the robot, to obtain an evaluation result; in the case that the evaluation result meets a preset humming condition, determining a target melody segment from a preset melody library based on a target emotion category determined based on the object state information, the preset melody library comprising a plurality of melody segments and a plurality of attribute tags of the plurality of melody segments, the attribute tag of the target melody segment matching the target emotion category; determining humming audio for the robot to hum based on the target melody segment; and controlling the robot to play the humming audio.

Description

Technical Field

[0001] This disclosure relates to the technical fields of intelligent robots, human-computer interaction, affective computing, and audio processing, and more specifically, to a method, device, electronic device, and medium for a robot to autonomously hum. Background Technology

[0002] With the rapid development of smart home and emotional interaction technologies, intelligent robots are widely used in various scenarios such as early childhood education, elderly care, and home leisure. Among these applications, music and audio functions have become one of the main mediums for intelligent robots to engage in emotional interaction with users.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related technology: Currently, the music playback function of intelligent robots mainly plays music by passively responding to user playback commands and recommending and playing music based on the user's historical preferences, lacking an active and flexible playback control mechanism, which reduces the user experience. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus, electronic device and medium for a robot to hum autonomously.

[0005] One aspect of this disclosure provides a method for a robot to autonomously hum, the method comprising: evaluating the robot's humming permissible state based on object state information of a target object, task state information of the robot, and interaction state information between the target object and the robot, and obtaining an evaluation result; if the evaluation result meets preset humming conditions, determining a target melody fragment from a preset melody library based on a target emotion category determined by the object state information, wherein the preset melody library includes multiple melody fragments and attribute labels for each of the multiple melody fragments, and the attribute labels of the target melody fragment match the target emotion category; determining a humming audio for the robot to hum based on the target melody fragment; and controlling the robot to play the humming audio.

[0006] According to embodiments of this disclosure, the permissible state of the robot's humming is evaluated based on the object state information of the target object, the task state information of the robot, and the interaction state information between the target object and the robot, to obtain an evaluation result. This includes: evaluating the matching degree between the emotional state of the target object and the robot's humming scenario based on the object state information, and determining an emotional evaluation index for the target object; evaluating the compatibility degree between the robot's task state and the robot's humming task based on the task state information, and determining a task state evaluation index for the robot; evaluating the matching degree between the current moment and the robot's humming timing based on the interaction state information, and determining a humming timing evaluation index; and determining the evaluation result based on the emotional evaluation index, the task state evaluation index, and the humming timing evaluation index.

[0007] According to embodiments of this disclosure, the matching degree between the current moment and the robot's humming timing is evaluated based on interaction state information to determine humming timing evaluation indicators. This includes: determining a silence duration influence factor affecting the humming timing evaluation indicators based on the duration of the non-interaction state between the target object and the robot; determining an interval duration influence factor affecting the humming timing evaluation indicators based on the interval between the current moment and the robot's last humming; determining a context influence factor affecting the humming timing evaluation indicators based on the interaction duration state of the target object and the robot after the current round of interaction; and determining the humming timing evaluation indicators based on the silence duration influence factor, the interval duration influence factor, and the context influence factor.

[0008] According to embodiments of this disclosure, determining a target melody segment from a preset melody library based on the target emotion category determined by object state information includes: determining the target time scene category to which the current time period belongs; determining the target melody segment from the preset melody library based on the joint state category determined by the target emotion category and the time scene category, wherein the attribute label of the target melody segment matches the joint state category.

[0009] According to embodiments of this disclosure, the attribute tag includes the target object's preference information for the melody segment and the playback timing information of the melody segment; determining the target melody segment from a preset melody library based on the joint state category determined by the target emotion category and the target time scene category includes: determining multiple candidate melody segments from the preset melody library based on the joint state category; and determining the target melody segment from the multiple candidate melody segments based on the preference information and playback timing information of the candidate melody segments.

[0010] According to embodiments of this disclosure, determining a humming audio for a robot to hum based on a target melody fragment includes: determining melody adjustment parameters for fine-tuning the target melody fragment based on the target emotion intensity determined by object state information and the target sub-segment of the target time scene category to which the current moment belongs; adjusting the target melody fragment based on the melody adjustment parameters to obtain an updated target melody fragment; and determining a humming audio for the robot to hum based on the updated target melody fragment.

[0011] According to embodiments of this disclosure, based on the target emotion intensity determined by object state information and the target sub-period of the target time scene category to which the current moment belongs, melody adjustment parameters for fine-tuning the target melody segment are determined, including: determining melody body adjustment parameters and vocal rendering adjustment parameters based on the target emotion intensity and the target sub-period; the melody body adjustment parameters include at least one of pitch adjustment parameters, rhythm adjustment parameters, and duration adjustment parameters; the vocal rendering adjustment parameters include at least one of humming mode adjustment parameters, volume adjustment parameters, and fade-in / fade-out adjustment parameters; and determining the melody adjustment parameters based on the melody body adjustment parameters and the vocal rendering adjustment parameters.

[0012] According to an embodiment of this disclosure, determining a humming audio for a robot to hum based on a target melody fragment includes: inputting the target melody fragment into a human voice humming synthesis module and outputting the humming audio, wherein the human voice humming synthesis module is used to combine the target melody fragment and preset human voice timbre information to perform human voice synthesis processing to generate the humming audio.

[0013] According to embodiments of this disclosure, the method further includes: in response to detecting a target event, controlling the robot to not perform or interrupt the playback of the humming audio; the target event includes at least one of the following: the target object performing a target interactive behavior, the robot performing a target task, the current time period meeting the target object's predetermined restricted playback time period, and the target object's environment meeting preset environmental conditions.

[0014] According to embodiments of this disclosure, after interrupting the playback of the humming audio, the method further includes: in response to detecting that the target event is in an ended state and the evaluation result meets the preset humming conditions, controlling the robot to resume playback of the humming audio at the interrupted position.

[0015] Another aspect of this disclosure provides a robot autonomous humming device, comprising: an evaluation module for evaluating the robot's humming permissible state based on the object state information of the target object, the robot's task state information, and the interaction state information between the target object and the robot, and obtaining an evaluation result; a first determination module for determining a target melody segment from a preset melody library based on the target emotion category determined by the object state information, provided that the evaluation result meets preset humming conditions, wherein the preset melody library includes multiple melody segments and attribute labels for each of the multiple melody segments, and the attribute labels of the target melody segment match the target emotion category; a second determination module for determining a humming audio for the robot to hum based on the target melody segment; and a control module for controlling the robot to play the humming audio.

[0016] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0017] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the methods described above.

[0018] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.

[0019] According to embodiments of this disclosure, the timing of the robot's automatic humming is evaluated by comprehensively considering object state information, robot task operation status information, and human-machine interaction status information. When the humming permission conditions are met, the robot proactively generates and plays a personalized humming audio by matching a melody fragment with a corresponding tag based on the user's emotions. This avoids scenarios unsuitable for humming, such as task conflicts or disturbing the user, while also providing lightweight human voice humming companionship output that aligns with the user's current emotional needs, thereby enhancing the user experience. Furthermore, through the robot's proactive humming, companionship signals can be output without user interaction, filling emotional voids in scenarios such as solitude, low mood, and insomnia, continuously delivering gentle and healing emotional comfort, significantly enhancing the warmth and comfort in scenarios such as home care and solitary rest, and constructing an immersive companionship experience with emotional healing attributes. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 A schematic diagram of an example environment 100 to which the robot autonomous humming method according to an embodiment of this application can be applied is shown;

[0022] Figure 2 A flowchart illustrating a robot autonomous humming method according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 3 A flowchart illustrating an embodiment of the present disclosure for determining an evaluation result is shown schematically;

[0024] Figure 4 A flowchart illustrating a method for determining a target melody segment according to an embodiment of the present disclosure is shown schematically;

[0025] Figure 5 A flowchart illustrating a robot autonomous humming method according to another embodiment of the present disclosure is shown schematically;

[0026] Figure 6 A block diagram of a robot autonomous humming device according to an embodiment of the present disclosure is shown schematically;

[0027] Figure 7 A block diagram of an electronic device suitable for implementing a robot's autonomous humming method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0028] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0033] Currently, the music recommendation function of intelligent robots is usually passively triggered in response to the user's playback command, and the playback is the complete music from an existing music playlist. In companionship scenarios, intelligent robots lack a mechanism for proactively recommending music. Furthermore, companionship scenarios are more suitable for short, lightweight humming behaviors that can be stopped at any time, while the longer duration of complete music and the companionship context are mismatched, reducing the user experience.

[0034] In view of this, this disclosure provides a method for a robot to autonomously hum, the method comprising: evaluating the robot's humming permissible state based on the object state information of the target object, the task state information of the robot, and the interaction state information between the target object and the robot, and obtaining an evaluation result; if the evaluation result meets preset humming conditions, determining a target melody fragment from a preset melody library based on the target emotion category determined by the object state information, wherein the preset melody library includes multiple melody fragments and attribute labels for each of the multiple melody fragments, and the attribute label of the target melody fragment matches the target emotion category; determining a humming audio for the robot to hum based on the target melody fragment; and controlling the robot to play the humming audio.

[0035] By comprehensively considering the object's state information, the robot's task operation status information, and the human-machine interaction status information, the timing of the robot's automatic humming is evaluated. When the humming permission conditions are met, the robot proactively generates and plays a personalized humming audio by matching the melody fragments with corresponding tags based on the user's emotions. This not only avoids scenarios that are unsuitable for humming, such as task conflicts or disturbing the user, but also achieves lightweight human voice humming companionship output that fits the user's current emotional needs, thereby improving the user experience.

[0036] Figure 1 A schematic diagram of an example environment 100 to which the robot autonomous humming method according to an embodiment of this application can be applied is shown. In this example environment 100, an application 125 is installed on a terminal device 110. A user 140 can interact with the application 125 via the terminal device 110 and / or an attachment device to the terminal device 110.

[0037] In some embodiments, application 125 can be downloaded and installed on terminal device 110. In some embodiments, application 125 can also be accessed in other ways, such as through a web page. Figure 1 In environment 100, in response to application 125 being launched, terminal device 110 can display the interface 150 of application 125.

[0038] In some embodiments, terminal device 110 can communicate with server 130 to provide services to application 125. Terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including robots, smart speakers, mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal digital assistants (PDAs), audio / video players, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 can also support any type of user-facing interface. Application 125 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, and computing devices in cloud environments.

[0039] In some embodiments, application 125 can provide interaction capabilities with an intelligent agent. Application 125 may include an application specifically designed to provide services to an intelligent agent, or an application integrated with an intelligent agent. Although Figure 1 The image shows a single application, but in reality, multiple applications can be installed on the terminal device 110.

[0040] In embodiments of this application, multiple intelligent agents 160 can be deployed locally on terminal device 110 or remotely. In the case of remote deployment, terminal device 110 can directly invoke the intelligent agents, or it can invoke the intelligent agents via server 130. Exemplarily, intelligent agents 160 may have intelligent dialogue and task processing capabilities. Terminal device 110 provides an interface 150 that can present interactions with intelligent agents 160. In interface 150, user 140 can initiate task requests to intelligent agents 160 by inputting natural language (e.g., text input or voice input). Optionally, user 140 can upload online or offline file dialogues to instruct intelligent agents 160 to assist in completing various tasks.

[0041] In the embodiments of this application, during interaction with user 140, intelligent agent 160 can respond to user 140's requests and handle tasks instructed by the user. In some embodiments, during task processing, intelligent agent 160 can invoke one or more tools 165 to assist in task execution and the provision of task results as needed. These tools 165 can be any type of tool, such as text generation tools, file reading tools, information search tools, online or offline databases, image processing tools, chart generation tools, web page creation tools, etc.

[0042] In some embodiments, environment 100 may further include a management node for multiple agents 160, which can interact with the agents 160. In some examples, the management node may, in response to a task request from user 140, determine the task requirements corresponding to the task request. The management node may then, based on the task requirements, assign the task request to the agent 160 that matches the task requirements, requesting that agent 160 to perform the task. In other examples, the management node may also determine an execution plan for the task based on the task requirements. The execution plan may indicate one or more subtasks required to complete the task. The management node may assign these one or more subtasks to one or more agents 160, which will then execute their respective subtasks. Regarding the management node, in some examples, the management node may be implemented by one of the multiple agents 160. In other examples, the management node may be implemented by a machine learning model, such as a language model.

[0043] In some embodiments, agent 160 may be constructed based on one or more machine learning models. In some embodiments, the machine learning model on which agent 160 is based may include at least a language model, such as a large language model. In some embodiments, the machine learning model on which agent 160 is based may include a multimodal model capable of handling multiple modal inputs, such as text input, visual input (e.g., images, videos), audio input, etc. These machine learning models may include content-generating models capable of generating corresponding outputs based on model inputs. In some embodiments, the machine learning model may receive text-modal model inputs (e.g., natural language and / or machine language) and / or non-text-modal model inputs (e.g., images, speech, videos, etc.), and may obtain corresponding model outputs based on model inputs and prompts, thereby completing the task execution.

[0044] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and are not intended to limit the scope of this application in any way.

[0045] Figure 2 A flowchart illustrating a robot autonomous humming method according to an embodiment of the present disclosure is shown schematically.

[0046] like Figure 2 As shown, the method includes operations S210 to S240.

[0047] In operation S210, the permissible state of the robot's humming is evaluated based on the object state information of the target object, the task state information of the robot, and the interaction state information between the target object and the robot, and the evaluation result is obtained.

[0048] In operation S220, if the evaluation result meets the preset humming conditions, the target melody segment is determined from the preset melody library based on the target emotion category determined by the object state information.

[0049] In operation S230, based on the target melody fragment, the humming audio for the robot to hum is determined.

[0050] Operating the S240, control the robot to play a humming audio.

[0051] The target object refers to the user using the robot. Object state information represents the perceived data of the user's real-time emotional and behavioral state. This can be obtained by collecting multimodal data such as speech, dialogue text, posture, and behavior of the target object through microphones, cameras, sensors, etc., and then extracting features from this multimodal data.

[0052] The robot's task status information can include the robot's current pending tasks, task priority, computing power usage, battery level, whether it is charging, whether it is moving or navigating, and whether the speaker / communication resources are occupied, among other device operation data.

[0053] Interaction status information is time-series data that records the sequential interaction behavior between humans and machines. It may include the duration of continuous silence by the user, the duration of the interval between two interactions, historical dialogue commands, whether the user has refused to hum, and whether a dialogue is currently in progress.

[0054] The humming-permissible state is used to characterize whether the robot is currently suitable for automatically playing humming audio, including two states: allowing automatic humming and disallowing automatic humming.

[0055] In some embodiments, object state information, task state information, and interaction state information can be scored separately according to preset rules, and then the scores can be weighted to obtain the evaluation result. This disclosure is not limited to this; object state information, task state information, and interaction state information can also be input into the evaluation model, and the evaluation model can fuse the three types of state information and output the evaluation result. The evaluation model can be, for example, a tree model, a lightweight neural network, or other types of models.

[0056] For example, the preset humming condition can be a scoring threshold. When the score value represented by the evaluation result is greater than the scoring threshold, it is determined that the robot is allowed to hum, and when the score value represented by the evaluation result is less than the scoring threshold, it is determined that the robot is not allowed to hum.

[0057] For example, emotion categories can be predefined, such as comfort-seeking, happiness and joy, sadness and longing, calm and relaxation, boredom-seeking, and sleepiness. Emotions can also be mapped to a continuous emotional space. For instance, emotional tendency and emotional activation intensity can be used as X and Y axes to form a two-dimensional plane, with the target's emotion at any given time corresponding to a coordinate point in the plane. Emotional tendency can range from -1 to 1, where closer to 1 indicates a more positive emotion, and closer to -1 indicates a more negative emotion. Emotional activation intensity can range from 0 to 1, where closer to 1 indicates a more excited target, and closer to -1 indicates a more tired target. The region can then be discretized into multiple blocks, each block representing an emotion category.

[0058] Melody fragments represent short sequences of note timing data stored in a preset melody library. They can be stored in digital formats such as Musical Instrument Digital Interface (MIDI) note sequences or structured note arrays.

[0059] The preset melody library includes multiple melody fragments and their respective attribute labels. The attribute labels of the target melody fragment are matched with the target emotion category.

[0060] Attribute tags can record information such as the melody ID, base key, base rhythm, duration, style tag, mood tag, most recently played time, and whether it can be looped for each melody fragment. You can search the preset melody library for mood category tags that match the target mood category to determine the target melody fragment.

[0061] Based on the target melody fragment, a short, accompanying humming audio clip suitable for the current state of the target object can be generated and played to the target object.

[0062] According to embodiments of this disclosure, the timing of the robot's automatic humming is evaluated by comprehensively considering object state information, robot task operation status information, and human-machine interaction status information. When the humming permission conditions are met, the robot proactively generates and plays a personalized humming audio by matching a melody fragment with a corresponding tag based on the user's emotions. This avoids scenarios unsuitable for humming, such as task conflicts or disturbing the user, while also providing lightweight human voice humming companionship output that aligns with the user's current emotional needs, thereby enhancing the user experience. Furthermore, through the robot's proactive humming, companionship signals can be output without user interaction, filling emotional voids in scenarios such as solitude, low mood, and insomnia, continuously delivering gentle and healing emotional comfort, significantly enhancing the warmth and comfort in scenarios such as home care and solitary rest, and constructing an immersive companionship experience with emotional healing attributes.

[0063] According to embodiments of this disclosure, the permissible state of the robot's humming is evaluated based on the object state information of the target object, the task state information of the robot, and the interaction state information between the target object and the robot. The evaluation result may include: evaluating the matching degree between the emotional state of the target object and the robot's humming scenario based on the object state information to determine the emotional evaluation index of the target object; evaluating the compatibility degree between the robot's task state and the robot's humming task based on the task state information to determine the task state evaluation index of the robot; evaluating the matching degree between the current moment and the robot's humming timing based on the interaction state information to determine the humming timing evaluation index; and determining the evaluation result based on the emotional evaluation index, the task state evaluation index, and the humming timing evaluation index.

[0064] The system can preset suitable and unsuitable scenarios for the robot to hum. Suitable scenarios include those described earlier, such as comforting needs, happiness, or sadness. Unsuitable scenarios include intense arguments, deep sleep, or serious studying. The system can identify the target object's emotions based on its state information, match them to the current humming scenario, calculate a matching score, and obtain an emotion assessment index.

[0065] If the target's emotional state matches a suitable scenario for the robot to hum, the score of the emotional assessment index will be high; if the target's emotional state does not match a suitable scenario for the robot to hum, the score of the emotional assessment index will be low.

[0066] The task status evaluation index is used to assess whether the robot's currently executing task and the humming task can run simultaneously. It can read the robot's currently running task, compare it with the humming task for conflict resolution, calculate a compatibility score, and obtain the task status evaluation index.

[0067] When the robot is idle, in standby mode, or slowly patrolling and monitoring without high-priority tasks, its compatibility is high, and its task status evaluation score is high. However, when the robot is performing high-priority tasks such as emergency alarms, high-precision following, voice interaction responses, and emergency monitoring, and its vocalizations would interfere with the execution of the main task, its compatibility is low, and its task status evaluation score is low.

[0068] The humming timing assessment index is used to evaluate whether the current moment is suitable for humming. It can extract the current moment and human-computer interaction behavior from the interaction status information, and score the humming timing based on preset rules for appropriate or prohibited timing, thus obtaining a humming timing assessment score.

[0069] In some embodiments, the emotion assessment index, task status assessment index, and humming timing assessment index can be assigned their respective weights and then summed to obtain the assessment result. Alternatively, thresholds can be set for each of the three indicators, and humming can be prohibited if any one of them falls below the threshold.

[0070] According to embodiments of this disclosure, by integrating three types of evaluation indicators—emotion evaluation indicators, task status evaluation indicators, and humming timing evaluation indicators—humming can be triggered in scenarios where emotions, tasks, and interaction timing are all suitable, making the humming interaction behavior more in line with the user's current feelings and the overall interaction process.

[0071] According to embodiments of this disclosure, the matching degree between the current moment and the robot's humming timing is evaluated based on interaction state information. The evaluation index for humming timing can include: determining a silence duration influence factor based on the duration of the non-interaction state between the target object and the robot; determining an interval duration influence factor based on the interval between the current moment and the robot's last humming; determining a context influence factor based on the interaction duration after the current round of interaction between the target object and the robot; and determining the humming timing evaluation index based on the silence duration influence factor, the interval duration influence factor, and the context influence factor.

[0072] The duration of the non-interaction state represents the continuous time elapsed from the end of the previous round of effective human-machine interaction to the current moment, during which there is no effective two-way interaction between the target object and the robot, such as voice, touch, or command response.

[0073] Based on interaction status information, it can calculate the continuous duration of no effective interaction between the target object and the robot from the end of the last interaction to the current moment. A silence duration impact factor can be calculated based on preset segmented thresholds. If the silence is too short, it indicates that the user still has interaction needs, thus reducing the score of the silence duration impact factor; if the silence reaches an appropriate range, the score of the silence duration impact factor will be increased.

[0074] Based on the interaction status information, the time when the robot last hummed can be determined, the time interval between the current time and that time point can be calculated, and an interval duration influence factor can be generated according to the length of the interval. If the interval is too short, the score of the interval duration influence factor is reduced to limit repeated humming. If the interval reaches a reasonable period, the score of the interval duration influence factor is restored, thereby avoiding interference caused by frequent humming in a short period of time.

[0075] It can identify the ongoing interaction status after the completion of this round of human-computer interaction, determine whether the interaction has ended, whether the user has unexpressed needs, and whether the interaction atmosphere is relaxed, and generate contextual impact factors accordingly. If the interaction is not finished or the user has communication needs, the score of the contextual impact factor will be reduced. If the interaction is complete and ends naturally, the score of the contextual impact factor will be increased.

[0076] The influence factors of silence duration, interval duration, and context can be weighted and fused to calculate the final humming timing evaluation index, which fully quantifies the suitability of humming behavior at the current moment.

[0077] Figure 3 A flowchart illustrating an embodiment of the present disclosure for determining an evaluation result is shown schematically.

[0078] like Figure 3 As shown, based on object state information 301, the matching degree between the target object's emotional state and the robot's humming scene is evaluated, and the target object's emotional evaluation index 304 can be determined. Based on task state information 302, the compatibility degree between the robot's task state and the robot's humming task is evaluated, and the robot's task state evaluation index 305 is determined. Based on the interaction state information 303 between the target object and the robot, the duration of the non-interaction state between the target object and the robot, the interval between the current moment and the robot's last humming, and the duration of interaction between the target object and the robot after the current round of interaction can be determined. Then, the silence duration influence factor 306, the interval duration influence factor 307, and the context influence factor 308 that affect the humming timing evaluation index are obtained in sequence. Combining the three influence factors, the humming timing evaluation index 309 can be determined. Based on the emotional evaluation index 304, the task state evaluation index 305, and the humming timing evaluation index 309, the evaluation result 310 is determined.

[0079] According to embodiments of this disclosure, the duration of silence is used to determine whether the user is in a relaxed and idle state without communication; the duration of humming intervals is used to avoid repeated humming in a short period of time, which can cause auditory fatigue in the user; the interaction context is used to distinguish different interaction stages such as the end of a conversation and the middle of a conversation to avoid interrupting effective communication; and multi-factor collaborative quantification of timing suitability is used to achieve fine-grained humming triggering sequence, prevent humming from being triggered in inappropriate scenarios such as during a conversation, too short an interval, or insufficient silence, and improve the naturalness of the timing of humming interaction and user comfort.

[0080] According to embodiments of this disclosure, determining a target melody fragment from a preset melody library based on the target emotion category determined by object state information may include: determining the target time scene category to which the current time period belongs; determining the target melody fragment from the preset melody library based on the joint state category determined by the target emotion category and the time scene category, wherein the attribute label of the target melody fragment matches the joint state category.

[0081] For example, the time scene category may include multiple categories such as morning, afternoon, evening, night, and late night. A joint state category can be formed based on the intersection of the target emotion category and the time scene category.

[0082] Attribute tags can include emotion tags and time scene tags. The target melody segment can be determined from the preset melody library based on the matching degree between the target emotion category and time scene category and the emotion tag and time scene tag in the attribute tags.

[0083] For example, the combination of comfort needs and evening can be matched with target melodic fragments that are low in volume, slow in tempo, closed-mouth humming, and short in a nostalgic style.

[0084] A cheerful and pleasant mood combined with an afternoon mood can be matched with a target melody fragment that is of medium volume, medium-fast tempo, half-open humming, and short duration.

[0085] For those experiencing drowsiness before bedtime and late at night, the ideal melody is one that is played at an extremely low volume, with a very slow tempo, hummed breaths, and a decaying, looping style.

[0086] In some embodiments, additional dimensional information such as location, task context, and the target object's historical preferences can be overlaid on top of the target emotion category and time scene category to obtain a joint state category. Correspondingly, attribute labels can also record the corresponding dimensional information. The target melody segment can be determined based on the overall matching degree between the joint state category and the attribute labels.

[0087] By constructing a dual classification dimension through pre-defined emotion categories and time scene categories, and using this as the filtering basis to match the target melody from the melody library, it can take into account both the emotional atmosphere and the coordination of the time scene atmosphere, greatly improving the targeting and rationality of melody selection, and making the output melody fit the real-time mood and current environmental time characteristics of the target audience.

[0088] According to embodiments of this disclosure, the attribute tags include the target object's preference information for melody segments and the playback sequence information of the melody segments.

[0089] Determining a target melody segment from a preset melody library based on a joint state category determined by the target emotion category and the target time scene category may include: determining multiple candidate melody segments from the preset melody library based on the joint state category; and determining the target melody segment from the multiple candidate melody segments based on the preference information and playback sequence information of the candidate melody segments.

[0090] User preference information can be learned through a pre-set user feedback closed-loop learning module. This module scores each melody based on both explicit and implicit feedback from the target audience, and these scores are continuously updated and personalized over a long period. Explicit feedback includes expressing liking or disliking, adding to a blacklist, etc., while implicit feedback includes whether to interrupt, whether to listen to the entire melody, and whether the user's mood improved after humming it.

[0091] Playback sequence information includes the historical playback time of the melody. Based on the playback sequence information, the freshness of the melody can be determined. Freshness can be calculated based on the interval between the most recent playback time and the current time. The longer the interval, the higher the freshness. If it has just been played, the freshness is lower.

[0092] The matching degree can be calculated in real time based on the joint state category and the attribute information in the attribute tags of the melody itself. The top candidate melody segments are filtered and sorted by joint state category.

[0093] Based on this, a weighted sum can be calculated from preference information, playback sequence information, and matching degree, and the melody with the highest score can be selected as the target melody. The weights can be customized or dynamically adjusted according to the scenario and learning process.

[0094] By leveraging user preferences, we can ensure that the selected melodies match the user's subjective auditory tastes. By quantifying the freshness of melodies through playback sequence information, we can avoid auditory fatigue caused by repeatedly playing the same melody in a short period of time. This balances the two core needs of personalized adaptation and temporal freshness, achieving a harmonious unity between personalization and temporal experience.

[0095] According to embodiments of this disclosure, determining the humming audio for the robot to hum based on a target melody fragment may include: determining melody adjustment parameters for fine-tuning the target melody fragment based on the target emotional intensity determined by object state information and the target sub-segment of the target time scene category to which the current moment belongs; adjusting the target melody fragment based on the melody adjustment parameters to obtain an updated target melody fragment; and determining the humming audio for the robot to hum based on the updated target melody fragment.

[0096] While analyzing the target object's state information to determine the target emotion category, it can also analyze the user's current emotional intensity, i.e., obtain the target emotion strength. Multiple gradients can be divided for each emotion category; for example, low mood can be divided into mild, moderate, and severe, while pleasant mood can be divided into slightly happy, highly relaxed, and extremely cheerful.

[0097] You can first divide the scene into broad categories based on the time of day, and then further subdivide each category into multiple sub-time periods. For example, the "morning" category includes sub-time periods such as morning washing up and commuting to breakfast; the "evening" category includes sub-time periods such as post-dinner relaxation and pre-sleep relaxation. The target sub-time period is the subdivided time period matched at the current moment. Different sub-time periods correspond to different environmental noise levels, user behavior habits, and suitable auditory atmospheres.

[0098] A mapping database can be pre-built, storing adjustment parameters corresponding to different emotional intensities and different sub-time periods. Melody adjustment parameters can be determined from a pre-defined mapping table based on two types of constraints: the target emotional intensity value and the target sub-time period matched at the current moment.

[0099] The target melody fragment obtained by matching can be used as a base to modify the audio features by adjusting the melody parameters, and output an updated target melody fragment that is adapted to the current user's emotional intensity and the segmented time period.

[0100] Instead of directly generating humming audio using a fixed target melody fragment, the system first generates exclusive adjustment parameters based on the user's emotional intensity gradient and finely divided time-segment sub-scenes. This allows for fine-tuning of the target melody fragment, enabling the humming audio to not only match the user's current emotional intensity but also adapt to the environmental atmosphere requirements of different time segments, achieving dynamic adaptation and differentiated customization of melody output.

[0101] According to embodiments of this disclosure, based on the target emotion intensity determined by object state information and the target sub-period of the target time scene category to which the current moment belongs, determining the melody adjustment parameters for fine-tuning the target melody segment may include: determining melody body adjustment parameters and vocal rendering adjustment parameters based on the target emotion intensity and the target sub-period; the melody body adjustment parameters include at least one of pitch adjustment parameters, rhythm adjustment parameters, and duration adjustment parameters; the vocal rendering adjustment parameters include at least one of humming mode adjustment parameters, volume adjustment parameters, and fade-in / fade-out adjustment parameters; and determining the melody adjustment parameters based on the melody body adjustment parameters and the vocal rendering adjustment parameters.

[0102] The melody body adjustment parameters modify the melody notes themselves, altering the melody's framework. The vocal rendering adjustment parameters do not change the notes or melody; they only enhance the robotic humming vocal effect.

[0103] It allows for the preset mapping relationship between emotional intensity, sub-time periods, and tone shift. The overall tone can be adjusted based on the user's emotional intensity, and further refined based on the target sub-time period. For example, if the target is experiencing deep depression or late-night sleep, the overall tone can be lowered and high notes weakened; if the target is experiencing excitement or morning relaxation, the tone can be raised.

[0104] The tempo scaling factor can be set based on the intensity of the emotion, and boundary limits can be imposed in conjunction with sub-time periods. For example, the higher the intensity of a soothing emotion, the smaller the tempo scaling factor, and the slower the melody tempo; the maximum tempo limit is forcibly restricted in the nighttime sub-time period to avoid a fast rhythm.

[0105] The duration of humming can be extended or reduced based on the intensity of the emotion and the target sub-period. For example, the duration can be extended for relaxing emotions and reduced for late-night sub-periods. Segmented loops with gradually decreasing duration are also allowed.

[0106] Multiple humming modes can be pre-stored, such as closed-mouth humming, half-open humming, and breath humming. The appropriate humming mode can be matched to the intensity of the emotion and the time period. For example, breath humming can be used for deep relaxation or at night; half-open humming can be used for mild pleasure or daytime leisure.

[0107] A base volume gain coefficient can be set, with the base loudness determined by the intensity of the emotion, and then attenuation / enhancement correction values ​​are added based on the target sub-period. For example, the volume can be reduced during the nighttime sub-period to avoid noise; the volume can be appropriately increased for cheerful emotions; and upper and lower volume thresholds can be set to prevent excessive loudness or inaudibility.

[0108] The fade-in and fade-out durations at the beginning and end of a melody can be configured. For intense relaxation or during the pre-sleep period, the fade-in and fade-out durations can be lengthened to achieve a gradual increase and decrease in volume, avoiding abrupt changes in sound that interrupt interaction. For example, during normal daytime hours, the fade-in and fade-out durations can be shortened to ensure smooth transitions between humming and dialogue, and a gentle transition between the start and stop of vocals.

[0109] When generating melody adjustment parameters, the target emotional intensity can be used as the primary control variable. The higher the emotional intensity, the greater the changes in pitch, rhythm, and duration. A secondary calibration is then performed using the correction coefficient of the target sub-period. When generating vocal rendering adjustment parameters, vibrato, breathy voice, and other vocal expressions are adjusted according to the emotional intensity, while the overall volume and fade-in / fade-out duration are constrained by the target sub-period.

[0110] The generated melody body adjustment parameters (such as pitch, rhythm, and duration) are combined with vocal rendering adjustment parameters (such as humming style, volume, and fade-in / fade-out) to form the melody adjustment parameters.

[0111] Therefore, it can take into account both the underlying melody and the optimization of the auditory quality of humming, and achieve fine-grained control of the melody and vocal layers, thereby enhancing the atmosphere and auditory comfort of humming audio.

[0112] Figure 4 A flowchart illustrating a method for determining a target melody segment according to an embodiment of the present disclosure is shown schematically.

[0113] like Figure 4 As shown, the object state information 401 and the current time 402 of the target object can be obtained. The target time scene category 404 to which the current time 402 belongs is determined, and the object state information is parsed to determine the target emotion category 403. Based on the joint state category determined by the target emotion category 403 and the time scene category 404, multiple candidate melody segments 406 are determined from a preset melody library 405. Based on the preference information 408 and playback sequence information 407 of each of the multiple candidate melody segments 406, the target melody segment 409 is determined from the multiple candidate melody segments 406.

[0114] According to embodiments of this disclosure, determining the humming audio for a robot to hum based on a target melody fragment may include: inputting the target melody fragment into a human voice humming synthesis module and outputting the humming audio, wherein the human voice humming synthesis module is used to combine the target melody fragment and preset human voice timbre information to perform human voice synthesis processing to generate the humming audio.

[0115] For example, the vocal humming synthesis module can generate humming audio by parametrically splicing pre-recorded vocal timbre fragments. Specifically, it can pre-record a library of vocal timbre fragments with different pitches, durations, and humming styles. By analyzing the target melody fragment, extracting the complete note sequence, and outputting parameters such as the target pitch, duration, tempo, rest interval, specified humming mode, and basic volume for each note, the module retrieves the corresponding vocal timbre fragments from the library based on these parameters and splices them to obtain the original humming waveform. After processing at the output layer, including adding fade-in / fade-out, subtle random vibrato, minor pitch detuning, weak reverb to suit the indoor environment, and dialogue avoidance, the final output is a smooth and coherent humming audio.

[0116] For example, the humming synthesis module can process the target melody segment using a pre-trained dedicated humming TTS (Text To Speech) model to generate humming audio. The dedicated humming TTS model can encode the input target melody segment, extract structured driving sequences such as pitch sequence, rhythm, note duration, and emotional timing adaptation parameters, and output frame-by-frame Mel spectrum, fundamental frequency, and glottal airflow features based on the melody driving sequence; simultaneously superimpose melody body adjustment parameters and vocal rendering parameters to simulate humming vocals in different weight, tempo, and opening / closing states to obtain the original humming waveform. After processing such as fade-in / fade-out, slight random vibrato, slight pitch detuning, weak reverberation adapted to the indoor environment, and dialogue avoidance in the output layer, the final output is a smooth and coherent humming audio.

[0117] For example, the humming synthesis module can also generate humming audio through a low-complexity processing method based on a singing / humming vocoder. A low-complexity vocoder optimized for humming scenarios can be used, preserving core modeling units such as pitch, duration, breath control, and timbre, adapting to low-computing-power operation at the robot's edge. The target melody segment is parsed by the melody feature encoding submodule to generate a melody feature sequence. The original humming time-domain waveform is obtained by inferring from the melody feature sequence through the vocoder inference submodule. After processing by adding fade-in / fade-out, subtle random vibrato, minor pitch detuning, weak reverberation adapted to the indoor environment, and dialogue avoidance at the output layer, a smooth and coherent humming audio is finally output.

[0118] By setting up a vocal humming synthesis module, driven by the target melody fragment and combined with preset vocal information, it can quickly output a natural and accompaniment-like humming audio that meets the needs.

[0119] According to embodiments of this disclosure, before or during the playback of a humming audio by controlling the robot, the process may further include: in response to detecting a target event, controlling the robot not to perform or interrupting the playback of the humming audio; the target event includes at least one of the following: the target object performing a target interactive behavior, the robot performing a target task, the current time period satisfying a predetermined restricted playback period of the target object, and the target object's environment satisfying preset environmental conditions.

[0120] Before or during the playback of a humming audio, the target event can be used as a hard interception rule. The target event has a higher priority than the evaluation rule. That is, as long as the target event is detected, the playback of the humming audio will not be executed.

[0121] The target interactive behavior could be a user speaking or calling the robot. When such an interactive behavior is detected, the humming is paused or stopped. This prevents the humming from masking the user's voice or interfering with normal conversation, ensuring clear and smooth communication.

[0122] The target task could be a voice broadcast, video or telephone call task that the robot is currently performing, or other tasks with higher priority than humming. When it is detected that the robot is performing such a task, humming should be paused or stopped to avoid the humming audio occupying the speaker channel and interfering with the task prompts and task execution logic.

[0123] The scheduled restricted playback period can be a period during which the user explicitly refuses to hum, a period when privacy mode or do-not-disturb mode is enabled, etc. If the current time falls within the scheduled restricted playback period, humming will not start or playback will be immediately stopped to meet the user's personalized needs.

[0124] Preset environmental conditions can include environmental data such as noise levels, number of people, and distance, ensuring that playback is prohibited. For example, if the ambient noise is too high, multiple people are talking, or the user is too far away, it will be determined that playing humming is not suitable, and an interception operation will be performed to adapt to the current ambient sound field and usage scenario.

[0125] By monitoring multiple target events in real time before and throughout the humming playback, problems such as humming interfering with human-computer dialogue, the robot's core tasks, user rest and learning, and special quiet environments can be effectively avoided, thereby improving the intelligent adaptability of robot interaction and user comfort.

[0126] According to embodiments of this disclosure, after interrupting the playback of the humming audio, the method may further include: in response to detecting that the target event is in an ended state and the evaluation result meets the preset humming conditions, controlling the robot to resume playback of the humming audio at the interrupted position.

[0127] It can continuously collect data from interaction, task, time period, and environmental sensors to determine whether the events that previously caused the interruption have been eliminated. For example, when the user ends the conversation and no longer issues interaction commands, the robot completes a high-priority target task, the current time exits the user's silent restriction period, or the environmental state leaves the prohibited playback condition, it can be determined that the target event has ended.

[0128] After the target event is monitored and ends, the humming tolerance assessment results obtained by integrating emotion, task, and humming timing indicators can be reviewed to confirm that the current user emotion, robot task load, and human-computer interaction timing are still suitable for humming playback; if the assessment results do not meet the preset humming conditions, humming will not be resumed even if the event ends.

[0129] It can synchronously record the playback progress information of the humming audio at the moment of interruption; after the double verification conditions are met, the progress information can be directly retrieved and the humming audio can be resumed from the cache position or the adjacent musical phrase position without regenerating the complete humming audio.

[0130] Therefore, the original humming tone can be used simultaneously to ensure that the melody and timbre are consistent and there will be no breaks in the melody or sudden changes in timbre.

[0131] Figure 5 A flowchart illustrating a robot autonomous humming method according to another embodiment of the present disclosure is shown schematically.

[0132] like Figure 5 As shown, the method includes operations S501 to S507.

[0133] The S501 is used to obtain object status information, robot task status information, and interaction status information between the target object and the robot.

[0134] If a target event is detected, operation S503 is executed to control the robot to stop playing the humming audio and to continue detecting the target event. If no target event is detected, operation S502 is executed to evaluate the robot's humming tolerance state based on the object's state information, the robot's task state information, and the interaction state information between the target object and the robot, and to obtain the evaluation result.

[0135] Even if the evaluation results do not meet the preset humming conditions, operation S503 will still be executed.

[0136] If the evaluation results meet the preset humming conditions, operation S504 is executed to determine the target melody segment from the preset melody library.

[0137] The S505 is operated to synthesize humming audio and control the robot to play the humming audio.

[0138] If no target event is detected, continue executing operation S505. If a target event is detected, stop playing the humming audio and continue detecting the target event.

[0139] If the target event has ended, return to operation S502. If the target event is detected, execute operation S507, provide feedback on the target object's preference information, and update the preset melody library.

[0140] Figure 6 A block diagram of a robot autonomous humming device according to an embodiment of the present disclosure is shown schematically.

[0141] like Figure 6 As shown, the device includes an evaluation module 610, a first determination module 620, a second determination module 630, and a control module 640.

[0142] The evaluation module 610 is used to evaluate the robot's humming permissible state based on the object state information of the target object, the task state information of the robot, and the interaction state information between the target object and the robot, and obtain the evaluation result.

[0143] The first determining module 620 is used to determine a target melody fragment from a preset melody library based on the target emotion category determined by the object state information when the evaluation result meets the preset humming conditions. The preset melody library includes multiple melody fragments and attribute labels for each of the multiple melody fragments, and the attribute labels of the target melody fragment match the target emotion category.

[0144] The second determining module 630 is used to determine the humming audio for the robot to hum based on the target melody fragment.

[0145] The control module 640 is used to control the robot to play humming audio.

[0146] According to embodiments of this disclosure, the evaluation module 610 includes a first evaluation submodule, a second evaluation submodule, a third evaluation submodule, and an evaluation determination submodule.

[0147] The first evaluation submodule is used to evaluate the degree of matching between the target object's emotional state and the robot's humming scene based on the object's state information, and to determine the target object's emotional evaluation index.

[0148] The second evaluation submodule is used to evaluate the compatibility between the robot's task status and the robot's humming task based on task status information, and to determine the robot's task status evaluation index.

[0149] The third evaluation submodule is used to evaluate the degree of matching between the current moment and the robot's humming timing based on the interaction state information, and to determine the evaluation index of the humming timing.

[0150] The assessment determination submodule is used to determine the assessment results based on emotion assessment indicators, task status assessment indicators, and humming timing assessment indicators.

[0151] According to embodiments of this disclosure, the third evaluation submodule includes a first factor determination unit, a second factor determination unit, a third factor determination unit, and a comprehensive factor determination unit.

[0152] The first factor determination unit is used to determine the silent duration influence factor that affects the humming timing evaluation index based on the duration of non-interaction between the target object and the robot.

[0153] The second factor determination unit is used to determine the interval duration influencing the humming timing evaluation index based on the interval between the current time and the robot's last humming.

[0154] The third factor determination unit is used to determine the contextual influence factors affecting the humming timing evaluation index based on the interaction continuity state of the target object and the robot after the current round of interaction.

[0155] The comprehensive factor determination unit is used to determine the evaluation index of humming timing based on the influence factors of silence duration, interval duration, and context.

[0156] According to embodiments of this disclosure, the first determining module includes a first determining submodule and a second determining submodule.

[0157] The first determination submodule is used to determine the target time scene category to which the current time period belongs.

[0158] The second determination submodule is used to determine the target melody fragment from the preset melody library based on the joint state category determined by the target emotion category and the time scene category. The attribute label of the target melody fragment matches the joint state category.

[0159] According to embodiments of this disclosure, the attribute tags include the target object's preference information for melody segments and the playback timing information of the melody segments. The second determination submodule includes a first determination unit and a second determination unit.

[0160] The first determining unit is used to determine multiple candidate melody segments from a preset melody library based on the joint state category.

[0161] The second determining unit is used to determine the target melody segment from multiple candidate melody segments based on the preference information and playback timing information of the candidate melody segments.

[0162] According to embodiments of this disclosure, the second determining module 630 includes a third determining submodule, an adjusting submodule, and a fourth determining submodule.

[0163] The third determination submodule is used to determine the melody adjustment parameters for fine-tuning the target melody segment based on the target emotion intensity determined by the object state information and the target sub-period belonging to the target time scene category at the current moment.

[0164] The adjustment submodule is used to adjust the target melody segment based on the melody adjustment parameters to obtain an updated target melody segment.

[0165] The fourth determination submodule is used to determine the humming audio for the robot to hum based on the updated target melody fragment.

[0166] According to embodiments of this disclosure, the third determining submodule includes a third determining unit and a fourth determining unit.

[0167] The third determining unit is used to determine the melody body adjustment parameters and vocal rendering adjustment parameters based on the target emotional intensity and the target sub-time period. The melody body adjustment parameters include at least one of the pitch adjustment parameters, rhythm adjustment parameters, and duration adjustment parameters. The vocal rendering adjustment parameters include at least one of the humming method adjustment parameters, volume adjustment parameters, and fade-in / fade-out adjustment parameters.

[0168] The fourth determining unit is used to determine the melody adjustment parameters based on the melody body adjustment parameters and the vocal rendering adjustment parameters.

[0169] According to embodiments of the present disclosure, the second determining module 630 includes a synthesis submodule.

[0170] The synthesis submodule is used to input the target melody fragment into the vocal humming synthesis module and output the humming audio. The vocal humming synthesis module is used to combine the target melody fragment and preset vocal timbre information to perform vocal synthesis processing and generate the humming audio.

[0171] According to embodiments of this disclosure, it further includes:

[0172] The second control module is used to control the robot to stop or interrupt the playback of the humming audio in response to the detection of a target event. The target event includes at least one of the following: the target object performs a target interactive behavior, the robot performs a target task, the current time period meets the target object's predetermined limited playback time, and the target object's environment meets preset environmental conditions.

[0173] According to embodiments of this disclosure, after interrupting the playback of the humming audio, the method further includes:

[0174] The third control module is used to control the robot to resume playing the humming audio at the interrupted position of the humming audio when the target event is detected to be in an ended state and the evaluation result meets the preset humming conditions.

[0175] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0176] For example, any and more of the evaluation module 610, the first determining module 620, the second determining module 630, and the control module 640 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in one module / unit / subunit.

[0177] According to embodiments of this disclosure, at least one of the evaluation module 610, the first determination module 620, the second determination module 630, and the control module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the evaluation module 610, the first determination module 620, the second determination module 630, and the control module 640 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0178] It should be noted that the data processing device part in the embodiments of this disclosure corresponds to the data processing method part in the embodiments of this disclosure. The specific description of the data processing device part is referred to in the data processing method part, and will not be repeated here.

[0179] Figure 7 A block diagram of an electronic device suitable for implementing a robot's autonomous humming method according to an embodiment of the present disclosure is shown schematically.

[0180] Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0181] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0182] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0183] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0184] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0185] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0186] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0187] For example, according to embodiments of this disclosure, a computer-readable storage medium may include one or more memories other than ROM 702 and / or RAM 703 described above.

[0188] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0189] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0190] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0191] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the target computing device, partially on the target computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the target computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0193] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for a robot to autonomously hum, characterized in that, The method includes: Based on the object state information of the target object, the task state information of the robot, and the interaction state information between the target object and the robot, the humming tolerance state of the robot is evaluated to obtain the evaluation result. If the evaluation result meets the preset humming conditions, a target melody segment is determined from the preset melody library based on the target emotion category determined by the object state information. The preset melody library includes multiple melody segments and attribute labels for each of the multiple melody segments. The attribute label of the target melody segment matches the target emotion category. Based on the target melody fragment, determine the humming audio for the robot to hum; and Control the robot to play the humming audio.

2. The method according to claim 1, characterized in that, The evaluation of the robot's humming permissibility state, based on the target object's object state information, the robot's task state information, and the interaction state information between the target object and the robot, yields an evaluation result, including: Based on the object state information, the matching degree between the target object's emotional state and the robot's humming scene is evaluated, and the emotional evaluation index of the target object is determined. Based on the task status information, the compatibility between the robot's task status and the robot's humming task is evaluated, and the task status evaluation index of the robot is determined. Based on the interaction state information, the degree of matching between the current moment and the robot's humming timing is evaluated, and the humming timing evaluation index is determined. The evaluation result is determined based on the emotion evaluation index, the task status evaluation index, and the humming timing evaluation index.

3. The method according to claim 2, characterized in that, Based on the interaction state information, the degree of matching between the current moment and the robot's humming timing is evaluated, and humming timing evaluation indicators are determined, including: Based on the duration of the non-interaction between the target object and the robot, the silence duration influence factor affecting the humming timing evaluation index is determined; Based on the time interval between the current moment and the robot's last humming, determine the interval duration influencing factor that affects the humming timing evaluation index; Based on the interaction continuity state of the target object and the robot after the current round of interaction, determine the contextual influence factors that affect the humming timing evaluation index; Based on the silence duration influence factor, the interval duration influence factor, and the context influence factor, the humming timing evaluation index is determined.

4. The method according to claim 1, characterized in that, Based on the target emotion category determined by the object state information, a target melody segment is determined from a preset melody library, including: Determine the target time scenario category to which the current time period belongs; Based on the joint state category determined by the target emotion category and the time scene category, the target melody segment is determined from the preset melody library, and the attribute label of the target melody segment matches the joint state category.

5. The method according to claim 4, characterized in that, The attribute tags include the target object's preference information for the melody segment and the playback sequence information of the melody segment; Based on the joint state category determined by the target emotion category and the target time scene category, the target melody segment is determined from a preset melody library, including: Based on the joint state category, multiple candidate melody segments are determined from the preset melody library; Based on the preference information and playback timing information of the candidate melody segments, the target melody segment is determined from a plurality of candidate melody segments.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining the humming audio for the robot to hum based on the target melody fragment includes: Based on the target emotion intensity determined by the object state information and the target sub-period of the target time scene category to which the current moment belongs, melody adjustment parameters for fine-tuning the target melody segment are determined; The target melody segment is adjusted based on the melody adjustment parameters to obtain an updated target melody segment; Based on the updated target melody fragment, a humming audio for the robot to hum is determined.

7. The method according to claim 6, characterized in that, Based on the target emotional intensity determined by the object state information and the target sub-period of the target time scene category to which the current moment belongs, melody adjustment parameters for fine-tuning the target melody segment are determined, including: Based on the target emotional intensity and the target sub-time period, melody body adjustment parameters and vocal rendering adjustment parameters are determined; the melody body adjustment parameters include at least one of pitch adjustment parameters, rhythm adjustment parameters, and duration adjustment parameters; the vocal rendering adjustment parameters include at least one of humming mode adjustment parameters, volume adjustment parameters, and fade-in / fade-out adjustment parameters. The melody adjustment parameters are determined based on the melody body adjustment parameters and the vocal rendering adjustment parameters.

8. The method according to claim 1, characterized in that, Based on the target melody fragment, determine the humming audio for the robot to hum, including: The target melody fragment is input into the vocal humming synthesis module, and the humming audio is output. The vocal humming synthesis module is used to combine the target melody fragment and preset vocal timbre information to perform vocal synthesis processing to generate the humming audio.

9. The method according to claim 1, characterized in that, Also includes: In response to the detection of a target event, the robot is controlled to either not execute or interrupt the playback of the humming audio. The target event includes at least one of the following: the target object performing a target interactive behavior, the robot performing a target task, the current time period meeting the target object's predetermined restricted playback time period, and the target object's environment meeting preset environmental conditions.

10. The method according to claim 9, characterized in that, After interrupting the playback of the humming audio, the following is also included: In response to detecting that the target event is in an ended state and the evaluation result meets the preset humming conditions, the robot is controlled to resume playing the humming audio at the interruption position of the humming audio.

11. A robot autonomous humming device, characterized in that, The device includes: The evaluation module is used to evaluate the humming tolerance state of the robot based on the object state information of the target object, the task state information of the robot, and the interaction state information between the target object and the robot, and obtain the evaluation result. The first determining module is used to determine a target melody fragment from a preset melody library based on the target emotion category determined by the object state information when the evaluation result meets the preset humming conditions. The preset melody library includes multiple melody fragments and attribute labels for each of the multiple melody fragments, and the attribute label of the target melody fragment matches the target emotion category. The second determining module is used to determine, based on the target melody fragment, a humming audio for the robot to hum; and The control module is used to control the robot to play the humming audio.

12. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images