A method, system and apparatus for controlling a robot

Abstract

The embodiment of the present specification provides a method, system and device for controlling a robot. The method comprises: determining an interaction intention based on external stimulus information; determining an expression parameter based on the external stimulus information, the interaction intention and an interaction frequency corresponding to the interaction intention; generating an audio waveform curve and a motor motion curve based on the expression parameter; extracting a first characteristic peak of the audio waveform curve and a second characteristic peak of the motor motion curve respectively; adjusting the motor motion curve based on the first characteristic peak and the second characteristic peak, so that a time difference between the first characteristic peak and the second characteristic peak of the adjusted motor motion curve is less than a preset tolerance threshold; and controlling the robot to perform audio output based on the audio waveform curve, and controlling the robot to perform motion based on the adjusted motor motion curve.

Description

Technical Field

[0001] This specification relates to the field of robot control technology, and in particular to a method, system and apparatus for controlling a robot to output audio and move. Background Technology

[0002] With the development of artificial intelligence, human-computer interaction and robot control technology, robots can output speech based on text-to-speech technology and perform corresponding actions by calling preset action materials or keyframe animations, thereby interacting with users in a way that combines sound and action.

[0003] However, in practical applications, robot audio is typically generated using text-to-speech technology, while robot movements are generated by calling preset motion materials or keyframe animations. Because audio and movements are generated separately and executed through different control chains, key changes in the audio may not accurately correspond to key motion moments in the movements. Simultaneously, the response delays of the audio output chain and the motor execution chain may differ, easily leading to asynchronous audio output and motion execution, and inconsistencies between motion amplitude and sound intensity. Furthermore, movements generated from preset motion materials are difficult to dynamically change according to the type of interaction, emotional intensity, interaction frequency, or robot operating state, resulting in relatively simplistic voice and motion expressions that affect the naturalness and flexibility of robot interaction.

[0004] Therefore, it is necessary to provide a method, system, and apparatus for controlling robots that can coordinate the temporal relationship between the robot's audio output and motion execution, thereby improving the synchronization of the robot's sound and movement and the naturalness of its interactive performance. Summary of the Invention

[0005] This specification provides one or more embodiments of a method for controlling a robot. The method includes: determining an interaction expression category based on external stimulus information received by the robot; determining expression parameters based on the external stimulus information, the interaction expression category, and the interaction frequency corresponding to the interaction expression category, wherein the expression parameters include emotional polarity, energy level, and expected duration; generating an audio waveform curve and a motor motion curve based on the expression parameters; extracting a first characteristic peak of the audio waveform curve and a second characteristic peak of the motor motion curve, respectively; adjusting the motor motion curve based on the first characteristic peak and the second characteristic peak, such that the time difference between the first characteristic peak and the adjusted second characteristic peak of the motor motion curve is less than a preset tolerance threshold; and controlling the robot to output audio based on the audio waveform curve and controlling the robot to move based on the adjusted motor motion curve.

[0006] This specification provides one or more embodiments of a system for controlling a robot. The system includes a determining module, a generating module, an extracting module, an adjusting module, and a controlling module. The determining module is configured to determine an interaction expression category based on external stimulus information received by the robot; the determining module is further configured to determine expression parameters based on the external stimulus information, the interaction expression category, and the interaction frequency corresponding to the interaction expression category, wherein the expression parameters include emotional polarity, energy level, and expected duration; the generating module is configured to generate an audio waveform curve and a motor motion curve based on the expression parameters; the extracting module is configured to extract a first characteristic peak of the audio waveform curve and a second characteristic peak of the motor motion curve, respectively; the adjusting module is configured to adjust the motor motion curve based on the first characteristic peak and the second characteristic peak, such that the time difference between the first characteristic peak and the adjusted second characteristic peak of the motor motion curve is less than a preset tolerance threshold; the controlling module is configured to control the robot to output audio based on the audio waveform curve and to control the robot to move based on the adjusted motor motion curve.

[0007] This specification provides one or more embodiments of an apparatus for controlling a robot, including a processor for performing a method of controlling the robot. Attached Figure Description

[0008] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0009] Figure 1 These are schematic diagrams illustrating application scenarios of a robot control system according to some embodiments of this specification; Figure 2 This is an exemplary block diagram of a system for controlling a robot according to some embodiments of this specification; Figure 3 This is an exemplary flowchart of a method for controlling a robot according to some embodiments of this specification; Figure 4 This is an exemplary schematic diagram illustrating the process of determining expression parameters according to some embodiments of this specification; Figure 5 This is an exemplary schematic diagram illustrating the process of generating audio waveform curves and motor motion curves according to some embodiments of this specification; Figure 6 This is an exemplary schematic diagram illustrating the process of superimposing an additional noise component and an additional breathing sound component onto an audio waveform curve according to some embodiments of this specification. Figure 7 This is an exemplary schematic diagram illustrating the process of adjusting the motor motion curve according to some embodiments of this specification. Detailed Implementation

[0010] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar situations without creative effort. Unless obvious from the linguistic context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0011] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0012] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0013] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0014] Figure 1 This is a schematic diagram illustrating an application scenario of a robot control system 100 (hereinafter referred to as system 100) according to some embodiments of this specification. For example... Figure 1 As shown, the application scenarios of system 100 may include processing device 110, network 120, terminal 130, storage device 140, and robot 150. In some embodiments, system 100 may be used to implement the methods and / or processes disclosed in this specification.

[0015] It should be understood that Figure 1The system 100 shown is for illustrative purposes only and does not constitute a limitation on the implementation methods described in this specification. In other embodiments, the number of devices and functional divisions included in system 100 can be adjusted according to actual application needs. For example, system 100 can be applied to family companionship scenarios, educational and entertainment scenarios, service reception scenarios, etc. In the above scenarios, robot 150 can interact with users through voice, actions, touch, etc. Robot 150 in system 100 can collect external stimulus information and send the external stimulus information to processing device 110 through network 120; processing device 110 can process the external stimulus information, generate audio waveform curves and motor motion curves, and send the audio waveform curves and adjusted motor motion curves to robot 150 to control robot 150 to perform audio output and movement. Storage device 140 can store data required or generated by processing device 110 during robot control and provide corresponding data to processing device 110.

[0016] The processing device 110 can be used to process data and / or information from various components of the system 100 and / or external data sources. The processing device 110 can execute program instructions based on this data, information, and / or processing results, thereby performing one or more functions described in this specification. For example, the processing device 110 can determine the interaction expression category based on external stimulus information received by the robot; determine expression parameters based on the external stimulus information, the interaction expression category, and the interaction frequency corresponding to the interaction expression category; generate an audio waveform curve and a motor motion curve based on the expression parameters; extract a first characteristic peak of the audio waveform curve and a second characteristic peak of the motor motion curve, respectively; adjust the motor motion curve based on the first and second characteristic peaks so that the time difference between the first characteristic peak and the adjusted second characteristic peak of the motor motion curve is less than a preset tolerance threshold; and control the robot to output audio based on the audio waveform curve and control the robot to move based on the adjusted motor motion curve.

[0017] In some embodiments, the processing device 110 may be implemented by a single server, a server cluster, a local device, a remote device, or a cloud platform.

[0018] Network 120 may include any suitable network that facilitates the exchange of information and / or data. In some embodiments, one or more components of system 100 may exchange information and / or data via network 120. For example, processing device 110 receives external stimulus information. Network 120 may include a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, or any combination thereof.

[0019] Terminal 130 may be an electronic device used by a user to interact with processing device 110 and / or robot 150. The user can send interaction commands to processing device 110 or robot 150 through terminal 130, and can also view the interaction and operating status of robot 150 through terminal 130. For example, terminal 130 may include mobile phone 131, tablet computer 132, computer 133, etc. In some embodiments, terminal 130 may have an application installed for interacting with processing device 110 or robot 150, through which the user can perform the above operations.

[0020] Storage device 140 can store data or information generated by other devices. In some embodiments, storage device 140 can store data related to system 100. In some embodiments, storage device 140 can store data and / or information processed by processing device 110, such as audio waveform curves and motor motion curves. Storage device 140 may include one or more storage device components, each of which may be a separate device or part of another device. Storage device 140 may be local or implemented via the cloud. In some embodiments, storage device 140 may be implemented on a cloud platform.

[0021] Robot 150 can be a robotic device that is placed on a desktop or other near-field interactive environment, has audio output and movement capabilities, and can interact with a user. Robot 150 may include sensing components, speakers, and motors. The sensing components can be used to receive external stimulus information; for example, the sensing components may include microphones, cameras, and touch sensors. The speakers can output audio. The motors can drive the head, torso, arms, or other movable parts of the robot 150 to move. In some embodiments, robot 150 can be a desktop companion robot, a desktop educational robot, a desktop entertainment robot, a desktop service robot, etc.

[0022] Figure 2 This is an exemplary block diagram of a system 200 (hereinafter referred to as system 200) for controlling a robot, as shown in some embodiments of this specification. In some embodiments, such as Figure 2 As shown, system 200 may include a determining module 210, a generating module 220, an extracting module 230, an adjusting module 240, and a controlling module 250. In some embodiments, one or more of the above modules may be integrated into processing device 110.

[0023] The determination module 210 is configured to determine the type of interaction expression based on the external stimulus information received by the robot.

[0024] The determining module 210 is further configured to determine expression parameters based on the external stimulus information, the interaction expression category, and the interaction frequency corresponding to the interaction expression category, wherein the expression parameters include emotional polarity, energy level, and expected duration.

[0025] The generation module 220 is configured to generate an audio waveform curve and a motor motion curve based on the expression parameters.

[0026] The extraction module 230 is configured to extract the first characteristic peak of the audio waveform curve and the second characteristic peak of the motor motion curve, respectively.

[0027] The adjustment module 240 is configured to adjust the motor motion curve based on the first characteristic peak and the second characteristic peak, so that the time difference between the first characteristic peak and the second characteristic peak of the adjusted motor motion curve is less than a preset tolerance threshold.

[0028] The control module 250 is configured to control the robot to output audio based on the audio waveform curve, and to control the robot to move based on the adjusted motor motion curve.

[0029] It should be understood that Figure 2 The system and its modules shown can be implemented in various ways. It should be noted that the above description of the robot control system and its modules is for convenience only and should not limit this specification to the scope of the illustrated embodiments. It is understood that those skilled in the art, after understanding the principles of the system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. In some embodiments, Figure 2 The determining module 210, generating module 220, extracting module 230, adjusting module 240, and controlling module 250 disclosed herein can be different modules within a single system, or a single module can implement the functions of two or more of the aforementioned modules. For example, the modules can share a single storage module, or each module can have its own separate storage module. Such variations are all within the scope of protection of this specification.

[0030] Figure 3 This is an exemplary flowchart of a method for controlling a robot according to some embodiments of this specification. In some embodiments, process 300 may be executed by a processing device.

[0031] Step 310: Determine the type of interaction expression based on the external stimulus information received by the robot.

[0032] External stimulus information refers to external input information collected by the robot through its sensing components. External stimulus information can be related to at least one of the following: user voice, user gestures, user approaching the robot, and user touching the robot.

[0033] In some embodiments, the robot can receive external stimulus information through sensing components and transmit this information to a processing device via a network. For example, the robot can collect user voice data via a microphone; it can also collect user image data via a camera; and it can detect user touch behavior information via a touch sensor. The aforementioned audio information, image data, and touch behavior information can serve as external stimulus information. In some embodiments, the external stimulus information may further include intensity information characterizing the strength of the interactive behavior, such as the volume of the user's voice, the duration of the user's voice, the amplitude of the user's movement, and the speed at which the user approaches the robot. This intensity information can be determined by the robot or by the processing device based on the external stimulus information collected by the robot.

[0034] Interaction expression categories refer to the categories of emotions or behaviors that a robot needs to exhibit when responding to external stimuli. For example, interaction expression categories can include the robot's confusion, surprise, grievance, and responding to greetings.

[0035] In some embodiments, the processing device can determine the category of the interaction expression based on external stimulus information using an intent parsing model. In some embodiments, the intent parsing model can be a machine learning model. For example, the intent parsing model can be a multimodal neural network (MNN).

[0036] In some embodiments, the input to the intent parsing model may include external stimulus information, and the output of the intent parsing model may be an interaction expression category. In some embodiments, the input to the intent parsing model may also include environmental features.

[0037] Environmental characteristics refer to features related to the robot's interactive environment. For example, environmental characteristics may include ambient noise (noise floor) and the distance between the user and the robot. In some embodiments, the robot can determine environmental characteristics through sensing components. For example, ambient noise (noise floor) can be determined by collecting ambient sound through a microphone; the distance between the user and the robot can also be determined by capturing images through a camera.

[0038] In some embodiments, the processing device can acquire multiple labeled training samples to form a training sample set, and perform multiple rounds of iteration based on the training sample set. The training samples include external stimulus information and environmental features, and the labels corresponding to the training samples are the sample interaction expression categories.

[0039] In some embodiments, the processing device can acquire historical data. Historical data may include multiple historical human-computer interaction records generated during robot interactions with users over a historical period. The processing device can determine multiple preferred interaction records from these historical records. For each preferred interaction record, the processing device can use the external stimulus information and environmental features corresponding to the preferred interaction record as training samples, and the historical interaction expression category corresponding to the preferred interaction record as the label corresponding to the training sample.

[0040] The method for determining preferred interaction records includes: in response to the robot outputting audio and moving based on historical interaction expression categories, if the user's willingness to continue interacting scores are greater than a preset scoring threshold, the processing device can determine the corresponding historical human-computer interaction record as the preferred interaction record. The preset scoring threshold can be pre-set based on prior knowledge.

[0041] At least one iteration includes: selecting one or more training samples from the training sample set; inputting the one or more training samples into the initial intent parsing model to obtain the model prediction output corresponding to the one or more training samples; substituting the model prediction output corresponding to the one or more training samples, and the labels corresponding to the one or more training samples, into the formula of a predefined loss function to calculate the value of the loss function; iteratively updating the model parameters in the initial intent parsing model based on the value of the loss function until the iteration termination condition is met, at which point the iteration ends, and the trained intent parsing model is obtained. The iterative updating of the model parameters of the initial intent parsing model can be performed using various methods, such as gradient descent.

[0042] Step 320: Determine the expression parameters based on external stimulus information, interaction expression categories, and the interaction frequency corresponding to the interaction expression categories.

[0043] Interaction frequency refers to the number of times the same interaction expression category is triggered within a preset time range. In some embodiments, the processing device may record historical interaction expression categories and the trigger time corresponding to each historical interaction expression category. After determining the current interaction expression category, from multiple historical interaction expression categories, the number of historical interaction expression categories whose trigger times are within the preset time range and are the same as the current interaction expression category is determined, and this number is taken as the interaction frequency corresponding to the current interaction expression category. The preset time range may be pre-set based on prior knowledge.

[0044] Expression parameters refer to the parameters that constrain a robot's audio output and movement. In some embodiments, expression parameters may include emotional polarity, energy level, and expected duration. Here, emotional polarity refers to the direction of the emotion the robot intends to express, energy level refers to the sound intensity of the audio and the amplitude of the robot's movement, and expected duration refers to the expected duration of the audio output and movement.

[0045] Figure 4 This is an exemplary schematic diagram illustrating the process of determining expression parameters according to some embodiments of this specification. Through Figure 4 The process 400 shown can determine the expression parameters. In some embodiments, such as Figure 4 As shown, the processing device can determine the emotional polarity 421, the initial energy level 423, and the expected duration 422 based on external stimulus information 411, the interaction expression category 412, and the interaction frequency 413 corresponding to the interaction expression category; determine the virtual endurance value 440 based on the robot's cumulative operating characteristics 430; and adjust the initial energy level 423 based on the virtual endurance value 440 to determine the energy level 450 in the expression parameters.

[0046] In some embodiments, the processing device can identify external stimulus information and extract intensity information to characterize the strength of the interactive behavior; based on the intensity information, the type of interactive expression, and the frequency of interaction corresponding to the type of interactive expression, it queries a preset parameter table to determine the emotional polarity, initial energy level, and expected duration. The preset parameter table may include the correspondence between different intensity information, interactive expression types, and interaction frequencies with emotional polarity, initial energy level, and expected duration. The preset parameter table may be pre-set based on prior knowledge or historical data.

[0047] For example, when the interaction category is a robot's question, the louder the user's voice, the higher the initial energy level can be determined; the longer the duration of the user's voice, the longer the expected duration can be determined. When the robot's question is triggered multiple times consecutively within a preset time range, the interaction frequency corresponding to the robot's question is high, and the corresponding initial energy level can be reduced, so that the robot can present different expression effects for the same interaction category.

[0048] Cumulative operational characteristics refer to features related to the robot's long-term operational status. For example, cumulative operational characteristics may include the robot's power-on time, number of actions executed, cumulative action duration, cumulative motion volume, historical sound recordings, cumulative motor operating time, and motor workload. Historical sound recordings may include the frequency and duration of high-volume sounds emitted by the robot within a preset historical time range. For example, the preset historical time range could be one minute prior to the current moment.

[0049] In some embodiments, the processing device can record the robot's runtime, the volume and duration of each audio output, and the runtime and workload of each motor operation during the robot's operation; and perform statistical analysis on the recorded data to determine the robot's cumulative operating characteristics. The preset statistical time range can be the time range from the robot's start-up to the current moment, or it can be a preset time range prior to the current moment.

[0050] Virtual endurance value refers to a state value used to characterize the robot's current energy level. The higher the virtual endurance value, the higher the robot's current energy level; the lower the virtual endurance value, the lower the robot's current energy level.

[0051] In some embodiments, the processing device can set an initial value and a preset upper limit for the virtual endurance value. For example, both the initial value and the preset upper limit of the virtual endurance value can be 100. It periodically acquires the robot's cumulative operating characteristics; inputs the cumulative operating characteristics into a preset energy decay function to determine the endurance consumption. For example, the higher the frequency of vocalizations in historical vocalization records, the longer the duration of high-volume vocalizations, the longer the cumulative operating time of the motor, or the greater the motor's workload, the greater the endurance consumption output by the preset energy decay function can be; and subtracts the endurance consumption from the virtual endurance value, using the value after subtracting the endurance consumption as the current virtual endurance value.

[0052] In some embodiments, in response to the robot being in a standby or low-activity state, the processing device may gradually increase the virtual endurance value based on the robot's standby or low-activity duration and a preset recovery rate, until the virtual endurance value reaches a preset upper limit. The preset recovery rate may be pre-set based on prior knowledge.

[0053] For example, the initial value and preset upper limit of the virtual endurance value are both 100. After the robot has been running for a period of time, the processing device can input the acquired cumulative running characteristics into a preset energy decay function and determine the endurance consumption as 8. The processing device subtracts 8 from the initial value of the virtual endurance value, so the current virtual endurance value after subtraction is 92. When the robot enters standby mode, if the preset recovery rate is 1 unit per minute, the virtual endurance value can be increased by 3 after the robot has been in standby mode for 3 minutes, at which point the current virtual endurance value is 95. If the robot continues to remain in standby mode, the virtual endurance value can continue to increase according to the preset recovery rate until the virtual endurance value reaches 100. In some embodiments, the processing device can determine the ratio between the current virtual endurance value and the preset upper limit as the endurance adjustment coefficient; and determine the level value corresponding to the energy level in the expression parameters by multiplying the level value corresponding to the initial energy level by the endurance adjustment coefficient, thereby determining the energy level.

[0054] For example, the preset upper limit for virtual endurance is 100, the current virtual endurance is 80, and the initial energy level corresponds to a level value of 60. The processing device can determine the ratio 0.8 between the current virtual endurance and the preset upper limit as the endurance adjustment coefficient, and determine the product of 60 and 0.8, 48, as the level value corresponding to the energy level in the expression parameters.

[0055] In some embodiments of this specification, by determining a virtual endurance value and adjusting the initial energy level based on the virtual endurance value, the energy level in the expression parameters can be matched with the robot's current energy level. This allows the robot to appropriately reduce the audio intensity and movement amplitude after long-term operation or continuous audio output and movement, preventing the robot from always using the same expression intensity for the same type of interaction, thereby improving the naturalness of the robot's interaction expression.

[0056] In some embodiments, the emotional polarity, expected duration, and energy level can be directly determined by querying a preset parameter table, omitting the energy level adjustment operation.

[0057] Step 330: Based on the expression parameters, generate the audio waveform curve and the motor motion curve.

[0058] An audio waveform curve is a curve that characterizes how the amplitude of an audio signal changes over time. An audio waveform curve can include multiple moments and the corresponding audio signal amplitudes at each moment. For example, an audio waveform curve could be the curve showing how the amplitude of the audio signal changes over time when a robot makes onomatopoeic expressions such as "hmm," "ah," or "hoo."

[0059] A motor motion curve is a curve that characterizes how the motor control parameters of a robot change over time. For example, motor control parameters may include at least one of the following: motor position, speed, acceleration, torque, etc.

[0060] In some embodiments, the processing device can establish a unified timeline based on the expected duration in the expression parameters. For example, the unified timeline can start from 0, and its length can be equal to the expected duration. Based on the expression parameters, a preset mapping rule is queried to generate an audio waveform curve and a motor motion curve under the unified timeline. The preset mapping rule can be pre-established based on prior knowledge or historical data.

[0061] Figure 5 This is an exemplary schematic diagram illustrating the process of generating audio waveform curves and motor motion curves according to some embodiments of this specification. Figure 5 Process 500 in the process is a process of generating audio waveform curves and motor motion curves under a unified time axis by querying preset mapping rules based on expression parameters.

[0062] like Figure 5 As shown, the processing device can generate an audio waveform curve 520 based on the expression parameter 510 and extract the audio features 531 of the audio waveform curve.

[0063] In some embodiments, the processing device can query a preset mapping rule based on the expression parameters to determine the audio waveform curve corresponding to the expression parameters from a preset audio curve library. The preset mapping rule can include the correspondence between different combinations of expression parameters and audio waveform curves. For example, the processing device can determine the audio waveform curve corresponding to the emotional polarity, energy level, and expected duration in the expression parameters.

[0064] Audio features refer to the characteristic information extracted from audio waveform curves that characterizes the changes in audio signals over time. Audio features can include energy envelope features, fundamental frequency variation features, and audio rhythm features. Among them, energy envelope features characterize the changes in the energy of the audio signal over time, reflecting the fluctuations in the intensity of the audio output at different times; fundamental frequency variation features characterize the changes in the fundamental frequency of the audio signal over time, reflecting the pitch change trend of the audio output at different times, such as pitch rising, pitch falling, or pitch remaining stable; audio rhythm features characterize the speed of sound changes in the audio waveform curve, the time interval between adjacent sounds, or the location of sound pauses.

[0065] In some embodiments, the processing device can divide the audio waveform curve into multiple audio frames according to a preset time length, and extract features from each audio frame. For example, the processing device can determine the audio energy corresponding to the audio frame based on the amplitude of multiple audio signals in each audio frame, and arrange the corresponding audio energies according to the time order of the multiple audio frames to obtain energy envelope features; determine the fundamental frequency corresponding to the audio frame based on the periodicity of the audio signals in each audio frame, and arrange the corresponding fundamental frequencies according to the time order of the multiple audio frames to obtain fundamental frequency variation features; determine multiple sound start times and pause intervals in the audio waveform curve based on the energy envelope features, and determine audio rhythm features based on the time interval between multiple sound start times and the position and duration of the pause intervals. For example, the processing device can detect the moment when the audio energy first reaches or exceeds a preset energy threshold (based on prior knowledge) and determine this moment as the sound start time; determine the time interval when the audio energy is continuously below the preset energy threshold as the pause interval; arrange multiple sound start times in time order and determine the time interval between adjacent sound start times to obtain audio rhythm features such as sound change frequency, sound interval distribution, pause position, and pause duration.

[0066] In some embodiments, such as Figure 5As shown, the processing device can determine the basic trajectory 542 based on the interactive expression category 412.

[0067] A base trajectory refers to a pre-set motion reference trajectory. A base trajectory can consist of multiple trajectory points arranged in chronological order, with each trajectory point including the corresponding time and the reference motor control parameters for that time. For example, when the robot's interaction type is "confusion," the corresponding base trajectory could be a trajectory used to tilt or raise the robot's head; when the robot's interaction type is "responding to a greeting," the corresponding base trajectory could be a trajectory used to make the robot perform a waving motion.

[0068] In some embodiments, the processing device can query preset mapping rules based on the interaction expression category to determine the basic trajectory corresponding to the interaction expression category. The preset mapping rules can include the correspondence between different interaction expression categories and different basic trajectories, and each basic trajectory can be pre-set and stored. For example, the preset mapping rules can include the correspondence between confusion and head tilt trajectory, the correspondence between confusion and head raise trajectory, and the correspondence between responding to a greeting and waving trajectory. In response to the interaction expression category being confusion, the preset mapping rules can be used to determine the basic trajectory for controlling the robot to tilt or raise its head; in response to the interaction expression category being a response to a greeting, the preset mapping rules can be used to determine the basic trajectory for controlling the robot to perform a waving action.

[0069] In some embodiments, such as Figure 5 As shown, the processing device can further determine the dynamic modulation parameters 541 based on the audio features 531.

[0070] Dynamic modulation parameters refer to parameters that describe the direction and degree of adjustment of the motion characteristics of the base trajectory. Adjustment direction parameters indicate whether the adjustment is increased or decreased relative to the reference motor control parameters in the base trajectory. Adjustment degree parameters can include speed modulation, acceleration modulation, motion amplitude modulation, attitude offset, and angle offset. Specifically, speed modulation indicates the degree to which the motor speed needs to be increased or decreased relative to the speed in the base trajectory; acceleration modulation indicates the degree to which the motor acceleration needs to be increased or decreased relative to the acceleration in the base trajectory; motion amplitude modulation indicates the degree to which the motor amplitude needs to be increased or decreased relative to the amplitude in the base trajectory; and attitude offset or angle offset indicates the degree to which the robot's attitude or motion angle needs to be offset relative to the base trajectory.

[0071] In some embodiments, the processing device can determine dynamic modulation parameters based on changes in sound intensity and pitch characterized by audio features. Specifically, the processing device can determine speed modulation, acceleration modulation, or amplitude modulation based on the intensity of audio energy at different times and the degree of change in audio energy characterized by energy envelope features, wherein the greater the enhancement of audio energy, the greater the speed modulation, acceleration modulation, or amplitude modulation can be.

[0072] The processing device can determine the attitude offset and angle offset based on the pitch change direction and degree represented by the fundamental frequency change characteristics. Responding to the fundamental frequency change characteristics indicating an upward pitch trend, the device can determine the attitude offset or angle offset that causes the robot to deflect upward or tilt laterally relative to the base trajectory, based on the degree of pitch rise. For example, the greater the degree of pitch rise, the greater the determined attitude offset or angle offset can be. Similarly, responding to the fundamental frequency change characteristics indicating a downward pitch trend, the device can determine the attitude offset or angle offset that causes the robot to deflect downward, reduce the upward angle, or reduce the side tilt angle relative to the base trajectory, based on the degree of pitch decline. For example, the greater the degree of pitch decline, the greater the determined attitude offset or angle offset can be.

[0073] The processing device can also determine pause distribution, oscillation frequency, or action beat parameters based on the characteristics of audio rhythm. For example, the faster the sound rhythm, the higher the determined oscillation frequency can be; the longer the time interval between adjacent sounds, the longer the determined pause duration between adjacent actions can be.

[0074] For reference again Figure 5 After determining the dynamic modulation parameters 541 and the basic trajectory 542, the processing device can generate the motor motion curve 550 based on the basic trajectory 542 and the dynamic modulation parameters 541.

[0075] In some embodiments, the processing device can acquire reference motor control parameters corresponding to multiple trajectory points in the basic trajectory, and adjust the reference motor control parameters corresponding to each trajectory point according to dynamic modulation parameters. For example, the reference velocity of the corresponding trajectory point can be adjusted based on the velocity modulation amount, the reference acceleration of the corresponding trajectory point can be adjusted based on the acceleration modulation amount, the reference position or motion amplitude of the corresponding trajectory point can be adjusted based on the motion amplitude modulation amount, and the reference position or motion direction of the corresponding trajectory point can be adjusted based on the attitude offset or angular offset. The processing device can arrange the adjusted multiple trajectory points according to their corresponding times to generate a motor motion curve.

[0076] In some embodiments of this specification, motor motion curves are generated by fusing a basic trajectory with dynamic modulation parameters. This ensures that the basic trajectory guarantees the robot performs semantically clear actions corresponding to the interactive expression category, while the dynamic modulation parameters adjust motion details such as speed, amplitude, or posture based on changes in audio energy, pitch, or rhythm. Therefore, while ensuring motion stability, the robot's motion can produce more natural dynamic changes in response to sound variations, improving the consistency between the robot's vocal and motor expressions and enhancing its mimicry effect.

[0077] In the above embodiments, the processing device can generate audio waveform curves and motor motion curves based on expression parameters. Considering that the robot's cumulative operating state may cause body posture deviations during continuous operation, resulting in corresponding changes in audio performance, the processing device can also adjust the audio waveform curves and motor motion curves based on the robot's cumulative operating characteristics. Figure 6 This is an exemplary schematic diagram illustrating the process of superimposing an additional noise component and an additional breathing sound component onto an audio waveform curve, according to some embodiments of this specification. Figure 6 The process shown in step 600 is a process for adjusting the audio waveform curve and the motor motion curve.

[0078] In some embodiments, such as Figure 6 As shown, before extracting the first feature peak from the audio waveform curve and the second feature peak from the motor motion curve, the processing device can determine the robot's body position offset 621 based on the robot's cumulative operating features 610.

[0079] Body position offset refers to the degree of deviation of a robot from its posture reference after prolonged operation. For example, body position offset can be used to simulate fatigue trends such as head-down, slumping, and hunching that robots exhibit after long-term operation.

[0080] In some embodiments, the processing device can continuously acquire the robot's cumulative operating characteristics, including power-on duration and the number of actions executed, and gradually increase the body posture deviation as the power-on duration and the number of actions executed increase. Specifically, the longer the power-on duration and the more actions executed, the greater the body posture deviation, but the deviation does not exceed the allowable posture deviation range of the robot mechanism. The posture deviation range can be preset based on the robot's mechanism's range of motion and safety constraints.

[0081] Furthermore, such as Figure 6 As shown, the processing device can adjust the attitude reference 630 of the motor motion curve based on the body offset 621.

[0082] An attitude reference refers to the neutral attitude that the motion trajectory corresponding to the motor motion curve references by default. For example, an attitude reference can be a neutral attitude reference point for a robot.

[0083] In some embodiments, the processing device can offset the neutral attitude reference point corresponding to the motor motion curve according to the body offset, determine the offset neutral attitude reference point as the adjusted attitude reference, and update the motor motion curve based on the adjusted attitude reference. For example, as the body offset increases, the processing device can gradually move the neutral attitude reference point corresponding to the robot's head downward, so that the robot presents a slightly lowered head or sunken posture when performing subsequent actions.

[0084] In some embodiments, such as Figure 6 As shown, the processing device can determine the additional noise component 622 based on the cumulative operating characteristics 610; and superimpose the additional noise component 622 onto the audio waveform curve 660.

[0085] Additional noise components refer to the low-frequency, rough acoustic components superimposed on the audio waveform curve, used to make the robot's output sound tired, low-pitched, or hoarse.

[0086] In some embodiments, the processing device can continuously acquire the robot's cumulative operating characteristics and determine the audio energy of the additional noise component based on the increase in the cumulative operating characteristics. Specifically, the longer the robot's operating time and the more times it performs actions, the greater the audio energy of the additional noise component.

[0087] In some embodiments, the processing device can align the additional noise component with the audio waveform curve to the same time axis, and add the waveform value of the additional noise component at each time point to the audio waveform value of the audio waveform curve at the corresponding time point to obtain the superimposed audio waveform curve. The additional noise component can be represented as a low-frequency, rough acoustic component, to make the robot's output sound tired, dry, or low-pitched.

[0088] In some embodiments of this specification, by determining the body posture offset based on the robot's cumulative operational characteristics and adjusting the attitude reference of the motor motion curve based on the body posture offset, the robot can gradually exhibit fatigue-like changes in posture, such as head-down and slumping, as the cumulative operational characteristics increase. Furthermore, by determining additional noise components based on the cumulative operational characteristics and superimposing these additional noise components onto the audio waveform curve, the robot's output sound can gradually exhibit a tired, dry, or low-pitched effect. Therefore, the robot can synchronously exhibit fatigue-like changes in posture and sound during long-term operation, preventing the robot from maintaining a completely consistent mental state, thereby enhancing the robot's sense of life and realism in long-term companionship scenarios.

[0089] In the above embodiments, after the processing device superimposes the additional noise component onto the audio waveform curve, in order to further reflect the robot's operating state when virtual endurance is insufficient and motor temperature is high, the processing device can superimpose an additional breathing sound component onto the audio waveform curve, making the robot's output sound like panting, heat dissipation, or fatigue. The process of superimposing the additional breathing sound component onto the audio waveform curve is as follows: Figure 6 As shown.

[0090] Figure 6 As shown, before extracting the first characteristic peak of the audio waveform curve and the second characteristic peak of the motor motion curve respectively, the processing device can obtain the motor temperature 640; in response to the virtual endurance value being lower than the preset lower limit and the robot's motor temperature 640 exceeding the preset temperature threshold, based on the magnitude of the motor temperature 640 exceeding the preset temperature threshold, an additional breathing sound component 650 is determined and superimposed on the audio waveform curve 660.

[0091] Motor temperature refers to temperature information related to the robot's motor. In some embodiments, the processing device can collect the motor temperature using a temperature sensor located at the robot motor and determine the collected temperature as the motor temperature.

[0092] Additional breathing sound component refers to the additional breathing sound component superimposed on the audio waveform curve, used to make the robot's output sound like panting, heat dissipation, or fatigue.

[0093] In some embodiments, in response to a virtual endurance value falling below a preset lower limit and the robot's motor temperature exceeding a preset temperature threshold, it can be determined that the robot is in a state of insufficient virtual endurance and high motor overheating. The processing device can determine the extent to which the motor temperature exceeds the preset temperature threshold based on the difference between the motor temperature and the preset temperature threshold. Based on the extent to which the motor temperature exceeds the preset temperature threshold, the audio energy of the additional breathing sound component is determined, and an additional breathing sound component with that audio energy is generated. The greater the extent to which the temperature threshold is exceeded, the greater the audio energy of the additional breathing sound component. The preset temperature threshold and the preset lower limit of the virtual endurance value can be preset based on prior knowledge and historical test data.

[0094] In some embodiments, the processing device can align the additional breathing sound component with the audio waveform curve to the same time axis, and add the waveform value of the additional breathing sound component at each time moment to the audio waveform value of the audio waveform curve at the corresponding time moment to obtain the superimposed audio waveform curve. The additional breathing sound component can be represented as airflow sound, exhalation sound, or heavy breathing sound.

[0095] In some embodiments of this specification, when the virtual endurance value is lower than a preset lower limit and the motor temperature exceeds a preset temperature threshold, an additional breathing sound component is determined and superimposed on the audio waveform curve. This allows the strength of the additional breathing sound component to match the robot's fatigue state and the degree of motor heating, thereby presenting a more natural panting, heat dissipation, or fatigue performance under high load conditions, enriching the robot's non-verbal expression layers.

[0096] Step 340: Extract the first characteristic peak of the audio waveform curve and the second characteristic peak of the motor motion curve, respectively.

[0097] The first characteristic peak refers to the representative peak value in the audio waveform curve that characterizes the change in sound energy. The first characteristic peak is used to reflect the moment when the loudness or energy of the non-verbal sound output by the robot is relatively strong, that is, the moment when the non-verbal sound has a strong sound impact.

[0098] In some embodiments, the processing device may determine one or more local energy peaks from the energy envelope characteristics, and based on the energy value or prominence of each local energy peak, determine a representative peak from the one or more local energy peaks, identifying the representative peak as a first characteristic peak. For example, the local energy peak with the largest energy value may be identified as the first characteristic peak.

[0099] The second characteristic peak refers to the representative peak value that characterizes the motion changes in the motor motion curve. The second characteristic peak is used to reflect the moment when the mechanical force is more obvious when the robot performs the corresponding motion, that is, the moment when the motor drives the robot to produce a large acceleration, a large torque change, or a strong motion impact.

[0100] In some embodiments, the processing device can determine one or more local motion peaks corresponding to acceleration, rate of change of velocity, or torque from the motor motion curve, and determine a representative peak from the one or more local motion peaks based on the peak size or prominence of each local motion peak, identifying the representative peak as a second characteristic peak. For example, the processing device can determine the ratio between each local motion peak and its corresponding preset peak threshold, and identify the local motion peak with the largest ratio as the second characteristic peak. Different types of local motion peaks can correspond to different preset peak thresholds.

[0101] Step 350: Based on the first characteristic peak and the second characteristic peak, adjust the motor motion curve so that the time difference between the first characteristic peak and the second characteristic peak of the adjusted motor motion curve is less than a preset tolerance threshold.

[0102] In some embodiments, the processing device can adjust the motor motion curve in various ways. For example, in the case of alignment based on a set of first and second characteristic peaks, the processing device can determine the first and second characteristic peaks as a set of corresponding peak values ​​and determine the time difference between the time point corresponding to the first characteristic peak and the time point corresponding to the second characteristic peak. In response to the time difference being greater than a preset tolerance threshold, the time axis of the motor motion curve is translated or scaled to adjust the time corresponding to each trajectory point in the motor motion curve, so that the time difference between the time point corresponding to the second characteristic peak and the time point corresponding to the first characteristic peak in the adjusted motor motion curve is less than the preset tolerance threshold. The preset tolerance threshold can be preset based on prior knowledge or historical data.

[0103] Figure 7 This is an exemplary schematic diagram illustrating the process of adjusting the motor motion curve according to some embodiments of this specification. For cases where there are multiple peaks that need to be aligned in both the audio waveform curve and the motor motion curve, the process is as follows: Figure 7 As shown in process 700, the processing device can first determine the correspondence between multiple local audio peaks and multiple local motion peaks, and then adjust the motor motion curve based on each set of corresponding peak values. Figure 7 As shown, in response to the existence of multiple local audio peaks in the audio waveform curve, the audio peak time series 721 is determined based on the time 711 corresponding to each of the multiple local audio peaks.

[0104] Multiple local audio peaks refer to multiple separate local peaks formed at different time positions in the energy envelope feature corresponding to the audio waveform curve. For example, when a robot continuously emits laughter, sighs, or hums, multiple local audio peaks can be formed in the energy envelope feature.

[0105] In some embodiments, the processing device can compare the energy value corresponding to each time moment in the energy envelope feature with the energy value corresponding to the adjacent time moment, and determine the energy value that is greater than the energy value corresponding to the adjacent time moment as the local maximum value; determine the local maximum value that reaches a preset energy threshold and whose time interval between it and the adjacent local maximum value is not less than a preset time interval as the local audio peak, thereby obtaining multiple local audio peaks. The preset energy threshold and preset time interval can be preset based on prior knowledge or historical data.

[0106] In some embodiments, the processing device may determine the time points corresponding to each of the multiple local audio peaks, and arrange the multiple time points in chronological order to determine the audio peak time sequence.

[0107] Furthermore, such as Figure 7As shown, in response to the existence of multiple local motion peaks in the motor motion curve, the motion peak time sequence 722 is determined based on the time 712 corresponding to each of the multiple local motion peaks.

[0108] Multiple local motion peaks refer to several separate local peaks formed at different time positions in the motor motion curve. They are used to characterize multiple relatively independent moments of mechanical force application or moments of significant motion change during a robot's continuous movement. For example, when a robot performs multiple nodding, swaying, or shaking actions in succession, multiple local motion peaks can be formed in the motor motion curve.

[0109] In some embodiments, the processing device can determine multiple local motion peaks based on acceleration, rate of change of velocity, or torque in the motor motion curve. Specifically, the processing device can determine multiple local maximum values ​​corresponding to acceleration, rate of change of velocity, or torque at different time positions, and determine the local maximum value that reaches a preset peak threshold as a local motion peak, thereby obtaining multiple local motion peaks. The preset peak threshold can be pre-set based on prior knowledge or historical data.

[0110] In some embodiments, the processing device can determine the time points corresponding to each of the multiple local motion peaks, and arrange the multiple time points in chronological order to determine the motion peak time sequence.

[0111] Refer again Figure 7 The processing device can further filter or merge multiple local motion peaks based on the audio peak time series 721 and the motion peak time series 722 to determine multiple target motion peaks 730 corresponding to the multiple local audio peaks.

[0112] Multiple target motion peaks refer to motion peaks identified from multiple local motion peaks, each corresponding to a different local audio peak.

[0113] In some embodiments, the processing device can determine multiple target motion peaks in various ways. For example, the processing device can count the number of multiple local audio peaks and determine the number of multiple local audio peaks as a first number; count the number of multiple local motion peaks and determine the number of multiple local motion peaks as a second number, wherein the second number is not less than the first number; in response to the second number being equal to the first number, the multiple local motion peaks can be determined as multiple target motion peaks; in response to the second number being greater than the first number, the multiple local motion peaks can be filtered to determine multiple target motion peaks. Specifically, the processing device can determine the prominence of each local motion peak and sort the multiple local motion peaks in descending order of prominence; select a first number of local motion peaks from the sorted multiple local motion peaks and determine the selected local motion peaks as multiple target motion peaks; and arrange the multiple target motion peaks according to the chronological order of their respective time points, so that the arranged multiple target motion peaks correspond sequentially to the multiple local audio peaks in the audio peak time series. The prominence of a local motion peak can be determined based on the corresponding peak acceleration, peak rate of change of velocity, or peak torque. For example, the larger the peak acceleration, peak rate of change of velocity, or peak torque, the greater the prominence of the corresponding local motion peak.

[0114] In some embodiments, the processing device can also merge multiple local motion peaks to determine multiple target motion peaks. Specifically, the processing device can determine the time interval between adjacent local motion peaks; in response to a time interval between two or more adjacent local motion peaks being less than a preset merging interval, the two or more adjacent local motion peaks are determined as a group of peaks to be merged, and the local motion peaks in the group of peaks to be merged are merged into one target motion peak; the above merging operation is repeated until the number of target motion peaks obtained after merging is the same as the number of multiple local audio peaks, and the multiple target motion peaks are arranged according to the chronological order of their respective times, so that the arranged multiple target motion peaks correspond sequentially to the multiple local audio peaks in the audio peak time series. The preset merging interval can be preset based on prior knowledge or historical data.

[0115] Furthermore, such as Figure 7 As shown, the processing device can use the corresponding peaks in the audio peak time series 721 and multiple target motion peaks 730 as anchor points, and perform time axis transformation and smooth connection on the motor motion curve based on the anchor points to obtain the adjusted motor motion curve 740, so that the time difference between multiple local audio peaks and the corresponding target motion peaks in the adjusted motor motion curve is less than the preset tolerance threshold.

[0116] An anchor point refers to one or more sets of corresponding peak points that constrain the temporal positional relationship between a target motion peak and its corresponding local audio peak. In some embodiments, the processing device can sequentially establish a correspondence between each time point in the audio peak time series and the time points corresponding to each of the multiple target motion peaks, and use the corresponding local audio peaks and target motion peaks as a set of anchor points. For example, the processing device can use the local audio peak corresponding to the i-th time point in the audio peak time series and the i-th target motion peak among the multiple target motion peaks as a set of anchor points.

[0117] In some embodiments, the processing device may divide the motor motion curve into multiple motion curve segments based on anchor points; perform time axis transformation on the multiple motion curve segments respectively; and smoothly connect the multiple motion curve segments after time axis transformation to obtain the adjusted motor motion curve, so that the time difference between each local audio peak and the corresponding target motion peak is less than a preset tolerance threshold.

[0118] In some embodiments, dividing the motor motion curve into multiple motion curve segments includes: the processing device dividing the motor motion curve based on the time points corresponding to the target motion peaks in each set of anchor points; determining the motor motion curve between the starting time point of the motor motion curve and the time point corresponding to the target motion peak in the first set of anchor points as the starting motion curve segment; for any two sets of anchor points that are adjacent in time, determining the motor motion curve between the time points corresponding to the target motion peaks in the two sets of anchor points as an intermediate motion curve segment; and determining the motor motion curve between the time point corresponding to the target motion peak in the last set of anchor points and the ending time point of the motor motion curve as the ending motion curve segment, thereby obtaining multiple motion curve segments.

[0119] In some embodiments, performing time axis transformation on multiple motion curve segments includes: the processing device can determine the time difference between the time point corresponding to each target motion peak and the time point corresponding to the corresponding local audio peak, and determine the time axis scaling parameter corresponding to each motion curve segment based on the time difference and a preset tolerance threshold; for each motion curve segment, the processing device can use the corresponding time axis scaling parameter and time axis scaling function to locally stretch or compress the time axis of the motion curve segment, so that the time difference between the time point corresponding to the target motion peak and the time point corresponding to the corresponding local audio peak is less than the preset tolerance threshold. The time axis scaling function can be a nonlinear time axis scaling function. For example, the processing device can use a cubic Hermite spline interpolation algorithm to perform time axis transformation on the motion curve segments.

[0120] In some implementations, smoothing the connection of multiple motion curve segments transformed by the time axis to obtain the adjusted motor motion curve includes: the processing device can arrange the starting motion curve segment, one or more intermediate motion curve segments, and the ending motion curve segment transformed by the time axis in chronological order; for any two motion curve segments that are adjacent in time, the motor control parameters of the latest trajectory point in the preceding motion curve segment and the motor control parameters of the earliest trajectory point in the following motion curve segment are obtained; based on the motor control parameters, the trajectory points in the preceding and following motion curve segments are interpolated to ensure that the motor position in the motor control parameters is continuous when the preceding and following motion curve segments are connected, and that the motor speed and motor acceleration remain continuous; and the multiple interpolated motion curve segments are connected in chronological order to obtain the adjusted motor motion curve.

[0121] In some embodiments of this specification, by determining the audio peak time series and the motion peak time series, multiple local motion peaks are filtered or merged to identify multiple target motion peaks corresponding to the multiple local audio peaks. These corresponding peaks are then used as anchor points to perform segmented time axis transformation and smooth connection of the motor motion curve. This allows for the individual adjustment of the time position of each target motion peak, avoiding misalignment of some peaks due to overall translation of the motor motion curve, and preventing discontinuities in the motion curve due to segmented adjustments. Therefore, it can improve the synchronization between audio output and robot motion in multi-peak non-verbal expression scenarios such as continuous laughter and continuous exclamations, and maintain the naturalness and continuity of robot motion.

[0122] In the above embodiments, the processing device can make the time difference between the first characteristic peak and the second characteristic peak of the adjusted motor motion curve less than a preset tolerance threshold. Considering that the robot's virtual endurance value may change after long-term operation or continuous interaction, if a fixed preset tolerance threshold is always used, the robot's sound and movement in different virtual endurance states may always maintain the same timing relationship, making it difficult to reflect the rhythm differences generated by the robot as its operating state changes. Therefore, in some embodiments, before adjusting the motor motion curve based on the first and second characteristic peaks, the processing device can also dynamically adjust the preset tolerance threshold based on the virtual endurance value. The preset tolerance threshold is negatively correlated with the virtual endurance value; for example, the higher the virtual endurance value, the smaller the adjusted preset tolerance threshold; the lower the virtual endurance value, the larger the adjusted preset tolerance threshold, and the adjusted preset tolerance threshold is within a preset allowable range. The preset allowable range can be preset based on prior knowledge or historical test data.

[0123] In some embodiments of this specification, by dynamically adjusting the preset tolerance threshold, a smaller preset tolerance threshold can be used when the virtual endurance value is high, so that the audio output and robot movement remain closely synchronized; when the virtual endurance value is low, the preset tolerance threshold can be appropriately increased, so that the robot movement exhibits a slight sluggishness and a more relaxed rhythmic characteristic relative to the audio output. Therefore, without significantly disrupting the overall synchronization between the audio output and the robot movement, the robot can exhibit corresponding rhythmic changes in a fatigued state, thereby enhancing the robot's state change performance during long-term interaction.

[0124] Step 360: Control the robot to output audio based on the audio waveform curve, and control the robot to move based on the adjusted motor motion curve.

[0125] In some embodiments, after completing the time alignment between the audio waveform curve and the motor motion curve, the processing device can send the audio waveform curve to the robot's audio output link and the adjusted motor motion curve to the robot's motor execution link. Through the audio output link, an audio signal is generated according to the audio waveform value corresponding to each moment in the audio waveform curve, and the audio signal is sent to the robot's speaker to play the audio signal through the speaker. Through the motor execution link, the robot's motor is controlled to run according to the motor control parameters corresponding to each moment in the adjusted motor motion curve, so that the motor drives the robot's movable parts to move according to the adjusted motor motion curve.

[0126] The response delays of the audio output link and the motor execution link from receiving control data to generating actual physical output may differ, leading to asynchrony between the robot's audio output and movement during actual execution. Therefore, the processing device can also compensate for the response delays of the audio output link and the motor execution link when controlling the robot to perform audio output and movement. In some embodiments, controlling the robot to output audio based on an audio waveform curve and controlling the robot to move based on an adjusted motor motion curve includes: determining the response delays of the audio output link and the motor execution link based on a hardware delay compensation table; and compensating for the delays of at least one of the audio waveform curve and the adjusted motor motion curve based on the response delays of the audio output link and the motor execution link, and controlling the robot to perform audio output and movement to synchronize the audio output and movement.

[0127] A hardware delay compensation table is a data table that pre-records the response delays of the audio output link and the motor execution link, reflecting the delay difference between receiving control commands and actually generating physical output. In some embodiments, the hardware delay compensation table may include the identification information of the audio output link and its corresponding response delay, as well as the identification information of the motor execution link and its corresponding response delay. The hardware delay compensation table can be pre-built based on the hardware test results of the audio output link and the motor execution link and stored in a storage device.

[0128] Response latency refers to the time interval between receiving a control command and actually generating a physical output in an audio output link or motor execution link. In some embodiments, the processing device can query a hardware latency compensation table based on the identification information of the currently used audio output link and motor execution link to obtain the response latency corresponding to the audio output link and the response latency corresponding to the motor execution link.

[0129] In some embodiments, the processing device can determine the delay difference between the response delays of the audio output link and the motor execution link. If the response delay of the audio output link is less than the response delay of the motor execution link, the processing device can temporarily store the audio waveform curve in a buffer and send it to the audio output link after delaying the delay difference. If the response delay of the motor execution link is less than the response delay of the audio output link, the processing device can temporarily store the adjusted motor motion curve in a buffer and send it to the motor execution link after delaying the delay difference, so as to synchronize the robot's audio output and motion.

[0130] In some embodiments of this specification, by determining the response delays of the audio output link and the motor execution link, and buffering the control flow corresponding to the faster-responding link, the hardware link delay differences that may still exist after curve alignment in the digital domain can be compensated. Therefore, the consistency between the actual sound emitted by the speaker and the actual movement of the motor can be improved, thereby enhancing the overall mimicry effect of the robot.

[0131] In some embodiments of this specification, by determining expression parameters and generating audio waveform curves and motor motion curves on a unified time axis based on these parameters, the sound intensity, motion amplitude, and duration can be matched with the current external stimulus information and interaction state. Furthermore, by extracting the first characteristic peak of the audio waveform curve and the second characteristic peak of the motor motion curve, and adjusting the motor motion curve based on these two characteristic peaks, the main moments of sound and movement can be aligned in time. This reduces the unnaturalness caused by the mismatch between motion amplitude and sound intensity. Simultaneously, since the expression parameters are also related to external stimulus information and interaction frequency, the same type of interaction expression can exhibit different sound and movement performances under different external stimulus information or different interaction frequencies, thereby improving the naturalness and biomimicry of the robot's nonverbal expression.

[0132] During the actual execution of audio output and robot movement, the robot may receive an interruption signal. Directly stopping the audio output and motor movement could easily cause the sound to abruptly cut off and the robot's movement to stop abruptly, affecting the natural transition between sound and motion. Therefore, the processing device can also perform corresponding interruption processing on the audio waveform curve and motor movement based on the relationship between the interruption time corresponding to the interruption signal and the adjusted second characteristic peak time.

[0133] In some embodiments, the processing device can perform different interruption processing on the audio waveform curve and motor motion based on the relationship between the interruption time and the adjusted second characteristic peak time. During the process of controlling the robot to output audio based on the audio waveform curve and controlling the robot to move based on the adjusted motor motion curve, in response to receiving an interruption signal, the processing device can determine the interruption time; in response to the interruption time being no later than the time corresponding to the second characteristic peak of the adjusted motor motion curve, the processing device performs low-pass filtering on the first curve segment of the audio waveform curve located after the interruption time and controls the robot to output audio based on the filtered first curve segment, while simultaneously applying a reverse braking torque pulse to the robot's motor to rapidly decelerate the motor and suppress the robot from continuing its current motion.

[0134] An interrupt signal is a signal used to terminate the robot's current audio output and movement. For example, an interrupt signal may include a new, higher-priority interaction command, or a signal generated by physical intervention such as touch or obstruction.

[0135] The interruption time refers to the current point in time when the interruption signal takes effect.

[0136] In some embodiments, during the process of the robot outputting audio according to the audio waveform curve and moving according to the adjusted motor motion curve, in response to detecting and confirming that the interruption signal is effective, the processing device can obtain the current time point, determine the current time point as the interruption time, and suspend the execution of the remaining curve segments of the two curves after that time point.

[0137] In some embodiments, the interruption time being no later than the time corresponding to the second characteristic peak of the adjusted motor motion curve indicates that the interruption occurred before the moment when the robot's current motion amplitude or intensity was at its maximum. Since the robot has not yet fully executed its current motion, directly stopping the audio output and motor motion at this point could easily cause an abrupt interruption in both sound and movement. Therefore, the processing device can determine the curve segment in the audio waveform curve located after the interruption time as the first curve segment, and filter the first curve segment using a preset low-pass filter to attenuate audio signal components in the first curve segment that are higher than a preset cutoff frequency, obtaining a filtered first curve segment. Based on the filtered first curve segment, the robot is controlled to output audio, making the sound after the interruption sound more muffled. Simultaneously, the processing device can apply a reverse braking torque pulse to the motor executing the adjusted motor motion curve, causing the motor to decelerate rapidly in a short time and inhibiting the robot from continuing its current motion.

[0138] In some embodiments, responding to an interruption time later than the time corresponding to the second characteristic peak of the adjusted motor motion curve indicates that the robot had completed the main action with the largest amplitude or intensity of motion in the current motion when the interruption occurred, and the current motion had entered the final stage. The processing device can lower the fundamental frequency value corresponding to each time in the first curve segment by a preset amplitude to obtain the first curve segment after fundamental frequency reduction, and control the robot to output audio based on the first curve segment after fundamental frequency reduction, so that the sound after the interruption presents a sinking or muffled sound. The preset amplitude can be preset based on prior knowledge or historical data. Fundamental frequency refers to the fundamental frequency in an audio signal that represents the pitch of a sound. For example, the higher the fundamental frequency, the higher the perceived pitch of the sound, and the lower the fundamental frequency, the lower the perceived pitch of the sound.

[0139] Meanwhile, the processing equipment generates a smooth reset curve according to a preset damping coefficient and controls the robot to move based on the smooth reset curve.

[0140] A smooth reset curve refers to a motion curve that smoothly restores the motor from its current position at the moment of interruption to its reset position. In some embodiments, the processing device can acquire the motor's current position and current speed at the moment of interruption, and determine the corresponding neutral position of the motor as the reset position. The neutral position refers to a preset motor position that restores the motor-driven movable component to its default posture, which can be preset based on prior knowledge or historical data. The current position and current speed are determined as the starting state of the smooth reset curve. The motor's speed is gradually reduced at subsequent moments according to a preset damping coefficient, and the motor's position at each subsequent moment is determined sequentially based on the gradually decreasing speed, so that the motor's position gradually approaches the reset position. A smooth reset curve is generated based on the position and speed corresponding to each subsequent moment, and the robot's motor is controlled to gradually decelerate and return to the neutral position based on the smooth reset curve. The preset damping coefficient can be preset based on prior knowledge or historical data.

[0141] In some embodiments of this specification, by employing different audio and motion interruption processing methods based on the relationship between the interruption time and the time corresponding to the adjusted second characteristic peak, the robot's sound and movements can be matched to the motion stage at the time of the interruption. Specifically, when the interruption time is no later than the time corresponding to the second characteristic peak, by low-pass filtering the first curve segment and applying a reverse braking torque pulse to the motor, the sound can be made to have a suppressed auditory feel, and the motor can be made to decelerate rapidly, thereby reducing the abruptness and mechanical impact caused by directly cutting off the sound and movement. When the interruption time is later than the time corresponding to the second characteristic peak, by lowering the fundamental frequency of the first curve segment and controlling the motor reset based on the smooth reset curve, the sound can be made to gradually sink, and the motor can be made to return to the neutral position smoothly. Thus, the robot can still present a coherent and natural sound and movement ending effect when interrupted, thereby improving the naturalness and biomimicry of the robot's non-verbal expression.

[0142] During the robot's movement, its moving parts may be obstructed by external forces, causing the motor to stall and unable to continue operating according to the adjusted motor motion curve. In this case, the adjusted motor motion curve still contains parts of motion that have not yet been actually executed. If the robot continues to output audio according to the original audio waveform curve, the sound performance may not match the actual obstructed motion. Therefore, in response to detecting a stall caused by external force blocking the motor, the processing device can compensate the audio waveform curve based on the unexecuted parts of the adjusted motor motion curve.

[0143] In some embodiments, in response to detecting that the motor is stalled due to an external force, the processing device can determine the stall time; determine the residual kinetic energy integral corresponding to the second curve segment in the adjusted motor motion curve after the stall time; determine the stall compensation coefficient based on the residual kinetic energy integral; determine the fundamental frequency boost and harmonic gain of the third curve segment in the audio waveform curve after the stall time based on the stall compensation coefficient; increase the fundamental frequency of the third curve segment based on the fundamental frequency boost; and increase the high-frequency harmonic gain of the third curve segment based on the harmonic gain.

[0144] Stalling refers to a state in which a motor is unable to continue moving according to its motion curve due to external obstruction during operation. For example, when a robot's head is held down or its arm is grabbed, the corresponding motor may stall.

[0145] In some embodiments, during the process of controlling motor operation based on the adjusted motor motion curve, the processing device can continuously detect the actual motion performed by the motor and compare the actual motion performed by the motor with the motion that should be performed according to the adjusted motor motion curve. If the deviation between the two exceeds a preset deviation threshold and continues for a preset duration, it can be determined that the motor has stalled due to external force. The preset deviation threshold and preset duration can be preset based on prior knowledge or historical data.

[0146] The stall moment refers to the point in time when the motor stalls due to external force. In some embodiments, in response to detecting and confirming that the motor has stalled due to external force, the processing device can obtain the current time point and determine the current time point as the stall moment.

[0147] The residual kinetic energy integral refers to the cumulative value of the planned but not actually executed motion intensity in the adjusted motor motion curve after the stall moment. The larger the residual kinetic energy integral, the more motion was not executed due to stall, or the greater the motion intensity.

[0148] In some embodiments, the processing device can define the curve segment in the adjusted motor motion curve that is located after the stall moment as the second curve segment; determine the speed difference between the motion speed corresponding to each moment in the second curve segment and the actual motion speed of the motor at the corresponding moment, or determine the acceleration difference between the motion acceleration corresponding to each moment in the second curve segment and the actual motion acceleration of the motor at the corresponding moment; and accumulate and integrate the speed difference or acceleration difference over time within the time range corresponding to the second curve segment, and determine the integration result as the residual kinetic energy integral corresponding to the second curve segment. Here, the actual motion speed refers to the detected motion speed of the motor at the corresponding moment, and the actual motion acceleration can be determined based on the difference between the actual motion speeds of the motor at adjacent moments and the time interval between adjacent moments. For example, during a stalled motor, if the actual speeds at 0, 0.1, and 0.2 seconds are 10, 5, and 0 respectively, and the actual speed remains 0 after 0.2 seconds, then the actual accelerations from 0 to 0.1 seconds and from 0.1 to 0.2 seconds are both -50, and the actual accelerations after 0.2 seconds are 0.

[0149] The hysteresis compensation coefficient is a parameter that characterizes the degree of compensation required by a robot when its motor fails to complete a movement due to external force. The larger the hysteresis compensation coefficient, the higher the degree of compensation required.

[0150] In some embodiments, the processing device can map the residual kinetic energy integral of the second curve segment to a stall compensation coefficient according to a preset mapping relationship. The stall compensation coefficient is positively correlated with the residual kinetic energy integral; the larger the residual kinetic energy integral of the second curve segment, the greater the intensity of motion that the motor failed to execute due to stall, and the larger the determined stall compensation coefficient. The preset mapping relationship can be determined in advance based on prior knowledge or historical data.

[0151] Furthermore, based on the hysteresis compensation coefficient, the fundamental frequency boost and harmonic gain of the third segment of the audio waveform curve located after the stall moment can be determined. The fundamental frequency boost refers to the frequency amplitude that needs to be increased based on the original fundamental frequency, used to characterize the degree of pitch increase. The greater the fundamental frequency boost, the greater the pitch increase.

[0152] Harmonic gain refers to the amount of gain that needs to be added to the original gain, and it is used to characterize the degree of enhancement of high-frequency harmonic components in an audio signal. The larger the harmonic gain, the greater the enhancement of high-frequency harmonic components.

[0153] In some embodiments, in response to detecting that the motor is stalled due to external force, the processing device can determine the curve segment in the audio waveform curve that is located after the stall moment as the third curve segment.

[0154] In some embodiments, the processing device can multiply the hysteresis compensation coefficient by a first preset amplitude and a second preset amplitude, respectively, and determine the two products as the fundamental frequency boost and harmonic gain of the third curve segment. The larger the hysteresis compensation coefficient, the larger the determined fundamental frequency boost and harmonic gain. The first preset amplitude is a preset amplitude for adjusting the fundamental frequency, and the second preset amplitude is a preset amplitude for adjusting the high-frequency harmonic gain; the first and second preset amplitudes can be preset based on prior knowledge or historical data.

[0155] In some embodiments, the processing device may increase the fundamental frequency value corresponding to each moment in the third curve segment by a fundamental frequency boost amount, thereby increasing the fundamental frequency of the third curve segment.

[0156] In some embodiments, the processing device can increase the gain value corresponding to the high-frequency harmonic components in the third curve segment by the harmonic gain amount, thereby improving the high-frequency harmonic gain of the third curve segment.

[0157] In some embodiments of this specification, by determining the residual kinetic energy integral, the intensity of motion that the motor failed to execute due to external force obstruction can be quantified. By determining the hysteresis compensation coefficient and, based on the hysteresis compensation coefficient, determining the fundamental frequency boost and harmonic gain of the third curve segment, and increasing the fundamental frequency and high-frequency harmonic gain of the third curve segment, the degree of boost in the fundamental frequency and high-frequency harmonic gain can be matched with the degree of robot motion obstruction. Thus, the mechanical state of robot motion obstruction can be converted into a sonic response, enabling the robot to not only stop moving when obstructed by external force, but also to express surprise, obstruction, or fright through sharper or more tense sounds, thereby enhancing the robot's interactive expressiveness and biomimicry.

[0158] As the robot continues to operate, its posture deviation may accumulate. When the posture deviation reaches a certain level, continuing to perform actions in the current state may affect subsequent motion performance. Therefore, when the posture deviation reaches a preset deviation threshold, the robot needs to perform a rest movement to reset the posture deviation. In some embodiments, in response to the posture deviation reaching the preset deviation threshold, the processing device can generate a rest movement curve; determine a transition audio curve based on the rest movement curve; control the robot to perform a rest movement based on the rest movement curve and control the robot to output a transition audio based on the transition audio curve; and reset the posture deviation in response to the end of the rest movement. The preset deviation threshold can be pre-set based on prior knowledge or historical data.

[0159] A rest movement curve refers to the movement trajectory used to control a robot to perform rest movements. For example, rest movements can include stretching, relaxing, yawning, or deep breathing.

[0160] In some embodiments, the processing device can periodically detect body posture deviations; in response to the body posture deviation reaching a preset deviation threshold, it queries a preset action library for a restorative action template corresponding to the interaction expression category, based on the interaction expression category. The restorative action template may include the duration of the restorative action and the displacement trajectories of each movable part of the robot within the duration of the action; a restorative motion curve is generated based on the duration of the action and the displacement trajectories. The preset action library may include the correspondence between different interaction expression categories and different restorative action templates. The preset action library may be pre-established based on prior knowledge.

[0161] For example, the processing device can generate a restorative motion curve based on the retrieved restorative motion template, which controls the corresponding motors of the robot's neck and back to slowly move backward to the maximum pitch angle.

[0162] A transition audio curve refers to the audio change curve of a robot as it enters a resting state, smoothly transitioning from its original state to the resting motion. For example, a transition audio curve can be the audio waveform curve corresponding to a long tone, an exhalation sound, or an upward glissando.

[0163] In some embodiments, the processing device can extract the motion duration and displacement trajectory from the resting motion curve, map the motion duration to the duration of the transition audio curve, and determine the pitch change trajectory based on the change of the displacement trajectory over time; based on the duration and pitch change trajectory, a transition audio curve is generated by an audio synthesis engine. For example, in response to the resting motion curve being a long-period, slow stretching motion trajectory, the processing device can generate a long glissando waveform with a long envelope and a smooth pitch change from low to high.

[0164] In some embodiments, the processing device can control the robot's motors to drive the corresponding movable parts to move according to the action duration corresponding to the rest movement curve and the displacement trajectory corresponding to each movable part of the robot, so that the robot can perform a rest movement.

[0165] In some embodiments, the processing device can generate a transition audio signal according to the audio waveform value corresponding to each moment in the transition audio curve, and send the transition audio signal to the robot's speaker to output the transition audio through the speaker.

[0166] In some embodiments, the processing device may record the start time of the resting motion curve and determine the execution duration of the resting motion curve based on the time difference between the current time and the start time. In response to the execution duration reaching the corresponding action duration of the resting motion curve, the processing device may determine that the resting motion execution has ended.

[0167] In some embodiments, in response to determining that the resting motion has ended, the processing device can reset the body offset to a preset initial value so that the subsequent motor motion curve can be regenerated with reference to the normal posture reference. The preset initial value can be zero or a pre-set initial offset.

[0168] In some embodiments, in response to the completion of the rest movement, the processing device may terminate the current interaction process, no longer resume the execution of audio waveform curves that were not completed before the rest movement was triggered, and put the robot into a waiting-to-interact state to await receiving the next external stimulus information. In response to receiving the next external stimulus information, the processing device may re-execute steps 310 to 360 to initiate a new interaction process.

[0169] In some embodiments of this specification, by generating a resting motion curve and a transition audio curve when the body posture offset reaches a preset offset threshold, the robot can naturally release the fatigue accumulated during long-term operation through coordinated restorative movements and transition audio. By resetting the body posture offset after the resting motion is completed, subsequent movements can be generated again with a normal posture reference. Thus, while maintaining the continuity and naturalness of the robot's fatigue expression, it is possible to avoid motion distortion or mechanical operation risks caused by the infinite accumulation of body posture offset.

[0170] This specification provides one or more embodiments of an apparatus for controlling a robot, the apparatus including at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least a portion of the computer instructions to implement a method for controlling the robot.

[0171] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0172] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0173] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0174] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0175] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0176] For each patent, patent application, patent application publication, and other material such as articles, books, specifications, publications, and documents referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0177] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A method for controlling a robot, characterized in that, The method includes: Based on the external stimulus information received by the robot, the interaction intention is determined; Based on the external stimulus information, the interaction intention, and the interaction frequency corresponding to the interaction intention, expression parameters are determined, wherein the expression parameters include emotional polarity, energy level, and expected duration. Based on the expressed parameters, an audio waveform curve and a motor motion curve are generated; The first characteristic peak of the audio waveform curve and the second characteristic peak of the motor motion curve are extracted respectively. Based on the first characteristic peak and the second characteristic peak, the motor motion curve is adjusted so that the time difference between the first characteristic peak and the adjusted second characteristic peak of the motor motion curve is less than a preset tolerance threshold; and The robot is controlled to output audio based on the audio waveform curve, and the robot is controlled to move based on the adjusted motor motion curve.

2. The method according to claim 1, characterized in that, In the process of controlling the robot to output audio based on the audio waveform curve and controlling the robot to move based on the adjusted motor motion curve, the method further includes: In response to receiving an interrupt signal, the interrupt time is determined; In response to the interruption time being no later than the time corresponding to the second characteristic peak of the adjusted motor motion curve, a low-pass filter is applied to the first curve segment of the audio waveform curve located after the interruption time, and the robot is controlled to output audio based on the filtered first curve segment. Simultaneously, a reverse braking torque pulse is applied to the robot's motor to rapidly decelerate the motor and suppress the robot from continuing its current motion; or In response to the interruption time being later than the time corresponding to the second characteristic peak of the adjusted motor motion curve, the fundamental frequency of the first curve segment is reduced by a preset amplitude, and the robot is controlled to output audio based on the first curve segment after the fundamental frequency is reduced. At the same time, a smooth reset curve is generated according to a preset damping coefficient, and the robot is controlled to move based on the smooth reset curve.

3. The method according to claim 2, characterized in that, The method further includes: In response to the detection that the robot's motor has stalled due to external force, the timing of the stall is determined; Determine the residual kinetic energy integral corresponding to the second curve segment in the adjusted motor motion curve that is located after the stall moment; Based on the residual kinetic energy integral, the stagnation compensation coefficient is determined; Based on the aforementioned hysteresis compensation coefficient, the fundamental frequency boost and harmonic gain of the third curve segment in the audio waveform curve located after the stall moment are determined. Based on the aforementioned fundamental frequency boost, the fundamental frequency of the third curve segment is increased; and Based on the harmonic gain, the high-frequency harmonic gain of the third curve segment is increased.

4. The method according to claim 1, characterized in that, The step of generating audio waveform curves and motor motion curves based on the expression parameters includes: Based on the expression parameters, the audio waveform curve is generated, and the audio features of the audio waveform curve are extracted; Based on the stated interaction intent, the basic trajectory is determined; Based on the aforementioned audio characteristics, determine the dynamic modulation parameters; and The motor motion curve is generated based on the basic trajectory and the dynamic modulation parameters.

5. The method according to claim 4, characterized in that, Before extracting the first characteristic peak of the audio waveform curve and the second characteristic peak of the motor motion curve, the method further includes: Based on the robot's cumulative operational characteristics, the body posture offset is determined; Based on the body posture offset, adjust the attitude reference of the motor motion curve; Based on the aforementioned cumulative operating characteristics, the additional noise component is determined; and The additional noise component is superimposed onto the audio waveform curve.

6. The method according to claim 1, characterized in that, The step of adjusting the motor motion curve based on the first characteristic peak and the second characteristic peak includes: In response to the presence of multiple local audio peaks in the audio waveform curve, an audio peak time series is determined based on the time corresponding to each of the multiple local audio peaks. In response to the existence of multiple local motion peaks in the motor motion curve, a motion peak time series is determined based on the time corresponding to each of the multiple local motion peaks. Based on the audio peak time series and the motion peak time series, the multiple local motion peaks are filtered or merged to determine multiple target motion peaks corresponding to the multiple local audio peaks; and Using the audio peak time series and the corresponding peaks among the plurality of target motion peaks as anchor points, the motor motion curve is transformed and smoothed based on the anchor points to obtain the adjusted motor motion curve, so that the time difference between the plurality of local audio peaks and the corresponding target motion peaks in the adjusted motor motion curve is less than the preset tolerance threshold.

7. The method according to claim 1, characterized in that, The control of the robot to output audio based on the audio waveform curve, and the control of the robot to move based on the adjusted motor motion curve, include: Based on the hardware latency compensation table, the response latency of the audio output link and the motor execution link is determined; and Based on the response delay of the audio output link and the motor execution link, delay compensation is performed on at least one of the audio waveform curve and the adjusted motor motion curve, and the robot is controlled to perform the audio output and the motion so that the audio output and the motion are synchronized.

8. The method according to claim 1, characterized in that, The determination of expression parameters based on the external stimulus information, the interaction intent, and the interaction frequency corresponding to the interaction intent includes: Based on the external stimulus information, the interaction intention, and the interaction frequency corresponding to the interaction intention, the emotional polarity, initial energy level, and expected duration are determined. Based on the robot's cumulative operational characteristics, a virtual endurance value is determined; and Based on the virtual endurance value, the initial energy level is adjusted to determine the energy level in the expression parameters.

9. A system for controlling a robot, comprising a determining module, a generating module, an extracting module, an adjusting module, and a controlling module; The determining module is configured to determine the interaction intent based on the external stimulus information received by the robot; The determining module is further configured to determine expression parameters based on the external stimulus information, the interaction intent, and the interaction frequency corresponding to the interaction intent, wherein... The expression parameters include emotional polarity, energy level, and expected duration; The generation module is configured to generate an audio waveform curve and a motor motion curve based on the expression parameters; The extraction module is configured to extract the first characteristic peak of the audio waveform curve and the second characteristic peak of the motor motion curve, respectively. The adjustment module is configured to adjust the motor motion curve based on the first characteristic peak and the second characteristic peak, so that the time difference between the first characteristic peak and the second characteristic peak of the adjusted motor motion curve is less than a preset tolerance threshold. The control module is configured to control the robot to output audio based on the audio waveform curve, and to control the robot to move based on the adjusted motor motion curve.

10. A device for controlling a robot, characterized in that, The device includes at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is configured to execute at least a portion of the computer instructions to implement the method as described in any one of claims 1 to 8.

Images