Motion choreography method, electronic device, and storage medium

Abstract

The present disclosure provides a motion choreography method, an electronic device and a storage medium, and relates to the technical field of robots. The motion choreography method comprises: obtaining a first action frame sequence and a second action frame sequence of a robot, the first action frame sequence comprising a plurality of basic action frames, the second action frame sequence comprising a plurality of semantic action frames, and the deduction time range of the second action frame sequence overlapping with the deduction time range of the first action frame sequence; calculating the similarity of the plurality of basic action frames and the plurality of semantic action frames, and determining a target basic action frame and a target semantic action frame based on the similarity of the plurality of basic action frames and the plurality of semantic action frames; generating a first transition frame sequence between the target basic action frame and the target semantic action frame; and generating a target action frame sequence of the robot based on the first action frame sequence, the second action frame sequence and the first transition frame sequence.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, specifically to a motion choreography method, electronic device, and storage medium. Background Technology

[0002] Currently, robots are increasingly widely used in service, education, and companionship. Humans use gestures and other body language to enhance semantic transmission and emotional resonance during voice communication, content explanation, or emotional expression. Therefore, to improve the naturalness of human-computer interaction, robots need to possess the ability to coordinate audio content with physical actions.

[0003] In current audio interpretation solutions, the timing alignment between the robot's body movements and the audio content is insufficient, resulting in poor movement continuity and a lack of interactive immersion. Summary of the Invention

[0004] In view of this, this disclosure provides a motion choreography method, an electronic device, and a storage medium to improve the human-computer interaction experience.

[0005] In a first aspect, a motion choreography method is provided for application to a robot. The method includes: acquiring a first motion frame sequence and a second motion frame sequence of the robot, wherein the first motion frame sequence includes multiple basic motion frames, the second motion frame sequence includes multiple semantic motion frames, and the evolution time range of the second motion frame sequence overlaps with that of the first motion frame sequence; calculating the similarity between the multiple basic motion frames and the multiple semantic motion frames, and determining a target basic motion frame and a target semantic motion frame based on the similarity between the multiple basic motion frames and the multiple semantic motion frames; generating a first transition frame sequence between the target basic motion frame and the target semantic motion frame; and generating a target motion frame sequence of the robot based on the first motion frame sequence, the second motion frame sequence, and the first transition frame sequence.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, generating a target action frame sequence for the robot based on a first action frame sequence, a second action frame sequence, and a first transition frame sequence includes: acquiring the audio to be performed by the robot, wherein the first action frame sequence is determined based on the audio to be performed; determining the prosodic information of the audio to be performed, wherein the prosodic information characterizes the audio energy distribution of the audio to be performed; determining the accent time point corresponding to the audio to be performed based on the prosodic information; and generating the target action frame sequence for the robot based on the accent time point, the first action frame sequence, the second action frame sequence, and the first transition frame sequence.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, generating the robot's target action frame sequence based on the accent time point, the first action frame sequence, the second action frame sequence, and the first transition frame sequence includes: if the accent time point falls within the deduction time range of the second action frame sequence, then extracting a semantic action frame fragment from the second action frame sequence, the semantic action frame fragment including semantic action frames between the target semantic action frame and the semantic action frame corresponding to the accent time point; generating a second transition frame sequence between the semantic action frame fragment and the first action frame sequence; and concatenating the first action frame sequence, the semantic action frame fragment, the first transition frame sequence, and the second transition frame sequence according to the deduction time range to obtain the robot's target action frame sequence.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the audio to be interpreted includes multiple audio frames; determining the accent time point corresponding to the audio to be interpreted based on prosodic information includes: determining the audio energy of each of the multiple audio frames based on prosodic information; selecting multiple target audio frames that meet the high-energy condition from the multiple audio frames based on the audio energy of each of the multiple audio frames; determining continuous high-energy audio segments based on the multiple target audio frames; and determining the accent time point based on the continuous high-energy audio segments.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the robot includes multiple motion joints. Calculating the similarity between multiple basic action frames and multiple semantic action frames includes: selecting a target joint from the multiple motion joints, the target joint including the robot's upper limb joints; and determining the similarity between the semantic action frame and the basic action frame based on the Euclidean distance between the joint angle of the target joint corresponding to the semantic action frame and the joint angle of the target joint corresponding to the basic action frame.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the multiple semantic action frames include a semantic action start frame, which represents the beginning of the semantic action. Determining the target semantic action frame based on the similarity between the multiple basic action frames and the multiple semantic action frames includes: determining a first number of semantic action frames in the second action frame sequence preceding the semantic action start frame; determining the semantic action start frame and the semantic action frame with the highest similarity among the first number of semantic action frames based on the similarity between the semantic action start frame and each of the first number of semantic action frames as candidate semantic action frames; and determining the candidate semantic action frame as the target semantic action frame if the candidate semantic action frame meets the matching conditions.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: when the candidate semantic action frames do not meet the matching conditions, determining a second number of semantic action frames located after the semantic action start frame in the second action frame sequence; and based on the similarity corresponding to each of the second number of semantic action frames, determining the semantic action frame with the highest similarity among the second number of semantic action frames as the target semantic action frame.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the first transition frame sequence includes multiple first transition frames. The method further includes generating a third number of interpolated frames between adjacent first transition frames, the third number being determined by the robot's motor drive frequency.

[0013] In a second aspect, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the action orchestration method provided in the first aspect by executing the executable instructions.

[0014] Thirdly, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the action orchestration method provided in the first aspect.

[0015] The disclosed motion orchestration method calculates the similarity between basic motion frames and semantic motion frames, uses this similarity to filter target basic motion frames and target semantic motion frames, and generates a first transition frame sequence between them. The addition of semantic motion frames improves the adaptability of robot actions to audio semantics. The first transition frame sequence eliminates abrupt pose changes, achieving a smooth transition between semantic and basic actions, avoiding abrupt robot action switching, and improving motion coherence and naturalness, thus enhancing the human-computer interaction experience. Attached Figure Description

[0016] Figure 1 The diagram shown illustrates a scenario applicable to an embodiment of this disclosure.

[0017] Figure 2 The diagram shown is a flowchart of an action choreography method provided in an embodiment of this disclosure.

[0018] Figure 3 The diagram shows a flowchart illustrating the steps of generating a target action frame sequence for a robot based on a first action frame sequence, a second action frame sequence, and a first transition frame sequence, according to an embodiment of this disclosure.

[0019] Figure 4 The diagram shows a flowchart illustrating the steps of generating a robot's target action frame sequence based on accent time points, a first action frame sequence, a second action frame sequence, and a first transition frame sequence, according to an embodiment of this disclosure.

[0020] Figure 5 The diagram shown is a flowchart illustrating the steps for determining the accent time points corresponding to the audio to be performed based on prosodic information, according to an embodiment of this disclosure.

[0021] Figure 6 The diagram shown is a flowchart illustrating the steps for calculating the similarity between multiple basic action frames and multiple semantic action frames according to an embodiment of this disclosure.

[0022] Figure 7 The diagram shows a flowchart illustrating the steps for determining a target semantic action frame based on the similarity between multiple basic action frames and multiple semantic action frames, according to an embodiment of this disclosure.

[0023] Figure 8 The diagram shown is a flowchart of an action arrangement method provided in another embodiment of this disclosure.

[0024] Figure 9 The diagram shown is a schematic diagram of the structure of an action choreography device provided in an embodiment of this disclosure.

[0025] Figure 10 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0026] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0027] Humanoid robots are intelligent and complex systems that mimic human appearance and function, highly integrating artificial intelligence and robotics technology. With the development of humanoid robot technology, they are gradually entering fields such as service, education, and companionship.

[0028] For example, robots can be used in exhibition halls to greet guests and provide guided tours. They play audio introductions and, during playback, perform corresponding actions with their limbs. Related technologies offer gesture generation schemes based on diffusion models (such as DSG and DIDiffGes), capable of synthesizing gestures based on audio. However, this scheme only focuses on the rhythmic matching of actions and audio melodies; the meaning of the gestures has poor correlation with the semantics of the audio. For instance, when the audio says "Look ahead," the robot might perform a "wave," completely deviating from the semantic requirements.

[0029] To address this issue, related technologies employ a "predefined action library + fixed-time trigger" approach. For example, a "pointing" action is triggered when the audio plays for 5 seconds. However, this results in direct switching between predefined actions and basic actions (such as standby, slow movement, or gestures synthesized based on the aforementioned gesture generation scheme), without smooth transitions, leading to noticeably abrupt movements.

[0030] Furthermore, in long-duration interaction scenarios, the robot continuously executes basic action sequences, while semantic actions (such as the "pointing" action mentioned above) need to be precisely inserted based on the semantic nodes of the audio content. The lack of a coordinated mechanism for temporal localization and action adaptation in related technologies leads to a mismatch between the timing of semantic action insertion and the state of the basic actions.

[0031] For example, the above scheme relies on timestamps to trigger semantic actions, without considering the real-time state of the basic action sequence: at the 5th second of the audio, the robot's hand is behind its body in the basic action. Jumping directly to the "pointing" action at this time would cause the hand to swing rapidly from behind to the front, and the joint movement trajectory does not conform to human action logic, making it difficult to meet the actual needs of immersive interaction scenarios. Furthermore, when calling semantic actions from the predefined action library, complete action segments are often directly extracted and inserted without selecting the most matching starting frame of the semantic action based on the current frame state of the basic gesture: for example, if the hand is in a "chest-level position" in the basic action, but the extracted semantic action is inserted with "the hand is raised from the waist" as the starting frame, this would disrupt the continuity of the basic action sequence.

[0032] To address the aforementioned technical problems, this disclosure provides a motion orchestration method applied to robots. The motion orchestration method includes: acquiring a first motion frame sequence and a second motion frame sequence for the robot, wherein the first motion frame sequence includes multiple basic motion frames, and the second motion frame sequence includes multiple semantic motion frames, and the evolution time range of the second motion frame sequence overlaps with that of the first motion frame sequence; calculating the similarity between the multiple basic motion frames and the multiple semantic motion frames, and determining target basic motion frames and target semantic motion frames based on the similarity between the multiple basic motion frames and the multiple semantic motion frames; generating a first transition frame sequence between the target basic motion frames and the target semantic motion frames; and generating a target motion frame sequence for the robot based on the first motion frame sequence, the second motion frame sequence, and the first transition frame sequence.

[0033] The following is a combination of... Figure 1 This invention introduces an application scenario to which an embodiment of the present disclosure is applicable.

[0034] Figure 1 The diagram illustrates a scenario applicable to an embodiment of this disclosure. Figure 1 As shown, the scenario includes terminal 110 and server 120, and terminal 110 communicates with server 120.

[0035] Terminal 110 may include electronic devices such as mobile phones, laptops, or personal computers. Users input and edit the first action frame sequence and the second action frame sequence through terminal 110. For example, users can annotate the start time points and action start frames of the first and second action frame sequences. Terminal 110 sends the first and second action frame sequences to server 120.

[0036] Server 120 can be a single server, a server cluster consisting of multiple servers, or a cloud server. Alternatively, server 120 can be integrated into a terminal. Server 120 performs orchestration processing on the first action frame sequence and the second action frame sequence based on the action orchestration method provided in this disclosure to generate a target action frame sequence. Then, server 120 sends the target action frame sequence to terminal 110 for user confirmation.

[0037] This scenario can also include robots. Users store a sequence of target action frames in the robot's storage medium. When the robot performs a corresponding task, it executes this sequence of target action frames to improve the naturalness and immersion of human-computer interaction. The robot can be a humanoid robot, with joint designs that enable anthropomorphic interactive movements.

[0038] The aforementioned target action frame sequence can be played in conjunction with audio playback, and this sequence is correlated with the audio. For example, in a presentation scenario, the robot can play audio descriptions of exhibits while simultaneously executing the target action frame sequence. For instance, when introducing an exhibit, the robot's arm joints might perform a "pointing" motion to guide visitors. Furthermore, during the presentation, the robot will perform corresponding actions in conjunction with the audio content; for example, when the word "time" appears in the audio, the robot's arm joints might perform a "raising wrist to check the watch" motion, thus enabling the robot to intelligently interpret and humanize the audio, enhancing the human-computer interaction experience.

[0039] The motion arrangement method provided in this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] First, this disclosure provides an action programming method that can be executed by any electronic device with computing capabilities.

[0041] Figure 2 The diagram shown is a flowchart illustrating an action orchestration method provided in an embodiment of this disclosure. Figure 2 As shown, the action choreography method provided in this embodiment includes the following steps.

[0042] S210, acquire the robot's first action frame sequence and second action frame sequence.

[0043] Both the first and second action frame sequences are sets of motion data used to control the robot to perform specified actions, and consist of multiple consecutive action frames. An action frame is the smallest data unit constituting the first and second action frame sequences, containing parameters such as the angle parameters of each joint of the robot, the duration of the action, and the joint movement speed. Each basic action frame corresponds to a momentary limb posture of the robot.

[0044] Specifically, the first action frame sequence includes multiple basic action frames. These basic action frames are used to control the robot to perform basic actions, which are typically not associated with specific semantics; for example, basic actions may include standby, slow movement, etc. Alternatively, they can be determined based on the audio to be interpreted, for example, based on the aforementioned gesture generation schemes based on diffusion models (such as DSG and DIDiffGes), generating basic actions that match the rhythm and melody of the audio. DSG is based on a Markov chain-based denoising gesture diffusion model, while DIDiffGes is based on a semi-implicit adversarial denoising diffusion model.

[0045] The audio to be interpreted is audio data played during the robot's execution of the first action frame sequence and the second action frame sequence (or the target action frame sequence described below). There is a correspondence between the first action frame sequence, the second action frame sequence, and the audio to be interpreted.

[0046] The second action frame sequence includes multiple semantic action frames. These semantic action frames control the robot to perform semantic actions, which are matched with the audio semantics in the audio to be interpreted. For example, if the audio semantics are "Please look at the screen," the corresponding semantic action is "pointing at an object"; if the audio semantics are "time," the corresponding semantic action is "raising your wrist to check your watch"; if the audio semantics are a greeting, the corresponding semantic action is "waving." The above examples are only used to illustrate the matching relationship between semantic actions and the audio semantics in the audio to be interpreted. The correspondence between semantic actions and audio semantics can be determined according to the actual application scenario, and this disclosure does not impose specific limitations on this.

[0047] The execution time range of the second action frame sequence overlaps with that of the first action frame sequence. The execution time range is the time interval from the start to the end of the execution of the first and second action frame sequences.

[0048] Specifically, the execution time of the first action frame sequence covers most or even all of the playback time of the audio to be performed. That is, during the playback of the audio to be performed, the robot will perform a complete audio interpretation along with the audio to achieve an immersive interactive experience.

[0049] The second action frame sequence corresponds to a specific audio segment in the audio to be interpreted, and therefore only covers a portion of the playback time of the audio to be interpreted. Within the second action frame sequence, the interpretation time of a specific semantic action frame should be aligned with a specific point in time in the audio to be interpreted. For example, if the second action frame sequence corresponds to the action of "raising the wrist to check the watch," the point in time when the audio to be interpreted begins playing the word "time" should be aligned with the interpretation time of the semantic action frame corresponding to the action of "raising the wrist upwards."

[0050] For example, if the length of the audio to be performed is 10 minutes, the performance time range of the first action frame sequence is 0:00-10:00, and the performance time range of the second action frame sequence is 5:00-5:20. It is understood that the performance time range of the second action frame sequence can be manually marked or automatically located through audio analysis, etc., and this disclosure does not impose specific limitations on this.

[0051] S220, calculate the similarity between multiple basic action frames and multiple semantic action frames, and determine the target basic action frame and the target semantic action frame based on the similarity between multiple basic action frames and multiple semantic action frames.

[0052] Similarity characterizes the degree of similarity between the limb postures of a base action frame and a semantic action frame. Similarity can be determined by the difference in joint angles between the base action frame and the semantic action frame. The smaller the difference, the higher the similarity between the base action frame and the semantic action frame.

[0053] Based on the similarity between multiple basic action frames and multiple semantic action frames, a target basic action frame is selected from the multiple basic action frames, and a target semantic action frame is selected from the multiple semantic action frames.

[0054] For example, the pair of basic action frames and semantic action frames with the highest similarity can be selected from multiple basic action frames and multiple semantic action frames as the target basic action frame and the target semantic action frame. For instance, if the hand is in a "horizontal position in front of the chest" in the basic action frame and the hand is also in a "horizontal position in front of the chest" in the semantic action frame, then the basic action frame is selected as the target basic action frame and the semantic action frame is selected as the target semantic action frame.

[0055] Alternatively, first select multiple pairs of basic action frames and semantic action frames with similarity higher than a preset similarity threshold, and then select one pair as the target basic action frame and target semantic action frame based on other constraints.

[0056] S230, Generate the first transition frame sequence between the target base action frame and the target semantic action frame.

[0057] After determining the target base action frame and the target semantic action frame, a first transition frame sequence between the target base action frame and the target semantic action frame is generated.

[0058] The first transition frame sequence is a set of motion data used to connect the target basic motion frame and the target semantic motion frame. The joint angles of the first transition frame sequence change continuously, which can achieve a smooth transition between the target basic motion frame and the target semantic motion frame, avoid abrupt changes in limb posture between the target basic motion frame and the target semantic motion frame, make the robot's motion switching smoother and more natural, and avoid stiffness and jerkiness caused by abrupt changes in joint angles, as well as hardware wear and tear.

[0059] The first transition frame sequence includes multiple first transition frames, which can be generated using a linear interpolation method. For example, the linear interpolation method can employ the following formula: in, This represents the joint angle corresponding to the k-th first transition frame. This represents the joint angle corresponding to the target's base motion frame. The joint angle corresponding to the target semantic action frame is represented by k, where N represents the number of the first transition frames and k = 1, 2, ..., N.

[0060] or, It could also be the joint angle corresponding to the target semantic action frame, and correspondingly, This indicates the joint angle corresponding to the target's basic motion frame, and this disclosure does not impose specific limitations on it.

[0061] The number of the first transition frames can be preset. For example, it can be set to 10. At a frame rate of 30 fps, the execution time of the first transition frame sequence is approximately 0.33 seconds, which is consistent with the human visual perception threshold for the smoothness of robot movements.

[0062] S240, based on the first action frame sequence, the second action frame sequence and the first transition frame sequence, generate the target action frame sequence of the robot.

[0063] The first action frame sequence, the second action frame sequence, and the first transition frame sequence are spliced ​​together according to the interpretation time range to obtain the target action frame sequence.

[0064] Specifically, the first transition frame sequence is used to connect the target base action frame to the beginning of the target semantic action frame. At this time, the base action frames in the first action frame sequence that are before the target base action frame, the first transition frame sequence, and the semantic action frames in the second action frame sequence that are after the target semantic action frame are concatenated.

[0065] In addition, when the second action frame sequence ends, the transition frame sequence between the second action frame sequence and the subsequent multiple basic action frames can also be obtained through the above steps, which will not be elaborated here.

[0066] In this way, the target action frame sequence of the robot is obtained.

[0067] In this embodiment, the similarity between basic action frames and semantic action frames is calculated, and target basic action frames and target semantic action frames are filtered based on the similarity. A first transition frame sequence between the two is then generated. The addition of semantic action frames improves the adaptability of robot actions to audio semantics. The first transition frame sequence eliminates abrupt changes in posture, achieving a smooth transition between semantic actions and basic actions, avoiding abrupt robot action switching, and improving the continuity and naturalness of actions, thereby enhancing the human-computer interaction experience.

[0068] To further improve the coordination between robot movements and audio, this disclosure provides an optional embodiment, the specific implementation of which is described below.

[0069] Figure 3 The diagram shows a flowchart illustrating the steps of generating a robot's target action frame sequence based on a first action frame sequence, a second action frame sequence, and a first transition frame sequence, according to an embodiment of this disclosure. Figure 3 As shown, the steps for generating the robot's target action frame sequence based on the first action frame sequence, the second action frame sequence, and the first transition frame sequence include the following steps.

[0070] S241, Obtain the audio to be performed by the robot.

[0071] The first action frame sequence is determined based on the audio to be performed.

[0072] The audio to be played is the audio data that the robot needs to play synchronously during the interaction. For example, the audio to be played could be an audio guide in an exhibition hall or museum, or a children's story in a child-accompaniment scenario.

[0073] S242, Determine the prosodic information of the audio to be performed.

[0074] Prosodic information represents the distribution of audio energy in the audio to be interpreted. Prosodic information includes features such as rhythm and dynamic variations in the audio to be interpreted.

[0075] S243, based on prosodic information, determine the accent time points corresponding to the audio to be performed.

[0076] The accent time point is the time point in the audio to be interpreted with a higher audio energy value, corresponding to the accented part of the audio (such as speech accent, keywords, strong musical beats, etc.).

[0077] The timing of accents can be determined based on prosodic information. For example, an audio energy threshold can be set, and the time points when the audio energy exceeds the threshold can be determined as accent points.

[0078] S244, based on the accent time point, the first action frame sequence, the second action frame sequence, and the first transition frame sequence, generate the robot's target action frame sequence.

[0079] In the first action frame sequence, the robot's actions are clearly presented at the accented time points to enhance the rhythmic coordination between the basic actions and the audio to be performed.

[0080] The accented timing may fall within the interpretation range of the second action frame sequence. The accented timing occurs when the semantic action is about to end. At this point, directly replacing the corresponding part in the first action frame sequence with a semantic action frame may cause a rhythmic transition problem between the semantic action and the basic action. For example, the robot may suddenly switch from a relatively gentle semantic action to a more rapid basic action, thereby reducing the coordination between the robot's actions and the audio rhythm.

[0081] Therefore, in the integration process of the first action frame sequence, the second action frame sequence and the first transition frame sequence, the prosodic information of the audio to be performed is considered simultaneously in the present embodiment.

[0082] If the accented timing falls within the execution time range of the second action frame sequence, it is first necessary to determine the location of the accented timing. If it falls on a non-key segment such as the end of a semantic action, the second action frame sequence can be extracted based on the accented timing.

[0083] Specifically, key segments of semantic actions can be manually labeled in advance. For example, the semantic action corresponding to the second action frame sequence is the action of "raising the wrist to look at the watch", which includes frames 1-50. Among them, frames 1-9 are the preparatory actions of the semantic action, frames 10-50 are the complete action segments of the semantic action, and frame 30 is the action of "raising the wrist to the chest". At this time, frame 30 is labeled as the key action frame, frames 10-30 are designated as key segments, and frames 31-50 are designated as non-key segments.

[0084] If the accented timing falls within the performance timeframe of a non-critical segment, the last few frames of the second action frame sequence can be appropriately deleted. The last few frames of the second action frame sequence are usually not the key actions of the semantic action; deleting them will not affect the core meaning of the semantic action while fully preserving the basic actions corresponding to the accented portion.

[0085] Alternatively, if the stressed punctuation point falls within the performance timeframe of a key segment, truncating the semantic action at this time might affect its core meaning. Therefore, semantic actions can be used to cover the basic action corresponding to the stressed punctuation point, thereby improving the adaptability between robot actions and audio semantics.

[0086] In this embodiment of the disclosure, the prosodic information of the audio to be interpreted is further considered during the generation of the target action frame sequence, and the distribution of stress within the semantic action is optimized so that the target action frame sequence can take into account both semantic integrity and audio prosodic synergy.

[0087] The following section details the specific implementation method of frame sequence integration based on accent time points.

[0088] Figure 4 The diagram shown is a flowchart illustrating the steps of generating a robot's target action frame sequence based on accent time points, a first action frame sequence, a second action frame sequence, and a first transition frame sequence, according to an embodiment of this disclosure. Figure 4 As shown, the steps for generating the robot's target action frame sequence based on the accent time point, the first action frame sequence, the second action frame sequence, and the first transition frame sequence include the following steps.

[0089] S2441, if the accent time point falls within the interpretation time range of the second action frame sequence, then extract the semantic action frame segment from the second action frame sequence.

[0090] Here, the interpretation time range of the second action frame sequence refers to the interpretation time of the semantic action. It is worth noting that in the above embodiment, in order to optimize the connection between the semantic action and the basic action, the second action frame sequence was appropriately truncated. Therefore, the interpretation time range of the current second action frame sequence has changed accordingly compared with the original interpretation time range.

[0091] The semantic action frame segment includes semantic action frames between the target semantic action frame and the semantic action frame corresponding to the accent time point. Among them, the semantic action frame corresponding to the accent time point refers to the semantic action frame in the second action frame sequence whose derivation time completely coincides with or is closest to the accent time point.

[0092] Starting from the target semantic action frame and ending at the semantic action frame corresponding to the accent time point, multiple semantic action frames between the two are extracted as semantic action frame segments.

[0093] S2442, Generate a second transition frame sequence between the semantic action frame fragment and the first action frame sequence.

[0094] In this step, the similarity between multiple semantic action frames in the semantic action frame segment and multiple basic action frames located near the accent time point in the first action frame sequence can be calculated respectively. Based on the similarity, appropriate semantic action frames and basic action frames are selected, and a second transition frame sequence is generated.

[0095] The specific method for generating the second transition frame sequence can refer to the method for generating the first transition frame sequence in the above embodiments, and will not be repeated here.

[0096] S2443, the first action frame sequence, semantic action frame fragment, first transition frame sequence and second transition frame sequence are spliced ​​together according to the deduction time range to obtain the robot's target action frame sequence.

[0097] Determine the respective interpretation time ranges of the first action frame sequence, semantic action frame fragment, first transition frame sequence, and second transition frame sequence, and splice them together according to the interpretation time order.

[0098] For example, the target action frame sequence is obtained by sequentially splicing together the basic action frames that are located before the first transition frame sequence in the first action frame sequence, the first transition frame sequence, the semantic action frame fragment, the second transition frame sequence, and the basic action frames that are located after the second transition frame sequence in the first action frame sequence.

[0099] In this embodiment of the disclosure, firstly, the robot performs basic actions following the rhythm of the audio to be interpreted in the first half of the first action frame sequence; then, the robot performs transition actions from basic actions to semantic actions, and performs semantic actions that are adapted to the audio semantics of the audio to be interpreted; finally, the robot performs transition actions from semantic actions to basic actions, and continues to perform basic actions following the rhythm of the audio to be interpreted.

[0100] In this process, the basic actions corresponding to the stressed parts in the audio, as well as the key segments of semantic actions, can be preserved, so that the target action frame sequence can take into account both semantic integrity and audio prosodic coordination, while ensuring smooth connection between multiple frame sequences.

[0101] Figure 5 The diagram shown is a flowchart illustrating the steps for determining the accent time points corresponding to the audio to be performed based on prosodic information, according to an embodiment of this disclosure. Figure 5 As shown, the steps for determining the accent time points corresponding to the audio to be performed based on prosodic information include the following steps.

[0102] S2431, based on prosody information, determines the audio energy of each of the multiple audio frames.

[0103] The audio to be performed consists of multiple audio frames. An audio frame is the smallest unit of audio data obtained after short-time framing of the audio to be performed, and each frame corresponds to a fixed duration. For example, the audio to be performed is framing based on a frame rate of 30 fps.

[0104] Next, the audio energy of each of the multiple audio frames is calculated. This step can be achieved using the Short-Time Fourier Transform (STFT), and the specific formula is as follows: in, This represents the audio energy of the t-th frame. Let F represent the spectral amplitude at the f-th frequency point in the t-th frame, and let F represent the number of spectral frequency points.

[0105] S2432, based on the audio energy of each of the multiple audio frames, selects multiple target audio frames that meet the high energy condition from the multiple audio frames.

[0106] The high-energy condition can be defined as the audio energy of the audio frame being greater than a high-energy threshold. This high-energy threshold can be determined using the following formula: in, Indicates a high energy threshold. This represents the average audio energy across multiple audio frames. The standard deviation of the audio energy across multiple audio frames. For example, as a preset constant, It can be 1.5.

[0107] Audio frames whose audio energy exceeds the high energy threshold are identified as target audio frames.

[0108] S2433 identifies continuous high-energy audio segments based on multiple target audio frames.

[0109] A continuous high-energy audio segment is an audio segment composed of multiple target audio frames that are consecutive in time. It is an audio segment in the audio to be interpreted that truly has accent features.

[0110] Specifically, multiple target audio frames can be sorted according to their frame start time (or frame index), and the sorted target audio frames can be traversed to determine the temporal continuity between two adjacent target audio frames.

[0111] Target audio frames that are sequential in time are integrated into a single continuous segment. Then, continuous segments with a frame count greater than or equal to a preset number of frames are grouped into a single continuous high-energy audio segment. For example, the preset frame count can be 3 frames, meaning that continuous segments with a frame count greater than or equal to 3 frames are grouped into a single continuous high-energy audio segment.

[0112] S2434 determines the accent timing based on continuous high-energy audio segments.

[0113] The starting frame of a series of high-energy audio segments can be used as the accent time point. This ensures that the underlying actions corresponding to the series of high-energy audio segments remain intact in subsequent steps.

[0114] In this step, the frame index of the starting frame can be mapped to the actual timestamp based on the audio sampling rate and frame step size to improve the accuracy of the accent timing.

[0115] In this embodiment of the disclosure, the audio energy of each audio frame is first calculated based on prosody information, then the target audio frame is filtered based on the high energy condition, and continuous high-energy audio segments are obtained, so as to avoid misjudging scattered noise frames as accents and improve the accuracy of accent timing.

[0116] The above embodiments describe how the target basic action frame and the target semantic action frame can be determined based on the similarity between the basic action frame and the semantic action frame. In order to further improve the accuracy of the target basic action frame and the target semantic action frame, and at the same time simplify the similarity calculation process, this disclosure provides an optional embodiment, the specific implementation of which is as follows.

[0117] Figure 6 The diagram shown is a flowchart illustrating the steps for calculating the similarity between multiple basic action frames and multiple semantic action frames according to an embodiment of this disclosure. Figure 6 As shown, the steps for calculating the similarity between multiple basic action frames and multiple semantic action frames include the following steps.

[0118] S610 selects the target joint from multiple moving joints.

[0119] The robot includes multiple motion joints. For example, motion joints include the robot's shoulder joint, elbow joint, wrist joint, hip joint, knee joint, ankle joint, etc.

[0120] During audio interpretation by the robot, greater attention is paid to upper limb movements. Therefore, when calculating similarity, target joints can be selected from multiple motion joints. Target joints include the robot's upper limb joints, specifically the shoulder joint, elbow joint, and wrist joint.

[0121] The target joint is the main part of the semantic action, and its angle change directly determines the core form of the semantic action, while non-core joints such as lower limb joints have little impact on the semantic action recognition.

[0122] S620, based on the Euclidean distance between the joint angle of the target joint corresponding to the semantic action frame and the joint angle of the target joint corresponding to the basic action frame, determine the similarity between the semantic action frame and the basic action frame.

[0123] For any semantic action frame and a base action frame, calculate the Euclidean distance between the joint angles of the target joints corresponding to the semantic action frame and the joint angles of the target joints corresponding to the base action frame, respectively. This distance is used as the similarity between the semantic action frame and the base action frame. The formula for calculating the similarity can be expressed as: in, The similarity is represented by upper_index, which represents the set of indexes of the target joints. p1 is the joint angle sequence of the target joints corresponding to the base action frame, and p2 is the joint angle sequence of the target joints corresponding to the semantic action frame.

[0124] In this embodiment, the upper limb joints are first selected as target joints from the robot's motion joints to avoid parameter interference from non-core joints such as the lower limb joints, thus improving calculation accuracy. Simultaneously, this reduces computational redundancy and improves operational efficiency.

[0125] The following section will introduce the specific implementation method of determining the target semantic action frame from multiple semantic action frames based on similarity.

[0126] Figure 7 The diagram shown is a flowchart illustrating the steps for determining a target semantic action frame based on the similarity between multiple basic action frames and multiple semantic action frames, according to an embodiment of this disclosure. Figure 7 As shown, the steps to determine the target semantic action frame based on the similarity between multiple basic action frames and multiple semantic action frames include the following steps.

[0127] S710, determine the first number of semantic action frames in the second action frame sequence that are located before the semantic action start frame.

[0128] The semantic action frames include a semantic action start frame, which represents the start of the semantic action.

[0129] Referring again to the example in the above embodiment, the semantic action corresponding to the second action frame sequence is the action of "raising the wrist to look at the watch", which includes frames 1-50; among them, frames 1-9 are the preparatory actions for the semantic action, and frames 10-50 are the complete action segments of the semantic action. At this time, frame 10 is determined as the starting frame of the semantic action.

[0130] It is worth noting that the semantic action start frame is not the first action frame in the second action frame sequence, but rather the first action frame that can define the complete segment of the semantic action. For example, frames 1-9 of the action "raise your wrist to check the watch" are preparatory actions, including raising the arm from a hanging position. This action may also be a preparatory action for other semantic actions, and even if the entire segment is deleted, it will not affect the semantics expressed by the remaining segments.

[0131] Determine the first number of semantic action frames preceding the semantic action start frame. This first number can be preset. Continuing with the example above, the first number could be 9, and if the 10th frame is the semantic action start frame, then the first number of semantic action frames would be frames 1-9.

[0132] S720, based on the semantic action start frame and the similarity of the first number of semantic action frames, the semantic action frame with the highest similarity among the first number of semantic action frames is determined as the candidate semantic action frame.

[0133] The similarity between the semantic action start frame and the first number of semantic action frames and multiple basic action frames is determined, and the semantic action frame with the highest similarity is determined as the candidate semantic action frame.

[0134] Understandably, we can first determine the derivation time corresponding to the semantic action start frame; then, we can determine multiple basic action frames that fall within a certain range before and after the derivation time, and determine the similarity between these basic action frames and the semantic action start frame and the first number of semantic action frames, respectively, in order to further reduce the amount of computation.

[0135] S730: If the candidate semantic action frame meets the matching conditions, the candidate semantic action frame is determined as the target semantic action frame.

[0136] Matching criteria may include a similarity score higher than a preset matching threshold for candidate semantic action frames. If the similarity score of a candidate semantic action frame is higher than the preset matching threshold, then the candidate semantic action frame is determined as the target semantic action frame.

[0137] In this embodiment of the disclosure, the semantic action frame before the semantic action start frame is the starting action before the semantic action. Selecting a suitable frame from it as the target semantic action frame can ensure the integrity of the semantic action and thus improve the human-computer interaction experience.

[0138] Figure 8 The diagram shown is a flowchart illustrating an action orchestration method provided in another embodiment of this disclosure. Figure 7 Extending from the illustrated embodiment Figure 8 The illustrated embodiment will be described in detail below. Figure 8 The illustrated embodiments and Figure 7 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0139] like Figure 8 As shown, after step (S720) of determining the semantic action frame with the highest similarity among the first number of semantic action frames based on their respective similarities, the action orchestration method further includes the following steps.

[0140] S810, if the candidate semantic action frame does not meet the matching condition, determine the second number of semantic action frames in the second action frame sequence that are located after the semantic action start frame.

[0141] If a candidate semantic action frame does not meet the matching conditions, it means that the first number of semantic action frames before the semantic action start frame are significantly different from multiple basic action frames and cannot be used as the target semantic action frame. In this case, a supplementary filtering mechanism needs to be activated.

[0142] Specifically, determine a second number of semantic action frames following the semantic action start frame. This second number can be preset, and can be the same as or different from the first number. Continuing with the example above, the first number can be 9, and if frame 10 is the semantic action start frame, then the second number of semantic action frames are frames 11-19.

[0143] S820, based on the similarity of each of the second number of semantic action frames, the semantic action frame with the highest similarity among the second number of semantic action frames is determined as the target semantic action frame.

[0144] Select the frame with the highest similarity from the second number of semantic action frames and determine it as the target semantic action frame.

[0145] In this embodiment, after candidate frame adaptation fails before the starting frame, the target semantic action frame is selected from semantic action frames following the semantic action starting frame. While this sacrifices some semantic action integrity, it avoids interrupting the orchestration process due to preceding frame matching failures, ensuring the continuity of action orchestration. Furthermore, by adjusting the second quantity setting, the loss of key action frames can be avoided, preserving the core semantics of the semantic action.

[0146] In some embodiments, after generating a first transition frame sequence between the target base action frame and the target semantic action frame, the action orchestration method further includes generating a third number of interpolation frames between adjacent first transition frames of the target.

[0147] The first transition frame sequence includes multiple first transition frames. Target adjacent first transition frames are any two temporally consecutive first transition frames in the first transition frame sequence.

[0148] Between consecutive first transition frames, a third number of interpolation frames are generated using an interpolation method. The third number is determined by the robot's motor drive frequency.

[0149] For example, if the robot's motor drive frequency is 240 Hz and the first transition frame generation frequency is 30 fps, in order to ensure that the position commands received by the motor are continuous and without gaps, seven interpolation frames can be inserted between adjacent first transition frames to further improve the smoothness of the robot's movements and avoid joint speed exceeding limits due to "frame loss".

[0150] The interpolated frame can be obtained through fifth-order polynomial interpolation, as shown in the following formula: in, - These are polynomial coefficients. The polynomial coefficients are solved using the start and end frames of the first transition frame adjacent to the target as a reference, with the continuity of start and end positions, velocities, and accelerations as boundary conditions. - After optimization, the robot's position curves are seamlessly connected, its velocity curves are first-order differentiable, and its acceleration curves have no spikes, ensuring that the robot operates continuously without joint vibration or shutdown protection, meeting the needs of long-term interaction scenarios.

[0151] The motion choreography method provided in the above embodiments can effectively improve the audio performance of robots. Specifically, this solution outperforms related technical solutions in terms of audio-gesture beat alignment accuracy, semantic-action matching accuracy, naturalness of motion transitions, and stability in long-term interaction scenarios.

[0152] Furthermore, user feedback indicates that the actions generated by this solution are "without noticeable lag," "understandable in terms of the audio's intended meaning," and "like a real conversation during interaction." It is evident that this solution surpasses related technologies in terms of naturalness of action, immersive audio-action experience, and clarity of semantic expression, thereby improving the user's human-computer interaction experience and satisfaction.

[0153] The above text combined Figures 1 to 8 The method embodiments of this disclosure have been described in detail below, in conjunction with... Figure 9 The apparatus embodiments of this disclosure are described in detail below. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0154] Figure 9 The diagram shown is a structural schematic of an action choreography device provided in an embodiment of this disclosure. Figure 9 As shown, the motion choreography device 900 of this embodiment includes: an acquisition module 910, a calculation module 920, a first generation module 930, and a second generation module 940.

[0155] The acquisition module 910 is configured to acquire a first action frame sequence and a second action frame sequence of the robot. The first action frame sequence includes multiple basic action frames, and the second action frame sequence includes multiple semantic action frames. The derivation time range of the second action frame sequence overlaps with the derivation time range of the first action frame sequence.

[0156] The calculation module 920 is configured to calculate the similarity between multiple base action frames and multiple semantic action frames, and to determine the target base action frame and the target semantic action frame based on the similarity between the multiple base action frames and the multiple semantic action frames.

[0157] The first generation module 930 is configured to generate a first transition frame sequence between the target base action frame and the target semantic action frame.

[0158] The second generation module 940 is configured to generate a target action frame sequence for the robot based on the first action frame sequence, the second action frame sequence, and the first transition frame sequence.

[0159] In some embodiments, the second generation module 940 is further configured to: acquire the audio to be performed by the robot, wherein the first action frame sequence is determined based on the audio to be performed; determine the prosodic information of the audio to be performed, wherein the prosodic information characterizes the audio energy distribution of the audio to be performed; determine the accent time point corresponding to the audio to be performed based on the prosodic information; and generate the target action frame sequence of the robot based on the accent time point, the first action frame sequence, the second action frame sequence, and the first transition frame sequence.

[0160] In some embodiments, the second generation module 940 is further configured to: if the accent time point falls within the derivation time range of the second action frame sequence, extract a semantic action frame fragment from the second action frame sequence, the semantic action frame fragment including the semantic action frames between the target semantic action frame and the semantic action frame corresponding to the accent time point; generate a second transition frame sequence between the semantic action frame fragment and the first action frame sequence; and splice the first action frame sequence, the semantic action frame fragment, the first transition frame sequence and the second transition frame sequence according to the derivation time range to obtain the robot's target action frame sequence.

[0161] In some embodiments, the second generation module 940 is further configured to: determine the audio energy of each of the multiple audio frames based on prosody information; select multiple target audio frames that meet the high-energy condition from the multiple audio frames based on the audio energy of each of the multiple audio frames; determine a continuous high-energy audio segment based on the multiple target audio frames; and determine the accent time point based on the continuous high-energy audio segment.

[0162] In some embodiments, the calculation module 920 is further configured to select a target joint from a plurality of motion joints, the target joint including the upper limb joint of the robot; and determine the similarity between the semantic action frame and the basic action frame based on the Euclidean distance between the joint angle of the target joint corresponding to the semantic action frame and the joint angle of the target joint corresponding to the basic action frame.

[0163] In some embodiments, the calculation module 920 is further configured to: determine a first number of semantic action frames in the second action frame sequence that are located before the semantic action start frame; based on the similarity of the semantic action start frame and the first number of semantic action frames, determine the semantic action start frame and the semantic action frame with the highest similarity among the first number of semantic action frames as candidate semantic action frames; and determine the candidate semantic action frame as the target semantic action frame if the candidate semantic action frame meets the matching conditions.

[0164] In some embodiments, the calculation module 920 is further configured to, when the candidate semantic action frames do not meet the matching conditions, determine a second number of semantic action frames located after the semantic action start frame in the second action frame sequence; and, based on the similarity of each of the second number of semantic action frames, determine the semantic action frame with the highest similarity among the second number of semantic action frames as the target semantic action frame.

[0165] In some embodiments, the first generation module 930 is further configured to generate a third number of interpolation frames between adjacent first transition frames of the target, the third number being determined by the motor drive frequency of the robot.

[0166] Below, for reference Figure 10 To describe an electronic device according to embodiments of the present disclosure. Figure 10 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this disclosure. Figure 10 As shown, the electronic device 1000 includes one or more processors 1010 and memory 1020.

[0167] The processor 1010 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.

[0168] The memory 1020 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1010 may execute the program instructions to implement the action orchestration methods of the various embodiments of this disclosure described above and / or other desired functions.

[0169] In some embodiments, the electronic device 1000 may further include an input device 1030 and an output device 1040, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0170] The input device 1030 may include, for example, a touch screen, microphone, keyboard, mouse, etc. The output device 1040 may include, for example, a display, speaker, and communication network and its connected remote output devices, etc.

[0171] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 1000 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1000 may include any other suitable components depending on the specific application.

[0172] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the action orchestration methods according to various embodiments of this disclosure described above.

[0173] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0174] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the action orchestration methods according to various embodiments of this disclosure described above.

[0175] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0176] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0177] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0178] It should also be noted that in the systems, apparatus, and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0179] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0180] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for choreographing movements, characterized in that, Applied to robots, the method includes: The robot acquires a first action frame sequence and a second action frame sequence. The first action frame sequence includes multiple basic action frames, and the second action frame sequence includes multiple semantic action frames. The interpretation time range of the second action frame sequence overlaps with the interpretation time range of the first action frame sequence. The first action frame sequence is determined based on the audio to be interpreted. Calculate the similarity between the plurality of basic action frames and the plurality of semantic action frames, and determine the target basic action frame and the target semantic action frame based on the similarity between the plurality of basic action frames and the plurality of semantic action frames; Generate a first transition frame sequence between the target basic action frame and the target semantic action frame; If the accent time point of the audio to be interpreted falls within the interpretation time range of the second action frame sequence, then a semantic action frame segment is extracted from the second action frame sequence. The semantic action frame segment includes the semantic action frames between the target semantic action frame and the semantic action frame corresponding to the accent time point. Generate a second transition frame sequence between the semantic action frame fragment and the first action frame sequence; The first action frame sequence, the semantic action frame fragment, the first transition frame sequence, and the second transition frame sequence are spliced ​​together according to the derivation time range to obtain the target action frame sequence of the robot.

2. The method according to claim 1, characterized in that, Before extracting semantic action frame segments from the second action frame sequence if the accent time of the audio to be interpreted falls within the interpretation time range of the second action frame sequence, the method further includes: Obtain the audio to be performed by the robot; Determine the prosodic information of the audio to be interpreted, wherein the prosodic information characterizes the audio energy distribution of the audio to be interpreted; Based on the prosodic information, the accent time points corresponding to the audio to be performed are determined.

3. The method according to claim 2, characterized in that, The audio to be interpreted includes multiple audio frames; determining the accent time points corresponding to the audio to be interpreted based on the prosodic information includes: Based on the prosody information, the audio energy of each of the plurality of audio frames is determined; Based on the audio energy of each of the multiple audio frames, select multiple target audio frames that meet the high energy condition from the multiple audio frames; Based on the multiple target audio frames, a continuous high-energy audio segment is determined; The accent time points are determined based on the continuous high-energy audio segments.

4. The method according to claim 1, characterized in that, The robot includes multiple motion joints; calculating the similarity between the multiple basic action frames and the multiple semantic action frames includes: Select a target joint from the plurality of motion joints, the target joint including the upper limb joint of the robot; The similarity between the semantic action frame and the basic action frame is determined based on the Euclidean distance between the joint angle of the target joint corresponding to the semantic action frame and the joint angle of the target joint corresponding to the basic action frame.

5. The method according to claim 1, characterized in that, The plurality of semantic action frames includes a semantic action start frame, which represents the start of a semantic action; Based on the similarity between the plurality of basic action frames and the plurality of semantic action frames, the target semantic action frame is determined, including: Determine a first number of semantic action frames in the second action frame sequence that precede the semantic action start frame; Based on the similarity of the semantic action start frame and the first number of semantic action frames, the semantic action start frame and the semantic action frame with the highest similarity among the first number of semantic action frames are determined as candidate semantic action frames. If the candidate semantic action frame meets the matching conditions, the candidate semantic action frame is determined as the target semantic action frame.

6. The method according to claim 5, characterized in that, Also includes: If the candidate semantic action frame does not meet the matching condition, determine a second number of semantic action frames in the second action frame sequence that are located after the semantic action start frame; Based on the similarity of each of the second number of semantic action frames, the semantic action frame with the highest similarity among the second number of semantic action frames is determined as the target semantic action frame.

7. The method according to claim 1, characterized in that, The first transition frame sequence includes a plurality of first transition frames; the method further includes: A third number of interpolation frames are generated between adjacent first transition frames of the target, the third number being determined by the motor drive frequency of the robot.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the action orchestration method of any one of claims 1 to 7 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the action arrangement method according to any one of claims 1 to 7.

Images