A method, apparatus, device and medium for aligning robot action and audio

Abstract

The application provides a robot action and audio alignment method, device, equipment and medium, the method comprises: obtaining a composite sequence instruction for a robot, the composite sequence instruction comprising an action list and an audio list; performing format conversion on each audio in the audio list to construct an audio item; binding the audio item and the corresponding action in the action list to obtain a play group; performing time length alignment on the action and the audio in the play group according to the actual time length of the audio and the original time length of the corresponding action in the play group to obtain a time length aligned play group; and controlling the robot to play the audio and perform the corresponding action based on the aligned play group. The problem of different synchronization of voice and action at the beginning and end in the robot performance process is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically, to a method, apparatus, device, and medium for aligning robot actions with audio. Background Technology

[0002] When service robots and performing robots are deployed in business scenarios such as greeting guests, explaining exhibits, and performing role-playing, it is usually necessary for the robot's physical movements and voice audio content to work together. For example, a waving gesture may match a welcoming message, a pointing gesture may correspond to the dialogue in an exhibit explanation, and a set of physical performances may be accompanied by background music. Currently, the industry mainly addresses the linkage and coordination between robot movements and audio by binding a single audio file to a single set of movement trajectories at the program level. When the movement is triggered, a designated player is automatically invoked to load and play the bound audio synchronously.

[0003] However, multithreading can only achieve loosely coupled parallel operation of actions and audio. The timing of the two is independent, and they can only start synchronously but cannot be dynamically calibrated and aligned. In addition, the synthesis time of text to speech (TTS) is affected by factors such as text content, network status, and switching playback engines. Fixed parameter arrangement is prone to obvious timing misalignment, such as the body action not finishing after the voice playback is completed, or the action ending prematurely before the audio is finished. It is impossible to achieve precise timing of key actions and semantic nodes of speech, and the anthropomorphic performance presentation effect is greatly reduced. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of the prior art by providing a method, apparatus, device, and medium for aligning robot actions and audio. This method acquires a composite sequence of instructions containing actions and audio, standardizes and encapsulates audio items, binds them to corresponding actions to generate playback groups, and finally aligns the durations of the audio and the corresponding actions based on their actual durations. Ultimately, this drives the robot to synchronously execute actions and play audio, thus solving the problem of asynchronous speech and actions during robot performances.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a method for aligning robot actions with audio, the method comprising: Obtain a composite sequence of instructions for the robot, the composite sequence of instructions including an action list and an audio list; The audio files in the audio list are converted to their original formats to create audio items. Bind the audio item to the corresponding action in the action list to obtain a playback group; Based on the actual duration of the audio in the playback group and the original duration of the corresponding action, the actions and audio in the playback group are time-aligned to obtain the time-aligned playback group. Based on the aligned playback group, the robot is controlled to play audio and perform corresponding actions.

[0006] In an optional implementation, the step of aligning the actions and audio in the playback group according to the actual duration of the audio in the playback group and the original duration of the corresponding actions to obtain a duration-aligned playback group includes: Based on the actual duration of the audio in the playback group and the original duration of the corresponding action, the original duration of the corresponding action is adjusted to obtain the adjusted duration of the corresponding action, wherein the adjusted duration of the corresponding action is the same as the actual duration of the audio. Based on the actual duration of the audio and the corresponding action with the adjusted duration, the duration-aligned playback group is constructed.

[0007] In an optional implementation, the audio item includes a semantic anchor table, which records the action keyframe identifiers corresponding to target audio points in the audio. The step of adjusting the original duration of the corresponding action based on the actual duration of the audio in the playback group and the original duration of the corresponding action to obtain the adjusted duration of the corresponding action includes: If there is an action keyframe identifier in the semantic anchor table, then the audio segment of the audio and the corresponding action segment of the audio segment are determined according to the action keyframe identifier. The frame rate of the corresponding action segment is determined based on the actual duration of the audio segment and the number of frames of the corresponding action segment. Based on the frame rate of the corresponding action segment, the original duration of the corresponding action segment is adjusted to obtain the adjusted duration of the corresponding action.

[0008] In an optional implementation, the step of adjusting the original duration of the corresponding action based on the actual duration of the audio in the playback group and the original duration of the corresponding action to obtain the adjusted duration of the corresponding action further includes: If there is no action keyframe identifier in the semantic anchor table, the scaling factor is determined based on the actual duration of the audio and the original duration of the corresponding action. Based on the scaling factor, the original duration of the corresponding action is adjusted to obtain the adjusted duration of the corresponding action.

[0009] In an optional implementation, the method further includes: If the adjusted duration of the corresponding action is lower than the minimum duration constraint value of the corresponding action, then the maximum duration of the corresponding action is obtained. The pause duration of the audio is determined based on the maximum duration of the corresponding action; The playback duration of the audio is determined based on the pause duration and the actual duration. Based on the audio of the playback duration and the corresponding action of the maximum duration, the playback group with the duration alignment is constructed.

[0010] In an optional implementation, the audio item includes: an audio type, and the method further includes: Based on the audio type, determine the target player; Based on the aligned playback group, controlling the robot to play audio and perform corresponding actions includes: Based on the aligned playback group, the robot is controlled to play audio using the target player.

[0011] In an optional implementation, the method further includes: If an interrupt command is received, the target playback group corresponding to the interrupt command, as well as the motion trajectory and audio cursor in the target playback group, are recorded. If a recovery command is received, the recovery playback group is determined based on the motion trajectory and audio cursor in the target playback group.

[0012] Secondly, embodiments of this application also provide a robot motion and audio alignment device, the device comprising: The acquisition module is used to acquire composite sequence instructions for the robot, the composite sequence instructions including an action list and an audio list; The construction module is used to convert the format of each audio file in the audio list and construct audio items; The binding module is used to bind the audio item to the corresponding action in the action list to obtain a playback group; The alignment module is used to align the actions and audio in the playback group according to the actual duration of the audio in the playback group and the original duration of the corresponding actions, so as to obtain the playback group after duration alignment. The control module is used to control the robot to play audio and perform corresponding actions based on the aligned playback group.

[0013] Thirdly, embodiments of this application also provide a control device, including: a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the control device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the robot action and audio alignment method as described in any of the first aspects.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot action and audio alignment method as described in any of the first aspects.

[0015] The beneficial effects of this application are: This application provides a method, apparatus, device, and medium for aligning robot actions and audio. The method includes: acquiring a composite sequence of instructions for the robot, the composite sequence of instructions including an action list and an audio list; converting the format of each audio in the audio list to construct an audio item; binding the audio item to the corresponding action in the action list to obtain a playback group; aligning the actions and audio in the playback group according to the actual duration of the audio in the playback group and the original duration of the corresponding action to obtain a duration-aligned playback group; and controlling the robot to play audio and execute corresponding actions based on the aligned playback group.

[0016] The method of this application obtains a composite sequence of instructions containing actions and audio, standardizes and encapsulates audio items and binds them to corresponding actions to generate playback groups, and finally aligns the durations of the audio and the corresponding actions according to their actual durations. Ultimately, it drives the robot to execute actions and play audio synchronously, effectively solving the problem of asynchronous voice and actions during robot performances, and enabling the entire robot's interaction process to have a complete and feasible standardized execution chain. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 One of the flowcharts for a robot motion and audio alignment method provided in this application embodiment; Figure 2 A second schematic flowchart illustrating a method for aligning robot actions with audio, provided as an embodiment of this application; Figure 3 The third flowchart illustrates a method for aligning robot actions with audio, as provided in this application embodiment. Figure 4 The fourth flowchart illustrates a method for aligning robot actions with audio, as provided in this application embodiment. Figure 5 The fifth flowchart illustrates a method for aligning robot actions with audio, as provided in this application embodiment. Figure 6 A flowchart illustrating a method for aligning robot actions with audio, provided as an embodiment of this application, is shown in Figure 6. Figure 7 The seventh flowchart illustrates a method for aligning robot actions with audio, as provided in this application embodiment. Figure 8 This is a functional module diagram of a robot motion and audio alignment device provided in an embodiment of this application; Figure 9 This is a schematic diagram of a control device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0020] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] In the description of this application, it should be noted that if the terms "upper", "lower", etc. appear to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in, it is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0022] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0024] To address the issue of audio-action mismatch in existing robots during performance scenarios, this application provides a method for aligning robot actions with audio. This method is executed by a control system within the robot control device. The control system includes: a multimodal sequence parsing and anchor point labeling unit, a heterogeneous audio routing unit, a runtime duration detection and timeline negotiation unit, a flexible alignment scheduling engine (including an action publishing thread and an audio scheduling thread), and a hierarchical interrupt control unit. The control system aligns the actions and audio in the playback group based on the actual duration of the audio in the playback group and the original duration of the corresponding action, thereby resolving the issue of audio-action mismatch in existing robots during performance scenarios.

[0025] The robot motion and audio alignment method provided in this application will be explained in detail below with reference to the accompanying drawings and specific examples. Figure 1 This is one of the flowcharts illustrating a method for aligning robot actions with audio, as provided in an embodiment of this application; Figure 1 As shown, the method includes: S101. Obtain the compound sequence instructions for the robot.

[0026] The compound sequence instruction includes an action list and an audio list.

[0027] In this embodiment, the robot's host computer or cloud scheduling platform will issue corresponding composite sequence instructions according to different performance tasks. The composite sequence instructions include an action list and an audio list. The performance tasks include greeting guests, explaining exhibits, and scene interaction.

[0028] The action list is a sequence of physical joint motion trajectories preset by the robot, including interactive actions such as waving, raising, pointing, turning, and returning to position. Each action corresponds to parameters such as a fixed number of frames, original running frame rate, joint speed and acceleration threshold.

[0029] The audio list includes locally stored audio files, such as "Hello" and "Welcome," as well as real-time TTS text content. Upon receiving a compound sequence command, the system first verifies the command's validity and then caches the complete action-audio mapping.

[0030] S102. Convert the format of each audio file in the audio list and construct an audio item.

[0031] Specifically, the multimodal sequence parsing and anchor point annotation unit traverses all audio resources in the audio list, performs differentiated format conversions for different types of audio, and obtains the audio items corresponding to each audio.

[0032] For local audio files, basic information such as file extension, storage path, original duration, and sampling rate are read. For TTS text audio, parameters such as complete playback text, preset playback speed, and language are extracted. All heterogeneous audio is then uniformly encapsulated into a standardized audio queue, `audio_items`. Each audio item in the standardized audio queue contains fixed fields: audio type (type, distinguishing between local files and real-time TTS synthesis), audio content (content, corresponding to the file path or playback text), sleep time (sleep_after, a custom pause duration between segments), and a semantic anchor table. The semantic anchor table records the keyframe identifier `keyframe_id` corresponding to the target audio point in the audio.

[0033] After normalization and encapsulation, the entire sequence has editing capabilities, allowing users to freely add, delete, and sort multiple audio entries, insert custom silence pauses at any audio node, and achieve point-to-point synchronization by binding action keyframes with semantic anchor tables. Finally, it splices together an integrated complex shared timeline containing multimodal temporal constraints, which can adapt to various long-process, multi-interactive robot performance tasks without secondary modifications to the underlying scheduling logic.

[0034] S103. Bind the audio item to the corresponding action in the action list to obtain the playback group.

[0035] Specifically, the multimodal sequence parsing and anchor annotation unit establishes a mapping relationship between the audio items in the composite sequence instructions and the actions in the action list based on the arrangement order of the composite sequence instructions and the preset audio-action association table, thus completing the basic binding.

[0036] For performance scenarios with interactive requirements, users can complete the annotation in the semantic anchor table of the audio item in advance, record the time offset of key semantic nodes in the voice text, and bind the key frame identifier keyframe_id corresponding to the action trajectory. For example, in the audio anchor of "(pointing) this way please", bind the highest frame of the robot raising its hand. The system will synchronously record the binding relationship into the playback group data structure.

[0037] S104. Based on the actual duration of the audio in the playback group and the original duration of the corresponding action, perform duration alignment on the actions and audio in the playback group to obtain the duration-aligned playback group.

[0038] Specifically, the runtime duration detection and time axis negotiation unit and the elastic alignment scheduling engine work together to obtain the actual duration of the audio in the current playback group and the original duration of the corresponding action before the current playback group is executed. The action and audio are then time-aligned to obtain the current playback group after the duration is aligned.

[0039] First, the audio scheduling thread detects the actual duration T_audio of the audio in the current playback group. For local audio files, the accurate duration can be obtained by directly reading the file container header information. For TTS audio, the actual duration is inferred from the sampling rate and number of bytes returned by the synthesis interface. In streaming synthesis scenarios, the duration is first estimated using the text length and preset speech rate, and then corrected in real time during playback. At the same time, the action publishing thread calculates the original duration T_motion of the action based on the total number of frames in the action trajectory and the original 100Hz default frame rate.

[0040] Based on the actual duration T_audio and the original duration T_motion, the actions and audio in the playback group are time-aligned, ultimately achieving end-to-end convergence alignment between actions and audio. This eliminates the misalignment problem of the hand not stopping after the speech ends or the speech not ending after the hand stops, and finally generates a playback group with matching duration, i.e., a playback group after duration alignment.

[0041] S105. Based on the aligned playback group, control the robot to play audio and perform corresponding actions.

[0042] Specifically, the elastic alignment scheduling engine's dual-thread collaborative drive drives an independent and non-blocking action publishing thread and audio scheduling thread. The audio scheduling thread reads the audio parameters of the aligned playback group, calls the adapted backend player, and plays the audio according to the standardized duration and semantic anchor point time. The action publishing thread reads the action parameters of the aligned playback group, issues action commands frame by frame, and drives the robot joints to execute the corresponding trajectory movements.

[0043] It should be noted that during the execution of the current playback group, the next audio segment is pre-synthesized and the next motion trajectory is pre-loaded to eliminate inter-segment silences and ensure a continuous and smooth viewing experience for multiple performance tasks.

[0044] In summary, this application provides a method for aligning robot actions with audio. The method includes: acquiring a composite sequence of instructions for the robot, the composite sequence including an action list and an audio list; converting the format of each audio item in the audio list to construct an audio item; binding the audio item to the corresponding action in the action list to obtain a playback group; aligning the actions and audio in the playback group according to the actual duration of the audio in the playback group and the original duration of the corresponding action to obtain a duration-aligned playback group; and controlling the robot to play audio and execute corresponding actions based on the aligned playback group.

[0045] The method of this application obtains a composite sequence of instructions containing actions and audio, standardizes and encapsulates audio items and binds them to corresponding actions to generate playback groups, and finally aligns the durations of the audio and the corresponding actions according to their actual durations. Ultimately, it drives the robot to execute actions and play audio synchronously, effectively solving the problem of asynchronous voice and actions during robot performances, and enabling the entire robot's interaction process to have a complete and feasible standardized execution chain.

[0046] Based on the above embodiments, this application also provides another possible implementation of the robot action and audio alignment method. Figure 2 This is a second flowchart illustrating a method for aligning robot actions and audio according to an embodiment of this application. Figure 2 As shown, based on the actual duration of the audio in the playback group and the original duration of the corresponding action, the actions and audio in the playback group are time-aligned to obtain the time-aligned playback group, including: S201. Based on the actual duration of the audio in the playback group and the original duration of the corresponding action, adjust the original duration of the corresponding action to obtain the adjusted duration of the corresponding action.

[0047] The adjusted duration of the corresponding action is the same as the actual duration of the audio.

[0048] S202. Based on the actual duration of the audio and the corresponding action after adjustment, construct a playback group with the duration aligned.

[0049] In this embodiment, the actual duration T_audio of the audio in the playback group and the original duration T_motion of the corresponding action are extracted. Based on the actual duration T_audio of the audio and the original duration T_motion of the corresponding action, the original duration T_motion of the corresponding action is adjusted to ensure that the adjusted duration of the corresponding action is the same as the actual duration of the audio, thereby achieving duration alignment of the actions and audio in the playback group.

[0050] Then, the motion trajectory data with the matched duration is re-integrated with the audio data of the actual duration, updating the core parameters of the playback group. This includes synchronously updating the motion delivery timeline, beat scaling factor, and audio playback duration parameters, while preserving the original motion trajectory frame content, audio content, and basic arrangement information. The reconstructed playback group achieves a perfect match between the total motion duration and the total audio duration, with no time misalignment or content tampering. It also retains all original scheduling attributes and can be directly executed by dual scheduling threads, meeting the basic synchronization requirements of general multimodal performance scenarios.

[0051] The method provided in this application embodiment adjusts the original duration of the corresponding action based on the actual duration of the audio in the playback group and the original duration of the corresponding action, resulting in an adjusted duration for the corresponding action. The adjusted duration of the corresponding action is the same as the actual duration of the audio. A playback group with synchronized durations is constructed based on the audio with its actual duration and the corresponding action with its adjusted duration. This method can quickly and accurately match the total duration of the entire action to the actual duration of the audio, rapidly achieving end-to-end basic alignment. While ensuring that the robot's motion trajectory remains undistorted, it eliminates segment-level action audio misalignment in a lightweight manner, adapting to low-customization scenarios such as large-scale routine explanations and welcoming announcements.

[0052] This application also provides another possible implementation of the robot action and audio alignment method, wherein the audio item includes a semantic anchor table, which is used to record the action keyframe identifiers corresponding to the target audio points in the audio. Figure 3 This is the third flowchart illustrating a method for aligning robot actions and audio according to an embodiment of this application. Figure 3 As shown, based on the actual duration of the audio in the playback group and the original duration of the corresponding action, the original duration of the corresponding action is adjusted to obtain the adjusted duration of the corresponding action, including: S301. If there is an action keyframe identifier in the semantic anchor table, then determine the audio segment and the corresponding action segment of the audio segment based on the action keyframe identifier.

[0053] In this embodiment, the elastic alignment scheduling engine traverses the semantic anchor point table of the current playback group. After detecting a valid keyframe_id keyframe identifier, the complete audio is divided into multiple continuous audio segments with the sounding time of each semantic anchor point as the segmentation node. Each audio segment corresponds to a specific semantic broadcast content.

[0054] Simultaneously, based on the keyframe identifiers, the corresponding keyframe positions in the motion trajectory are located. Using the keyframes as dividing points, the complete motion trajectory is broken down into motion segments that correspond one-to-one with the audio segments, ensuring that each semantic audio segment and each semantic node has a dedicated motion range to match it. For example, the anchor point "(waving) Welcome" corresponds to the audio segment from the start of the audio to the anchor point, matching the robot's hand-raising and waving motion segment, thus providing a precise alignment range for fine-tuning non-uniform time reshaping.

[0055] S302. Determine the frame rate of the corresponding action segment based on the actual duration of the audio segment and the number of frames of the corresponding action segment.

[0056] Specifically, the precise actual duration of each audio segment after segmentation is read segment by segment, and the total number of trajectory frames of the corresponding action segment is counted to generate a unique dynamic frame rate. The frame rate calculation formula is: Frame rate = Total number of frames of the corresponding action segment / Actual duration of the audio segment.

[0057] Unlike the traditional fixed 100Hz frame rate, this step configures an independent frame rate for each action segment bound to a semantic anchor point. Different audio segments and different action segments can correspond to different frame rates, adapting to the fast and slow rhythm of semantic playback and solving the problem that a fixed frame rate cannot match semantic nodes, causing the action and content to become disconnected.

[0058] For example, if a valid keyframe_id is detected, the complete audio is divided into two consecutive audio segments, and the corresponding action segments are determined. If the actual duration of the first audio segment is 3 seconds and the total number of frames in the corresponding action segment is 500, and the actual duration of the second audio segment is 7 seconds and the total number of frames in the corresponding action segment is 500, then the frame rate of the action segment corresponding to the first audio segment is determined to be 500 / 3, and the frame rate of the action segment corresponding to the second audio segment is determined to be 500 / 7.

[0059] S303. Adjust the original duration of the corresponding action segment according to the frame rate of the corresponding action segment to obtain the adjusted duration of the corresponding action.

[0060] Specifically, the execution duration of each action segment is adjusted according to the frame rate of each corresponding action segment, thereby adjusting the duration of the entire action and obtaining the adjusted duration of the corresponding action.

[0061] Fine-tuning the timing of intervals near anchor points ensures that key action frames are precisely timed to semantic broadcast nodes; non-key intervals are evenly adapted in duration to ensure smooth and natural overall movement. After adjustment, the total duration of the entire action segment is consistent with the total audio duration, and the core semantic nodes achieve precise synchronization between action and speech.

[0062] In the method provided in this application embodiment, audio segments and corresponding action segments are split according to semantic anchor points and segmented and adapted to independent frame rates to complete non-uniform duration adjustment. The synchronization dimension is upgraded from overall time alignment to precise binding at the content node level. Key action frames are forced to be triggered synchronously at the corresponding semantic nodes of the voice, which improves the semantic fit and human-like expressiveness of robot interaction and solves the defects of fixed frame rate synchronization being unable to match the semantic rhythm of the speech and key interactive actions being delayed or advanced.

[0063] This application also provides another possible implementation of the robot motion and audio alignment method. Figure 4 This is the fourth flowchart illustrating a method for aligning robot actions and audio provided in an embodiment of this application. Figure 4 As shown, based on the actual duration of the audio in the playback group and the original duration of the corresponding action, the original duration of the corresponding action is adjusted to obtain the adjusted duration of the corresponding action, which also includes: S401. If there is no action keyframe identifier in the semantic anchor table, the scaling factor is determined based on the actual duration of the audio and the original duration of the corresponding action.

[0064] In this embodiment, if the semantic anchor table of the current audio item is detected to be empty and there is no bound keyframe identifier, the actual duration T_audio of the current audio segment and the original duration T_motion of the corresponding action are directly retrieved, and the time scaling factor k is calculated uniformly using the ratio formula. The ratio formula is: k = T_audio / T_motion.

[0065] S402. Adjust the original duration of the corresponding action according to the scaling factor to obtain the adjusted duration of the corresponding action.

[0066] Specifically, the scaling factor k is applied to all trajectory frames of the entire motion segment, uniformly scaling and adjusting the original fixed frame rate of the motion to evenly modify the time interval between each frame's execution. When the audio playback speed is slow and the duration is long, the overall motion beat is stretched to allow the motion to complete slowly; when the audio playback speed is fast and the duration is short, the motion beat is compressed within the robot's joint speed and acceleration safety thresholds to speed up the motion execution. After adjustment, the total execution time of the entire motion segment perfectly matches the actual audio duration.

[0067] In the method provided in this application embodiment, when no anchor point identifier is configured, a global scaling factor is directly generated by the ratio of audio to the original action duration and the action playback duration is uniformly corrected. This eliminates the steps of segmentation and multi-segment parameter calculation, simplifies the scheduling engine's operation logic, reduces system computing power consumption, and balances running efficiency and basic timing alignment capabilities in use scenarios where fine-grained content synchronization is not required.

[0068] This application also provides another possible implementation of the robot motion and audio alignment method. Figure 5 This is the fifth flowchart illustrating a method for aligning robot actions and audio according to an embodiment of this application. Figure 5 As shown, the method also includes: S501. If the adjusted duration of the corresponding action is lower than the minimum duration constraint value of the corresponding action, then obtain the maximum duration of the corresponding action.

[0069] In this embodiment, the system incorporates physical constraint parameters such as the upper limit of speed, upper limit of acceleration, and minimum motion duration for each joint of the robot. After completing the recalculation of the action duration, it automatically verifies whether the adjusted duration is compliant, that is, whether the adjusted duration of the corresponding action is lower than the minimum duration constraint value of the corresponding action.

[0070] If the duration of an over-compressed motion falls below the minimum duration constraint for the corresponding motion, it will cause inter-frame displacement and motion speed to exceed the joint's physical threshold, which is considered an over-limit adjustment. In this case, the audio duration will no longer be forcibly adapted; instead, the physically feasible limit duration corresponding to the current motion trajectory will be retrieved, which is the shortest compliant duration for the robot joint to smoothly and safely execute this motion segment.

[0071] S502. Determine the pause duration of the audio based on the maximum duration of the corresponding action.

[0072] Specifically, the time difference between the action limit duration and the actual audio duration is calculated, and this difference is determined as the audio pause duration. The specific calculation logic is: pause duration = action limit duration - actual audio duration. This difference is the waiting time before the robot plays the audio. By actively increasing the audio pause time, the robot's physical movement limits are adapted, avoiding mechanical failures caused by forced acceleration.

[0073] S503. Determine the playback duration of the audio based on the pause duration and the actual duration.

[0074] The actual audio duration is superimposed with the dynamically calculated pause duration to obtain the corrected final audio playback duration. Simultaneously, this duration parameter is written to the shared timeline, updating the playback plan of the audio scheduling thread. The original audio playback content, speech rate, and semantic nodes remain unchanged; only a pre-playback silent pause is added.

[0075] S504. Based on the audio with playback duration and the corresponding action with maximum duration, construct a playback group with duration alignment.

[0076] The motion trajectory with the maximum duration and the corrected audio duration after overlapping pauses are integrated to reconstruct a complete playback group. The timeline parameters, timing beats, and inter-segment pause parameters within the playback group are updated to ensure that the audio also ends completely when the motion is completed, with the timing at both ends fully converged and aligned. This solution solves the alignment problem under dual constraints, ensuring both the robot's physical motion safety and completely eliminating modal misalignment issues, achieving a balance between safety and synchronization. In the method provided in this application embodiment, when the adjusted duration of the corresponding action is lower than the minimum duration constraint value of the corresponding action, the action trajectory is no longer forcibly compressed. The limit duration of the action that the robot can safely operate is locked, and the audio side adds an additional pre-pause for timing compensation to avoid damage to the equipment due to motor overload, sudden changes in posture, and motion jamming. At the same time, it is still ensured that the audio also ends completely when the action is completed. The two-way negotiation alignment mechanism will not fail due to hardware constraints, taking into account both equipment safety and multimodal timing consistency.

[0077] This application also provides another possible implementation of the robot action and audio alignment method, wherein the audio item includes: audio type, Figure 6This is the sixth flowchart illustrating a method for aligning robot actions and audio provided in this application. Figure 6 As shown, the method also includes: S601. Determine the target player based on the audio type.

[0078] Based on the aligned playback group, control the robot to play audio and perform corresponding actions, including: S602. Based on the aligned playback group, control the robot to use the target player to play audio.

[0079] In this embodiment, the system pre-configures a heterogeneous player candidate pool that includes TTS cloud / LAN services, the lightweight audio / video player ffplay, the media player in pure command-line mode cvlc, and the ALSA (Advanced Linux Sound Architecture) toolkit component aplay, enabling adaptive matching for different audio types. The system reads the type field of the audio item to complete player matching.

[0080] When the audio type is TTS, the low-latency cloud or local area network TTS synthesis service is preferentially called as the target player; when the audio type is a local file, the available system-level players in the candidate pool are polled according to the audio file extension (wav, mp3, etc.), and a player with compatible format and stable operation is selected as the target player; if the preferred player fails or the process is abnormal, it will automatically downgrade to the backup player to achieve fault tolerance in audio playback.

[0081] The heterogeneous audio routing unit starts the matched target player as a backend process, loads the standardized audio content, playback duration, and semantic anchor timing parameters within the aligned playback group. During playback, the target player synchronizes the audio playback progress to the shared timeline in real time, and links it with the action execution progress of the action publishing thread in real time to complete multimodal synchronization.

[0082] The method provided in this application embodiment automatically routes and matches the playback backend from a multi-player candidate pool according to the audio type, distinguishes and schedules TTS and local file audio, and supports abnormal player downgrade switching, constructing a highly fault-tolerant heterogeneous audio playback system. This avoids the problems of limited format adaptation of a single player and audio interruption due to process crashes, ensuring stable and continuous audio output. It also achieves audio-visual collaborative output with the aligned playback group, improving the overall system's environmental adaptability and operational robustness.

[0083] This application also provides another possible implementation of the robot motion and audio alignment method. Figure 7 This is the seventh flowchart illustrating a method for aligning robot actions and audio according to an embodiment of this application. Figure 7As shown, the method also includes: S701. If an interrupt command is received, record the target playback group corresponding to the interrupt command, as well as the motion trajectory and audio cursor in the target playback group.

[0084] In this embodiment, the system audio scheduling thread employs a 0.02s micro-loop fine-grained interruptible sleep mechanism to continuously poll the global playback status flag is_playing. When the system receives external interruption commands such as manual pause, scene switching, or emergency stop, it immediately sets is_playing to False and initiates the hierarchical interrupt handling logic.

[0085] Simultaneously, breakpoint data is accurately recorded, the target playback group currently being executed is located, and the motion trajectory cursor (currently issued action frame number, trajectory motion phase) and audio cursor (currently playing audio item number, audio playback progress, and triggered semantic anchor position) are recorded. All breakpoint data is cached to the system's local storage in real time.

[0086] Simultaneously, asymmetric hierarchical safety measures are implemented: the physical entity immediately issues a hard stop action, unloads the control node, automatically switches to a safe recovery posture and return trajectory, and avoids dangerous positions; the audio side performs volume fade-out processing and then smoothly terminates the playback subprocess to prevent popping sounds caused by sudden audio interruption.

[0087] S702. If a recovery command is received, determine the recovery playback group based on the motion trajectory and audio cursor in the target playback group.

[0088] After receiving an external command to resume the performance, the system retrieves locally cached breakpoint data. Based on the recorded target playback group, motion trajectory phase, and audio cursor position, it locates the nearest safety keyframe and corresponding audio playback node before the interruption. Starting from the safety keyframe, the system reconstructs and resumes the playback group, preserving the original motion alignment rules, semantic anchor binding relationships, and audio scheduling logic. It then resumes the execution of the incomplete motion trajectory and audio playback from the breakpoint position, without having to replay the entire performance from the beginning.

[0089] In the method provided in this application embodiment, after receiving the interruption command, the action and audio breakpoint cursor are retained, and the task can be directly resumed from the safety keyframe without repeating playback from the beginning, realizing the interruption resumption of the performance task and improving the task sustainability in human-computer interaction scenarios.

[0090] The robot motion and audio alignment device and control device provided in any of the above embodiments of this application will be explained below. The specific implementation process and the resulting technical effects are the same as those in the corresponding method embodiments. For the sake of brevity, the parts not mentioned in this embodiment can be referred to the corresponding content in the method embodiment.

[0091] Figure 8This is a functional module diagram of a robot motion and audio alignment device provided in an embodiment of this application. Figure 8 As shown, the robot motion and audio alignment device 100 includes: The acquisition module 110 is used to acquire composite sequence instructions for the robot, which include an action list and an audio list; Module 120 is used to convert the formats of each audio file in the audio list and construct audio items. Binding module 130 is used to bind audio items to corresponding actions in the action list to obtain playback groups; Alignment module 140 is used to align the actions and audio in the playback group according to the actual duration of the audio in the playback group and the original duration of the corresponding actions, so as to obtain the playback group after duration alignment. The control module 150 is used to control the robot to play audio and perform corresponding actions based on the aligned playback group.

[0092] Optionally, the device further includes: The adjustment module is used to adjust the original duration of the corresponding action based on the actual duration of the audio in the playback group and the original duration of the corresponding action, so as to obtain the adjusted duration of the corresponding action. The adjusted duration of the corresponding action is the same as the actual duration of the audio. Module 120 is also used to build a playback group with the duration aligned based on the audio of the actual duration and the corresponding action of the adjusted duration.

[0093] Optionally, the audio item includes a semantic anchor table, which records the action keyframe identifiers corresponding to the target audio points in the audio. The adjustment module is also used to determine the audio segment and the corresponding action segment of the audio segment based on the action keyframe identifier if there is an action keyframe identifier in the semantic anchor table; determine the frame rate of the corresponding action segment based on the actual duration of the audio segment and the number of frames of the corresponding action segment; and adjust the original duration of the corresponding action segment based on the frame rate of the corresponding action segment to obtain the adjusted duration of the corresponding action.

[0094] Optionally, the adjustment module is also used to determine the scaling factor based on the actual duration of the audio and the original duration of the corresponding action if there is no action keyframe identifier in the semantic anchor table; and to adjust the original duration of the corresponding action based on the scaling factor to obtain the adjusted duration of the corresponding action.

[0095] Optionally, the device further includes: The acquisition module 110 is also used to acquire the limit duration of the corresponding action if the adjusted duration of the corresponding action is lower than the minimum duration constraint value of the corresponding action. The determination module is used to determine the pause duration of the audio based on the limit duration of the corresponding action; and to determine the playback duration of the audio based on the pause duration and the actual duration. Module 120 is used to construct a duration-aligned playback group based on the audio with playback duration and the corresponding action with maximum duration.

[0096] Optionally, the audio item includes: audio type, a determining module, and is also used to determine the target player based on the audio type; The control module 150 is also used to control the robot to play audio using the target player based on the aligned playback group.

[0097] Optionally, the device further includes: The recording module is used to record the target playback group corresponding to the interrupt command, as well as the motion trajectory and audio cursor in the target playback group, if an interrupt command is received. The module determines the playback group to be restored based on the motion trajectory and audio cursor in the target playback group if a restoration command is received.

[0098] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0099] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0100] Figure 9 This is a schematic diagram of a control device provided in an embodiment of this application. This control device can be used to align robot movements with audio. Figure 9 As shown, the control device includes: processor 210, storage medium 220, and bus 230.

[0101] Storage medium 220 stores machine-readable instructions executable by processor 210. When the control device is running, processor 210 communicates with storage medium 220 via bus 230, and processor 210 executes the machine-readable instructions to perform the steps of the above method embodiment. The specific implementation and technical effects are similar and will not be described in detail here.

[0102] Optionally, this application also provides a storage medium 220, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the above-described method embodiments. The specific implementation and technical effects are similar, and will not be repeated here.

[0103] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0106] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for aligning robot actions with audio, characterized in that, The method includes: Obtain a composite sequence of instructions for the robot, the composite sequence of instructions including an action list and an audio list; The audio files in the audio list are converted to their original formats to create audio items. Bind the audio item to the corresponding action in the action list to obtain a playback group; Based on the actual duration of the audio in the playback group and the original duration of the corresponding action, the actions and audio in the playback group are time-aligned to obtain the time-aligned playback group. Based on the aligned playback group, the robot is controlled to play audio and perform corresponding actions.

2. The method according to claim 1, characterized in that, The step of aligning the actions and audio in the playback group according to the actual duration of the audio and the original duration of the corresponding actions to obtain a duration-aligned playback group includes: Based on the actual duration of the audio in the playback group and the original duration of the corresponding action, the original duration of the corresponding action is adjusted to obtain the adjusted duration of the corresponding action, wherein the adjusted duration of the corresponding action is the same as the actual duration of the audio. Based on the actual duration of the audio and the corresponding action with the adjusted duration, the duration-aligned playback group is constructed.

3. The method according to claim 2, characterized in that, The audio item includes a semantic anchor table, which records the action keyframe identifiers corresponding to target audio points in the audio. The step of adjusting the original duration of the corresponding action based on the actual duration of the audio in the playback group and the original duration of the corresponding action to obtain the adjusted duration of the corresponding action includes: If there is an action keyframe identifier in the semantic anchor table, then the audio segment of the audio and the corresponding action segment of the audio segment are determined according to the action keyframe identifier. The frame rate of the corresponding action segment is determined based on the actual duration of the audio segment and the number of frames of the corresponding action segment. Based on the frame rate of the corresponding action segment, the original duration of the corresponding action segment is adjusted to obtain the adjusted duration of the corresponding action.

4. The method according to claim 3, characterized in that, The step of adjusting the original duration of the corresponding action based on the actual duration of the audio in the playback group and the original duration of the corresponding action to obtain the adjusted duration of the corresponding action further includes: If there is no action keyframe identifier in the semantic anchor table, the scaling factor is determined based on the actual duration of the audio and the original duration of the corresponding action. Based on the scaling factor, the original duration of the corresponding action is adjusted to obtain the adjusted duration of the corresponding action.

5. The method according to claim 2, characterized in that, The method further includes: If the adjusted duration of the corresponding action is lower than the minimum duration constraint value of the corresponding action, then the maximum duration of the corresponding action is obtained. The pause duration of the audio is determined based on the maximum duration of the corresponding action; The playback duration of the audio is determined based on the pause duration and the actual duration. Based on the audio of the playback duration and the corresponding action of the maximum duration, the playback group with the duration alignment is constructed.

6. The method according to claim 1, characterized in that, The audio item includes: audio type, and the method further includes: Based on the audio type, determine the target player; Based on the aligned playback group, controlling the robot to play audio and perform corresponding actions includes: Based on the aligned playback group, the robot is controlled to play audio using the target player.

7. The method according to claim 1, characterized in that, The method further includes: If an interrupt command is received, the target playback group corresponding to the interrupt command, as well as the motion trajectory and audio cursor in the target playback group, are recorded. If a recovery command is received, the recovery playback group is determined based on the motion trajectory and audio cursor in the target playback group.

8. A robot motion and audio alignment device, characterized in that, The device includes: The acquisition module is used to acquire composite sequence instructions for the robot, the composite sequence instructions including an action list and an audio list; The construction module is used to convert the format of each audio file in the audio list and construct audio items; The binding module is used to bind the audio item to the corresponding action in the action list to obtain a playback group; The alignment module is used to align the actions and audio in the playback group according to the actual duration of the audio in the playback group and the original duration of the corresponding actions, so as to obtain the playback group after duration alignment. The control module is used to control the robot to play audio and perform corresponding actions based on the aligned playback group.

9. A control device, characterized in that, include: The device includes a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the control device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the robot motion and audio alignment method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the steps of the robot motion and audio alignment method as described in any one of claims 1 to 7.

Images