Robot control method, controller, control system, and robot

Abstract

The application discloses a robot control method, a controller, a control system and a robot, and relates to the technical fields of robots and multimedia control. The method comprises the following steps: acquiring multiple multi-axis actions of a robot playing audio, determining a time offset of each multi-axis action through a deviation between an execution time of an action item and a pronunciation time of a phoneme in a process of pre-arranging the multi-axis actions, and then applying an initial trigger time and the time offset in a playing process of the audio. The time offset supports independent configuration and on-site fine adjustment, improves alignment accuracy of multi-axis mouth shape actions of the robot and the pronunciation time of the phoneme, and can quickly complete timing calibration in the face of different models of robots or different running scenes, thereby improving scene adaptation capability of multi-axis synchronization of the audio. In addition, all action trigger tasks are arranged in batches at one time, and real-time polling in the audio playing process is not needed, so that long-term stability of audio-visual synchronization effect is ensured.

Description

Technical Field

[0001] This application relates to the field of robotics and multimedia control technology, and more specifically, to a robot control method, controller, control system, and robot. Background Technology

[0002] Multi-expression bionic heads and robots are widely used in exhibitions, guided tours, and human-computer interaction scenarios. These devices need to drive lip-sync / facial movements while playing audio, and ensure audio-visual synchronization. However, in technologies that achieve audio-visual synchronization based on a "phoneme-lip shape" mapping table, most directly use the audio playback timestamp to trigger lip-sync / facial movements, or compensate for delays by setting a fixed time advance. Significant audiovisual discrepancies still exist between the audio and the lip-sync / facial movements. Summary of the Invention

[0003] In view of this, embodiments of this application propose a robot control method, controller, control system, and robot, which can reduce the audiovisual discrepancy between audio and lip movements / facial expressions.

[0004] The following technical solution is adopted in this application.

[0005] In a first aspect, embodiments of this application provide a robot control method, the method comprising: acquiring multiple multi-axis actions of a robot playing audio, each multi-axis action including an action item and a corresponding phoneme, the action item including an initial trigger time; acquiring a time offset of each of the multiple multi-axis actions; the time offset of the multi-axis actions being used to indicate the deviation between the execution time of the action item and the sounding time of the phoneme during the pre-arrangement of the multi-axis actions; and applying the initial trigger time and time offset of each of the multi-axis actions to the audio playback process.

[0006] In some embodiments, obtaining the time offset of each of the plurality of multi-axis actions includes: during the pre-arrangement of the multi-axis actions, obtaining the execution time of the action item and the sounding time of the phoneme; and determining the time offset of the action item based on the deviation between the execution time and the sounding time.

[0007] In some embodiments, determining the time offset of the action item includes: if the execution time of the action item is after the pronunciation time of the phoneme, determining the time offset to be negative; if the execution time of the action item is before the pronunciation time of the phoneme, determining the time offset to be positive; if the execution time of the action item is the same as the pronunciation time of the phoneme, determining the time offset to be zero.

[0008] In some embodiments, applying the initial trigger time and time offset of each of the multi-axis actions to the audio playback process includes: adjusting the initial trigger time of each action item included in the plurality of multi-axis actions using the time offset to determine the target trigger time of each action item; determining the execution time of each action item according to the target trigger time; and executing each action item according to the execution time during the audio playback process.

[0009] In some embodiments, determining the target trigger time for each action item includes: obtaining a fixed compensation time for each action item; superimposing the fixed compensation time with the time offset of each action item to obtain the actual compensation time for each action item; and adjusting the initial trigger time of each action item using the actual compensation time to determine the target trigger time for each action item.

[0010] In some embodiments, determining the execution time of each action item includes: obtaining the playback start time of the audio; and determining the execution time of each action item based on the playback start time and the target trigger time of each action item.

[0011] In some embodiments, during the playback of the audio, the method of executing each action item according to the execution time further includes: periodically acquiring the currently playing phoneme, the actual sounding time of the phoneme, and the actual execution time of the action item corresponding to the phoneme; determining the deviation between the actual sounding time of the phoneme and the actual execution time of the action item; if the deviation is greater than a preset threshold, recording the trigger time and actual compensation time of the phoneme, the action item corresponding to the phoneme.

[0012] Secondly, embodiments of this application provide a robot control device, comprising: a first acquisition module, configured to acquire multiple multi-axis movements of a robot playing audio, each multi-axis movement including an action item and a corresponding phoneme, the action item including an initial trigger time; a second acquisition module, configured to acquire a time offset of each of the multiple multi-axis movements; the time offset of the multi-axis movement is used to indicate the deviation between the execution time of the action item and the sounding time of the phoneme during the pre-arrangement of the multi-axis movements; and an audio-visual control module, configured to apply the initial trigger time and time offset of each of the multi-axis movements to the audio playback process.

[0013] Thirdly, a controller is provided, comprising: a processor and a memory; the memory storing instructions or programs, and the processor being configured to: implement the robot control method of the first aspect when executing the instructions or programs.

[0014] Fourthly, a control system is also provided, comprising: an execution device and a controller of the third aspect; the execution device is used to receive instructions issued by the controller and to perform audio playback or multi-axis motion output, and the controller is used to execute the robot control method of the first aspect.

[0015] Fifthly, a robot is also provided, comprising: a body and a control system of the fourth aspect, the control system being disposed in or inside the body.

[0016] In a sixth aspect, a computer-readable storage medium is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable a computer to implement the robot control method of the first aspect.

[0017] In a seventh aspect, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the robot control method of the first aspect.

[0018] The technical solution provided in this application has at least the following beneficial effects: A composite timing compensation mechanism combining time offset and fixed compensation time is adopted. The adjustable offset serves as an independent time offset for each action item, which can be flexibly fine-tuned according to the actual audiovisual perception on site. Through dual compensation and collaborative optimization, the alignment accuracy of the robot's multi-axis mouth movements and the sounding time of phonemes is significantly improved. At the same time, the time offset supports independent configuration and on-site debugging. Timing calibration can be quickly completed for different robot models or different system operating load scenarios, greatly improving the scene adaptability of audio multi-axis synchronization and the convenience of subsequent operation and maintenance debugging.

[0019] In addition, by calculating the delay time of all actions before audio playback starts, all action triggering tasks can be arranged in batches at once, eliminating the need for real-time polling during audio playback. This significantly reduces the scheduling computation and timing jitter during the runtime phase, ensuring long-term stability of audio-visual synchronization.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 This is a schematic diagram of a control system provided in an embodiment of this application; Figure 2 This is a flowchart of the first robot control method provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the compensation relationship between an action sequence and an audio timeline, provided in an embodiment of this application. Figure 4 This is a flowchart illustrating the second robot control method provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the third robot control method provided in the embodiments of this application; Figure 6 This is a flowchart illustrating a method for determining a target trigger time provided in an embodiment of this application; Figure 7 This is a flowchart illustrating a method for determining execution time provided in an embodiment of this application; Figure 8 This is a schematic diagram of an audio playback process data recording method provided in an embodiment of this application; Figure 9 This is a flowchart illustrating the fourth robot control method provided in the embodiments of this application; Figure 10 This is a logical block diagram illustrating the relationship between pre-scheduled tasks and the scheduler, provided in an embodiment of this application. Figure 11 This is a schematic diagram of the structure of a robot control device provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a controller provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of a robot provided in an embodiment of this application.

[0023] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept of this application in any way, but rather to illustrate the concept of this application to those skilled in the art through specific embodiments. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0025] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0026] To achieve the visual effect of "speaking / singing" by aligning lip movements with audio content during audio playback in devices such as multi-expression bionic heads and humanoid robots, the following solutions are currently commonly used: First, pre-recording lip movements to correspond one-to-one with audio. This involves recording each frame of lip movement and audio in advance according to a fixed timeline, and then strictly adhering to that timeline during playback. This solution offers good consistency between lip movements and sound, but the content is fixed and cannot support arbitrary text or new tracks, resulting in poor scalability. Second, real-time driving based on phonemes / phonetic symbols. This involves obtaining a phoneme timeline through speech recognition or pre-annotation, and then issuing servo commands based on timestamps during playback according to a "phoneme-lip movement" mapping table. This solution supports dynamic content, but the system itself has latency (the latency mainly comes from audio decoding, rendering, driver issuance, and servo response). If actions are triggered directly by timestamps, lip movements often lag behind the sound, resulting in visual asynchrony. Thirdly, there is the simple fixed advance, which sets a fixed advance for the trigger time (such as uniformly advancing by several hundred milliseconds) to compensate for the delay. However, the delay varies greatly under different devices, different decoding paths and loads. It is difficult to achieve good synchronization in all scenarios with a single fixed advance, and it is also impossible to calibrate and adjust it according to the actual running effect after deployment.

[0027] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0028] Figure 1 This is a schematic diagram of a control system provided in an embodiment of this application. Figure 1As shown, the control system 100 may include a controller 110 and an execution device 120.

[0029] In one alternative implementation, controller 110 refers to a software unit or module. Controller 110 includes a storage unit 111, a configuration unit 112, a processing unit 113, and a scheduler 114. Storage unit 111 stores action sequences and can also store the phonetic transcription sets (SRTs) and mapping tables used to generate the action sequences. Configuration unit 112 stores or inputs fixed compensation values ​​and adjustable offsets. Processing unit 113 calculates the actual delay based on the trigger time and compensation amount and pre-schedules trigger tasks to the scheduler before playback. Scheduler 114 issues multi-axis control commands after a specified delay.

[0030] In another alternative implementation, the controller 110 and the execution device 120 refer to hardware devices; wherein, the execution device 120 includes, but is not limited to, audio playback devices (such as a multimedia player MediaPlayer), multi-axis execution devices (such as a mouth-shaped servo), etc.

[0031] Optionally, the controller 110 and the actuator 120 can communicate via a wired or wireless connection. The wired connection may include, but is not limited to, a bus, fiber optic cable, or network cable. Wireless connections may include, for example, transmission control protocol / internet protocol (TCP / IP), wireless local area network protocol (WLAN), and remote direct memory access (RDMA) over converged ethernet (RoCE) network protocol.

[0032] The following is combined with Figure 1 The control system 100 shown illustrates the robot control method provided in this application embodiment: The control system 100 can be applied to a multi-expression bionic head or similar device. The controller 110 first acquires multiple multi-axis movements corresponding to the robot playing audio. During the pre-arrangement of multi-axis movements, it calculates the time offset of each movement in the multiple multi-axis movements and then determines the execution time of each movement based on the offset. On the other hand, the controller 110 generates a playback command for the corresponding audio when interacting with the user, and generates servo commands in real time according to the predetermined execution time during audio playback. Finally, the controller 110 sends the playback command and / or servo commands to the execution device 120, which completes the synchronous execution of the audio and the corresponding movement based on the received command.

[0033] Figure 1 The control system in it can be used to achieve the following Figure 2 Please refer to the described robot control method. Figure 2 , Figure 2 This is a flowchart of a first robot control method provided in an embodiment of this application. In a specific embodiment, this robot control method can be applied to, for example... Figure 11 The robot control device 500 shown and the robot equipped with the robot control device 500 are shown. Figure 13 The following will address... Figure 2 The process shown is described in detail. Of course, it is understood that this method can be executed by an electronic device with computing power, such as a controller, vehicle-side server, cloud server, or other processor. The robot control method may specifically include the following steps 210 to 230.

[0034] Step 210: Obtain multiple multi-axis movements of the robot playing audio. Each multi-axis movement includes an action item and a corresponding phoneme. The action item includes an initial trigger time.

[0035] In the embodiments of this application, multiple multi-axis actions together constitute a multi-axis action sequence or a set of multi-axis actions that match the audio time axis.

[0036] For example, the audio file of the voice or music to be played by the robot can be read first, and then the action timeline aligned with the audio timing can be retrieved. The action timeline can be obtained by parsing the phonetic symbol SRT file with timestamps. Each timestamp is bound to the corresponding phonetic symbol. Then, according to the pre-stored phonetic symbol-lip shape mapping table, the phonetic symbols are converted into multi-degree-of-freedom servo drive data (such as 10-axis lip shape data), generating action items with initial trigger time and corresponding phonemes. Finally, the complete multi-axis action sequence is obtained by integration.

[0037] In a more specific example, the construction of the action sequence may include: parsing a phonetic SRT file (SubRipText) to obtain multiple entries, each containing a start time, an end time, and phonetic text; then, using the start time of each entry as the initial trigger time Si of the corresponding phonetic symbol, in milliseconds, with the time base being zero relative to the start of the audio; subsequently, converting each phonetic symbol into multi-axis lip-sync data, such as a short integer array, according to a pre-stored phonetic symbol-lip-sync mapping table; finally, generating a "trigger time-axis data" action sequence, such as a TimedAxisData list, and sorting the list by trigger time.

[0038] The format of a phonetic symbol SRT file can be divided into: sequence number, time line (regular expression matching HH:MM:SS, mmm-->HH:MM:SS, mmm), and phonetic symbol content line; when parsing a phonetic symbol SRT file, it is necessary to process BOM (Byte Order Mark) and whitespace characters, and convert the time line into milliseconds relative to the starting point.

[0039] It should be noted that in this embodiment, phonetic symbols are standardized written symbols; phonetic text is a string carrier formed by splicing several phonetic symbols and is stored in SRT files and action sequence data; a phoneme is the smallest actual speech unit in audio and is the timing alignment object for lip-syncing.

[0040] By using a preset phonetic symbol-lip shape mapping table, the corresponding phonemes are matched and obtained by using the phonetic symbols in the phonetic symbol text as an index; the phonemes corresponding to the action items are bound to the corresponding phonemes by symbol tags, thereby associating the phoneme's phonation time with the timing of multi-axis lip shape actions.

[0041] Step 220: Obtain the time offset of each of the multiple multi-axis actions; the time offset of the multi-axis action is used to indicate the deviation between the execution time of the action item and the sounding time of the phoneme during the pre-arrangement of the multi-axis actions.

[0042] In this embodiment, the time offset corresponding to a single multi-axis action is an adjustable offset (timeOffset), measured in milliseconds, with a default initial value of 0. This adjustable offset is configured independently for the lip-sync action corresponding to a single phoneme, allowing for fine-tuning of the single multi-axis action on-site based on the actual playback experience. When the parameter is positive, the trigger time of the corresponding action item is delayed; when the parameter is negative, the trigger time of the corresponding action item is advanced.

[0043] For example, Figure 3 This is a schematic diagram of the compensation relationship between an action sequence and an audio timeline. The compensation amount corresponding to each multi-axis action is divided into two independent categories: one is the time offset (adjustable offset), which is pre-arranged during the multi-axis action phase and can be adjusted one by one according to the visual effect of the actual audio and lip movements playing synchronously on site; the other is a globally unified fixed compensation value, for example, a value of 268ms. This constant is used to compensate for the inherent average delay of the system caused by audio decoding, output buffering, servo command issuance, and mechanical response. After the equipment has been calibrated by the initial environmental test, it is solidified into a system constant, such as TIME_COMPENSATION_MS (Time Compensation in Milliseconds).

[0044] In a more specific embodiment, the control system is equipped with a dedicated parameter configuration and debugging interface. Maintenance personnel can modify the adjustable offset (i.e., time offset) of each action item individually without changing the global fixed compensation value. The complete actual compensation amount of each action item can be recalculated by superimposing the values. The calculation formula is: compensation amount = fixed compensation value + time offset. Then, after updating the action pre-arrangement sequence according to the actual compensation amount, the accurate calibration of a single action can be completed on-site without rewriting the code and keeping the global basic delay unchanged.

[0045] Step 230: Apply the initial trigger time and time offset of each of the multi-axis actions to the audio playback process.

[0046] In this embodiment of the application, the initial trigger time and time offset corresponding to each multi-axis action are applied to the audio playback timing planning. That is, based on the initial trigger time obtained in step 210, the time offset (adjustable offset) determined in step 220, and the fixed compensation value, the timing is uniformly converted to determine the execution time of the multi-axis action and complete the synchronous pre-arrangement of audio and multi-axis lip-syncing.

[0047] For example, the audio playback start point is defined as the reference time T0, where T0 is the time when the audio playback start command is issued, such as the time when the player MediaPlayer.start() is called, or the time when the system records musicStartTime.

[0048] Furthermore, for the i-th action item in the action sequence, Si is the initial trigger time of the action item with the audio start point as zero. After superimposing the global fixed compensation value and the adjustable offset (i.e., the time offset) on Si, the actual delay time Di (i.e., the target trigger time) of the action item relative to the audio playback start point as zero can be obtained. The calculation formula is: Di = Si + fixed compensation value + adjustable offset, where the parameter i is a constant greater than zero.

[0049] If the calculated Di is negative, it means that the theoretical trigger time of the action is earlier than the audio playback start point T0. The system will judge it as an expired action, discard the action directly, and not generate a corresponding scheduling task.

[0050] Furthermore, by taking the audio playback start point as the reference time T0 and combining the actual delay time Di of each action item relative to the zero point of the audio playback start point, the actual trigger time of all action items can be uniformly converted to T0+Di; all actions will be scheduled by the servo to execute mouth movements according to this absolute time, and the overall mouth movement timing can be flexibly corrected to advance or delay relative to speech hearing through adjustable offset.

[0051] In a more specific embodiment, all action triggering tasks are pre-arranged before the audio is actually played, eliminating the need to query the playback progress in real time during audio playback and reducing timing jitter. Before audio playback starts or the player's start interface is called, the control system can perform batch pre-arrangement processing through a single-threaded scheduler (such as ScheduledExecutorService). The specific processing flow includes the following.

[0052] Traverse all action items in the multi-axis action sequence and calculate the actual delay time Di for the corresponding action item according to the above formula; for valid action items that satisfy Di≥0, submit a delayed task to the scheduler with a delay duration of Di in milliseconds.

[0053] Furthermore, when the scheduler reaches the corresponding delay duration, it automatically triggers the task control logic, which sends the multi-axis servo data carried by the current action item to the execution device via communication interfaces such as serial port and industrial bus, driving the multi-degree-of-freedom servo to complete the corresponding mouth shape action.

[0054] It is important to note that all delayed tasks use the audio playback reference time T0 as a unified time base and are registered and submitted all at once before playback begins. Compared to determining the triggering timing by polling getCurrentPosition() in real time, this embodiment of the application no longer relies on reading the current audio playback position in real time during playback, avoiding timing fluctuations and scheduling stuttering issues caused by high-frequency reading interfaces, and further improving the synchronization stability of audio and lip movements.

[0055] In this embodiment, a composite timing compensation mechanism with a fixed compensation value and an adjustable offset is adopted. The adjustable offset serves as an independent time offset for each action item, which can be flexibly fine-tuned according to the actual audiovisual perception on site. The dual compensation works together to optimize and significantly improve the alignment accuracy of the robot's multi-axis mouth movements and the sounding time of phonemes. At the same time, the fixed compensation value serves as the inherent delay of the device, and the adjustable offset supports independent configuration and on-site debugging. Timing calibration can be quickly completed for different robot models or different system operating load scenarios, which greatly improves the scene adaptability of audio multi-axis synchronization and the convenience of subsequent operation and maintenance debugging.

[0056] In addition, the delay time calculation for all actions can be completed before audio playback starts, and all action triggering tasks can be arranged in batches at once. There is no need to poll in real time during audio playback, which greatly reduces the scheduling calculation and timing jitter during the operation phase and ensures the long-term stability of audio-visual synchronization.

[0057] Based on the above, this application provides an optional implementation method for obtaining the time offset of each of the multiple multi-axis movements, such as... Figure 4The flowchart of the second robot control method shown may specifically include the following steps 310 to 340.

[0058] Step 310: Obtain multiple multi-axis movements of the robot playing audio. Each multi-axis movement includes an action item and a corresponding phoneme. The action item includes an initial trigger time.

[0059] Step 320: During the pre-arrangement of the multi-axis actions, obtain the execution time of the action item and the sounding time of the phoneme.

[0060] In this embodiment, the phonetic symbol SRT file is divided into entries by double line breaks. The entry number, start and end time axis, and phonetic symbol text are parsed one by one, and the time is uniformly converted to milliseconds. The phonetic symbol-lip shape mapping table is retrieved, such as a Map with phonetic symbols as keys and multi-axis lip shape data as values. The phonetic symbols include, but are not limited to, IPA, Pinyin, and common variants. The multi-axis lip shape data can be axis data composed of 10 shorts. The corresponding multi-axis lip shape data is matched with the phonetic symbols as indexes. If no matching phonetic symbol is found, the default lip shape is output or the lip shape of the previous frame is continued. Finally, the action items carrying the initial trigger time and corresponding phonemes are integrated to generate the TimedAxisData list, which is the multi-axis action sequence. After converting the phonetic symbol entry list into the TimedAxisData list, the multi-axis actions are pre-arranged and calibrated for the actual scene.

[0061] In one example, in an actual on-site pre-arrangement scenario, based on on-site audiovisual observation of audio and lip movements, the execution time of actions with asynchronous audio and lip movements and the pronunciation time of the corresponding phonemes / phonetic symbols are extracted.

[0062] Optionally, in addition to phonetic symbol SRT files, multi-axis motion sequences can also be generated by TTS outputting phoneme timelines or by custom handwritten timing scripts. As long as a structured sequence of "trigger time - multi-axis data" can be output, this process can be used to complete the timing pre-arrangement.

[0063] Step 330: Determine the time offset of the action item based on the deviation between the execution time and the sound emission time.

[0064] In one example, in a real-world calibration scenario, if lip movement is observed to be lagging, maintenance personnel can reduce the adjustable offset or set it to a negative value; if lip movement is observed to be ahead, the adjustable offset can be increased to a positive value.

[0065] Furthermore, after modifying the parameters, the audio and multi-axis motion sequences can be reloaded, and the delay calculation and task pre-scheduling process can be re-executed. The timing calibration results will take effect in the next round of audio playback, flexibly adapting to the synchronization needs of different devices and on-site working conditions.

[0066] In one implementation, step 330, regarding how to determine the time offset of the action item, may specifically include the following:

[0067] If the execution time of the action item is after the pronunciation time of the phoneme, the time offset is determined to be negative; if the execution time of the action item is before the pronunciation time of the phoneme, the time offset is determined to be positive; if the execution time of the action item is the same as the pronunciation time of the phoneme, the time offset is determined to be zero.

[0068] In this embodiment, the adjustable offset of the action item is determined by comparing the timing difference between the execution time of the action item and the pronunciation time of the phoneme, based on the direction of the deviation. Specifically, when the execution time of the action item is later than the pronunciation time of the phoneme, it indicates that the lip-shape movement lags behind the speech, and the timing offset is set to a negative value to advance the action triggering time; when the execution time of the action item is earlier than the pronunciation time of the phoneme, it indicates that the lip-shape movement precedes the speech, and the timing offset is set to a positive value to delay the action triggering time; when the timing of the two is perfectly aligned, the timing offset is assigned a value of 0, and no timing fine-tuning is required.

[0069] Step 340: Apply the initial trigger time and time offset of each of the multi-axis actions to the audio playback process.

[0070] For a detailed description of steps 310 and 340, please refer to steps 210 and 230, which will not be repeated here.

[0071] In this embodiment, the adjustable offset serves as an independent time offset for each action item, which can be flexibly fine-tuned according to the actual audiovisual experience on site, significantly improving the alignment accuracy between the robot's multi-axis mouth movements and the sounding time of phonemes; furthermore, the delay time calculation for all action items is completed before the audio playback starts, and all action triggering tasks are arranged in batches at once, eliminating the need for real-time polling during audio playback, greatly reducing the scheduling calculation load and timing jitter during the operation phase, and ensuring the stability of the audio-visual synchronization effect.

[0072] Based on the above, this application provides an optional implementation method for applying the initial trigger time and time offset of each multi-axis motion to the audio playback process, such as... Figure 5 The flowchart of the third robot control method shown may specifically include the following steps 410 to 450.

[0073] Step 410: Obtain multiple multi-axis actions of the robot playing audio. Each multi-axis action includes an action item and a corresponding phoneme. The action item includes an initial trigger time.

[0074] Step 420: Obtain the time offset of each of the multiple multi-axis actions; the time offset of the multi-axis action is used to indicate the deviation between the execution time of the action item and the sounding time of the phoneme during the pre-arrangement of the multi-axis actions.

[0075] Step 430: Adjust the initial trigger time of each action item included in the multiple multi-axis actions using the time offset to determine the target trigger time of each action item.

[0076] For example, since the initial trigger time Si is obtained only by parsing the phonetic timing file (i.e., the phonetic SRT file) without considering timing deviation, the adjustable offset (i.e., the time offset) corresponding to a single action item can be superimposed on the initial trigger time. After a single fine-tuning, the target trigger time Di that adapts to the on-site audiovisual effects is obtained.

[0077] In step 430, regarding how to determine the target trigger time for each action item, this application embodiment provides an optional implementation method, such as... Figure 6 The flowchart shown is a method for determining the target trigger time, which may specifically include the following steps 431 to 433.

[0078] Step 431: Obtain the fixed compensation time for each of the aforementioned action items.

[0079] For example, a globally uniform fixed compensation time (i.e., a fixed compensation value) is read to compensate for inherent system delays such as device audio decoding, command transmission, and servo mechanical response.

[0080] Step 432: The fixed compensation time is superimposed with the time offset of each action item to obtain the actual compensation time of each action item.

[0081] For example, the global fixed compensation time is superimposed with the time offset of a single action item to obtain the actual compensation time that takes into account both the inherent delay of the system and the personalized fine-tuning on site.

[0082] Step 433: Adjust the initial trigger time of each action item using the actual compensation time to determine the target trigger time of each action item.

[0083] In this embodiment, the target trigger time is the actual delay time Di of the action item relative to the audio playback start point as zero; the original initial trigger time of the action item is corrected by using the actual compensation time to generate the corrected target trigger time.

[0084] Optionally, only the adjustable offset can be used, without using the fixed compensation value. According to the calculation formula of the actual delay time Di (i.e., the target trigger time): Di = Si + fixed compensation value + adjustable offset, it can be seen that as long as the fixed compensation value is set to 0, all timing compensation can be achieved only through the adjustable offset. This solution is suitable for scenarios where all compensation parameters are configured by the operation and maintenance personnel to calibrate the synchronization of action and audio.

[0085] Step 440: Determine the execution time of each action item based on the target trigger time.

[0086] For example, since the target trigger time is the actual delay time of the action item relative to the zero point of audio playback (i.e. the starting point of audio playback), after obtaining the actual playback start time T0 of the audio, the execution time of the action item that the system can directly schedule and use can be calculated by using this start time as the absolute time reference and combining it with the target trigger time.

[0087] In step 440, regarding how to determine the execution time of each action item, this application embodiment provides an optional implementation method, such as... Figure 7 The flowchart shown is a method for determining execution time, which may specifically include the following steps 441 to 442.

[0088] Step 441: Obtain the playback start time of the audio.

[0089] Step 442: Determine the execution time of each action item based on the playback start time and the target trigger time of each action item.

[0090] The specific steps of steps 441 to 442 can be found in step 440, and will not be repeated here.

[0091] Step 450: During the playback of the audio, each of the action items is executed according to the execution time.

[0092] In one implementation, during audio playback, the system directly issues multi-axis drive commands sequentially according to the calculated execution time, driving the bionic head mouth-shaped servo to synchronously complete the corresponding mouth movements, thus achieving visual synchronization between speech and mouth movements.

[0093] Optionally, a synchronization scheme that dynamically calculates the delay during playback can be adopted. That is, instead of pre-scheduling all tasks, the current playback position is read periodically (e.g., every 50ms) during playback and compared with the action items that have not been triggered in the action sequence. If the current playback position is greater than or equal to the sum of the trigger time and compensation amount of the action item, then a multi-axis drive command is triggered and issued to achieve the stability of speech and lip-sync visual synchronization with simpler logic.

[0094] Alternatively, a multi-level compensation mechanism can be adopted for synchronization. This involves adding dynamic compensation items related to the current playback progress or device status on the basis of fixed and adjustable compensation amounts. For example, fine-tuning can be performed based on the buffer status to form a multi-level compensation mechanism, thereby further improving the synchronization adaptability in complex environments.

[0095] In step 450, during the playback of the audio, this embodiment of the application also provides an optional implementation method, such as... Figure 8 The diagram illustrates a method for recording data during audio playback, which may specifically include steps 451 to 453.

[0096] Step 451: Periodically acquire the currently playing phoneme, the actual sounding time of the phoneme, and the actual execution time of the action item corresponding to the phoneme.

[0097] For example, the control system periodically collects real-time operating data, including the phoneme currently emitting sound, the actual time of the phoneme's emission, and the actual execution time of the action item bound to the phoneme, and continuously tracks the timing matching status between the phoneme's emission and the mouth shape action.

[0098] Optionally, the control system can be configured with a time synchronization debugging function to periodically output the current audio playback position, the original initial trigger time Si of the action item, various compensation parameters, and the actual time of the servo command issuance during audio playback. Maintenance personnel can quickly debug and adapt the adjustable offset to the field conditions based on the output data.

[0099] Step 452: Determine the deviation between the actual sounding time of the phoneme and the actual execution time of the action item.

[0100] For example, the difference between the two sets of collected times is calculated to determine the timing deviation between the actual sounding time of the phoneme and the actual execution time of the corresponding action item.

[0101] Step 453: If the deviation is greater than a preset threshold, record the trigger time and actual compensation time of the phoneme, the corresponding action item of the phoneme.

[0102] In this embodiment of the application, the timing deviation is compared with a preset threshold. When the deviation value exceeds the threshold range, the control system retains and records complete parameters such as the current phoneme, the initial timing reference of the corresponding action item, and the actual compensation time, and retains the calibration basis.

[0103] In one example, when a scheduled task is triggered, the control system can simultaneously read the real-time audio playback position and compare it with the action execution time. If the deviation exceeds a set threshold (100ms for example), only an alarm log is generated and stored, without modifying the action instructions already issued to the actuator. By recording abnormal data without interfering with real-time action scheduling, complete calibration reference data can be preserved, while avoiding playback stuttering and action errors caused by dynamically modifying instructions, thus balancing debugging and traceability capabilities with real-time operational stability.

[0104] The specific steps of steps 410 to 420 can be found in steps 210 to 220, and will not be repeated here.

[0105] In this embodiment, by combining a fixed compensation value with an adjustable offset, the triggering tasks of each action item are pre-arranged to the scheduler according to the compensated time before the audio starts playing, so that the lip-sync triggering time is aligned with the actual audio output heard by the listener. At the same time, it supports on-site fine-tuning through the adjustable offset, thereby improving synchronization accuracy and stability.

[0106] Based on the foregoing, this application also provides an optional implementation method, such as... Figure 9 The flowchart of the fourth robot control method shown may include the following:

[0107] To address the numerous problems in the synchronous control of robot audio and lip movements (multi-axis motion), such as the significant lag between lip movement triggering and auditory triggering, insufficient synchronization accuracy, inability to adapt to different devices and operating states by relying solely on fixed lead time, inability to perform calibration after deployment, insufficient stability of the synchronization triggering strategy, and susceptibility to single scheduling deviations.

[0108] Combination Figure 10 The diagram shows the logical relationship between the pre-arranged tasks and the scheduler. This embodiment of the application employs a time-compensated audio and multi-axis motion synchronization control method, specifically as follows: A multi-axis motion sequence corresponding to the audio time axis is obtained, where each motion item in the sequence is associated with a trigger time; a time compensation amount is determined, including a fixed compensation value and an adjustable offset; before audio playback begins, the actual delay time of each motion item relative to the playback start point is calculated based on the trigger time and the time compensation amount; before or upon reaching the playback start point, the trigger tasks of each motion item are pre-arranged to the scheduler according to the actual delay time, so that each motion is triggered and multi-axis control commands are issued during playback according to the compensated time.

[0109] Optionally, the adjustable offset can be adjusted according to the actual synchronization effect to complete the on-site calibration.

[0110] In this embodiment, a composite timing compensation mechanism with a fixed compensation value and an adjustable offset is adopted. While covering the inherent delay of typical devices, it supports fine-tuning parameters according to the on-site synchronization effect, thereby improving the alignment accuracy between lip movements and audio. Furthermore, before audio playback, all triggering tasks are pre-arranged at once based on the complete timeline and compensation amount, reducing scheduling jitter during operation to ensure the synchronization stability between lip movements and audio. With the introduction of adjustable offset, calibration can be performed for different devices and different loads, improving the applicability and maintainability of the method.

[0111] To achieve the functions of the above embodiments, the robot control method includes hardware structures and / or software modules corresponding to each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed through hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0112] exist Figures 2 to 10 Based on the robot control method shown, this application also provides a robot control device for further explanation, such as... Figure 11 The diagram shows the structure of a robot control device 500, which includes: a first acquisition module 510, a second acquisition module 520, and an audio-visual control module 530.

[0113] The first acquisition module 510 is used to acquire multiple multi-axis movements of the robot playing audio. Each multi-axis movement includes an action item and a corresponding phoneme. The action item includes an initial trigger time. The first acquisition module 510 may include, for example: Figure 1 The controller 110 in the control system 100 shown.

[0114] The second acquisition module 520 is used to acquire the time offset of each of the plurality of multi-axis actions; the time offset of the multi-axis action is used to indicate the deviation between the execution time of the action item and the sounding time of the phoneme during the pre-arrangement of the multi-axis actions; wherein, the second acquisition module 520 may include, for example, Figure 1 The controller 110 in the control system 100 shown.

[0115] The audio-visual control module 530 is used to apply the initial trigger time and time offset of each of the multi-axis movements to the audio playback process; wherein, the audio-visual control module 530 may include, for example, Figure 1 The control system 100 shown includes a controller 110 and an actuator 120.

[0116] In some embodiments, the second acquisition module 520 includes: a pre-arrangement module, configured to acquire the execution time of the action item and the sounding time of the phoneme during the pre-arrangement of the multi-axis actions; and a determination module, configured to determine the time offset of the action item based on the deviation between the execution time and the sounding time.

[0117] In some embodiments, the second acquisition module 520 further includes: a first determining submodule, configured to determine the time offset as negative if the execution time of the action item is later than the sounding time of the phoneme; a second determining submodule, configured to determine the time offset as positive if the execution time of the action item is earlier than the sounding time of the phoneme; and a third determining submodule, configured to determine the time offset as zero if the execution time of the action item is the same as the sounding time of the phoneme.

[0118] In other embodiments, the audio-visual control module 530 includes: a trigger time adjustment module, used to adjust the initial trigger time of each action item included in the plurality of multi-axis actions using the time offset, and determine the target trigger time of each action item; an execution time determination module, used to determine the execution time of each action item according to the target trigger time; and an action item execution module, used to execute each action item according to the execution time during the playback of the audio.

[0119] In other embodiments, the audio-visual control module 530 further includes: a compensation time acquisition module for acquiring a fixed compensation time for each action item; a compensation time processing module for superimposing the fixed compensation time with the time offset of each action item to obtain the actual compensation time of each action item; and a compensation time adjustment module for adjusting the initial trigger time of each action item using the actual compensation time to determine the target trigger time of each action item.

[0120] In other embodiments, the audio-visual control module 530 further includes: a start time acquisition module for acquiring the playback start time of the audio; and an execution time processing module for determining the execution time of each action item based on the playback start time and the target trigger time of each action item.

[0121] In some other embodiments, the audio-visual control module 530 further includes: a periodic acquisition module, used to periodically acquire the currently playing phoneme, the actual sounding time of the phoneme, and the actual execution time of the action item corresponding to the phoneme; a deviation determination module, used to determine the deviation between the actual sounding time of the phoneme and the actual execution time of the action item; and a recording module, used to record the triggering time and actual compensation time of the phoneme, the action item corresponding to the phoneme, if the deviation is greater than a preset threshold.

[0122] According to one aspect of the embodiments of this application, Figure 12 This is a schematic diagram of the structure of a controller provided in an embodiment of this application. Figure 12 As shown, the controller 110 includes a processor 610 and one or more memories 620. The one or more memories 620 are used to store program instructions executed by the processor 610. When the processor 610 executes the program instructions, it implements the robot control method described above.

[0123] Furthermore, the processor 610 may include one or more processing cores. The processor 610 runs or executes instructions, programs, code sets, or instruction sets stored in the memory 620, and retrieves data stored in the memory 620. Optionally, the processor 610 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 610 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor and may be implemented using a separate communication chip.

[0124] According to one aspect of the embodiments of this application, Figure 13 This is a schematic diagram of the structure of a robot provided in an embodiment of this application. Figure 13 As shown, the robot includes a body 710 and a control system 100, the control system being disposed in the body or inside the body.

[0125] According to one aspect of the embodiments of this application, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable storage medium carries computer-readable instructions, which, when executed by a processor, implement the methods in any of the above embodiments.

[0126] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0127] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0130] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A robot control method, characterized in that, include: Acquire multiple multi-axis movements of the robot playing audio, each multi-axis movement including an action item and a phoneme corresponding to the action item, the action item including an initial trigger time; Obtain the time offset of each of the plurality of multi-axis actions; the time offset of the multi-axis action is used to indicate the deviation between the execution time of the action item and the sounding time of the phoneme during the pre-arrangement of the multi-axis actions; The initial trigger time and time offset of each of the multi-axis actions are applied during the playback of the audio.

2. The method according to claim 1, characterized in that, The step of obtaining the time offset of each of the multiple multi-axis actions includes: During the pre-arrangement of the multi-axis actions, the execution time of the action item and the sounding time of the phoneme are obtained; The time offset of the action item is determined based on the deviation between the execution time and the sound emission time.

3. The method according to claim 2, characterized in that, Determining the time offset of the action item includes: If the execution time of the action item is after the sounding time of the phoneme, the time offset is determined to be negative; If the execution time of the action item precedes the sounding time of the phoneme, the time offset is determined to be a positive value; If the execution time of the action item is the same as the sounding time of the phoneme, the time offset is determined to be zero.

4. The method according to claim 1, characterized in that, The step of applying the initial trigger time and time offset of each of the multi-axis actions to the audio playback process includes: The initial trigger time of each action item included in the multiple multi-axis actions is adjusted using the time offset to determine the target trigger time of each action item; Based on the target trigger time, determine the execution time of each action item; During the playback of the audio, each of the aforementioned actions is executed according to the execution time.

5. The method according to claim 4, characterized in that, Determining the target trigger time for each action includes: Obtain the fixed compensation time for each of the aforementioned action items; The fixed compensation time is added to the time offset of each action item to obtain the actual compensation time of each action item. The initial trigger time of each action item is adjusted using the actual compensation time to determine the target trigger time of each action item.

6. The method according to claim 4, characterized in that, Determining the execution time of each of the aforementioned actions includes: Obtain the start time of the audio playback; The execution time of each action item is determined based on the playback start time and the target trigger time of each action item.

7. The method according to claim 4, characterized in that, During the playback of the audio, the method of executing each action item according to the execution time further includes: The system periodically acquires the currently playing phoneme, the actual sounding time of the phoneme, and the actual execution time of the action item corresponding to the phoneme. Determine the deviation between the actual sounding time of the phoneme and the actual execution time of the action item; If the deviation is greater than a preset threshold, record the trigger time and actual compensation time of the phoneme, the corresponding action item of the phoneme, and the phoneme.

8. A controller, characterized in that, include: A processor and a memory; the memory storing instructions or programs, the processor being configured to, when executing the instructions or programs, implement the method of any one of claims 1-7.

9. A control system, characterized in that, include: The execution device and the controller according to claim 7; the execution device is used to receive instructions issued by the controller and perform audio playback or multi-axis motion output, and the controller is used to perform the method according to any one of claims 1-7.

10. A robot, characterized in that, include: The fuselage and the control system of claim 9, wherein the control system is disposed in the fuselage or inside the fuselage.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor or electronic device to execute the method as described in any one of claims 1 to 7.

Images