A robot voice-expression linkage method, system, device and storage medium

Abstract

This invention discloses a robot voice-expression linkage method, system, device, and storage medium. The robot voice-expression linkage method of this invention includes: S1, within a fixed-block-length audio callback period, using a speech synthesis block just written to the speaker output buffer as a far-end reference signal, aligning it at the sample level with the near-end audio signal collected by the microphone, and obtaining an echo-cancelled audio signal through acoustic echo cancellation; S2, using a neural network speech activity detection model to detect the echo-cancelled audio signal frame by frame, determining the end of user speech and capturing a segment of user speech when the duration of trailing silence reaches an endpoint threshold; S3, performing speech recognition on the user speech to obtain transcribed text. This robot voice-expression linkage method, system, device, and storage medium of this invention address the shortcomings of existing technologies in voice interaction and expression linkage, enabling autonomous chat where the robot head can hear, think, and respond.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of robotics technology, and in particular to a method, system, device and storage medium for robot voice-expression linkage. Background Technology

[0002] Humanoid head robots are an important vehicle for human-computer interaction research. In the prior art, patent document with publication number CN101618280B discloses a humanoid head robot device and behavior control method with human-computer interaction function. Its facial expressions are driven by a pre-set mechanism action library, and the interaction content depends on a pre-programmed behavior script. The drawback is that it cannot generate answers to open-ended voice questions from users, and the facial expressions are also unrelated to the dialogue content.

[0003] In recent years, the emergence of large-scale dialogue models has brought new opportunities for the development of humanoid head robots. While directly integrating these large models into the head of a physical robot can mitigate some of the aforementioned shortcomings, the following technical limitations still exist: First, there is the issue of self-auditory interference. When a robot speaks, the sound emitted by its speaker is picked up by its own microphone, causing false triggering of speech activity detection. This means that the robot's own words are treated as user input, creating a "self-questioning and self-answering" error loop (also known as "self-auditory interference"). The conventional approach to this problem is to mute the microphone during speaking, but this prevents the user from interrupting the robot with voice, turning interactive communication into a rigid one-way dialogue. Furthermore, patent document CN112735398B discloses a method and system for switching human-computer dialogue modes, which can automatically switch between full-duplex and half-duplex modes based on the dialogue domain to which the user's statement belongs. In half-duplex mode, recording stops during voice broadcasting to avoid synthesized speech being mistakenly recorded and triggering incorrect dialogue flows. However, this patent document's handling of self-auditory interference is essentially still an avoidance strategy of "mute the microphone during broadcasting." In half-duplex mode, the user still cannot interrupt the robot, and the usability of full-duplex mode is limited by the predefined dialogue domain.

[0004] Second, there is a disconnect between facial expressions and voice. The conventional solution to this problem is to first obtain the reply text, and then use an additional emotion classification model to perform secondary analysis on the text to obtain the emotion. This solution still has shortcomings, specifically including: the serial link introduces additional latency and secondary errors of hundreds of milliseconds or more; facial expressions usually switch only after the voice is played, presenting a delayed feeling of "the face changes only after half a sentence has been spoken," or an unnatural feeling of "the mouth is moving, but the face is expressionless," making the communication lack naturalness. Furthermore, patent application CN116714005A discloses a social robot and system with emotional interaction capabilities, which uses a multimodal emotion recognition model based on graph attention networks and a dialogue emotion analysis model based on DialogGCN to identify user emotions, and then combines a preset action library to select appropriate motion parameters and voice parameters for emotional expression. However, the patent document still has flaws. Its emotions come from the post-analysis of a dedicated classification model that is independent of the response generation link. "What to say" and "what emotion to bring" are generated by two separate links. The generation of emotions and the semantics of the response are easily disconnected and an additional reasoning delay is added. Furthermore, due to the lack of sequential design between facial expressions and speech playback, it cannot present a natural communication state of "expression first, then speech".

[0005] Third, it relies on third-party cloud interfaces and has high response latency. Calling large cloud model interfaces often introduces technical problems such as fluctuating network round-trip latency and uncontrollable tail latency; at the same time, general cloud interfaces can only output text and do not have the ability to customize settings for robots, so additional prompting engineering and post-processing are required, which will further increase latency.

[0006] Fourth, the sequential waiting is wasted. First, speech recognition, large-scale model inference, and whole-sentence speech synthesis are executed sequentially. The end-to-end latency from when the user "finishes speaking" to when the robot "speaks" is generally over 4.5 seconds, with the initial latency of the large-scale model's cold start reaching over ten seconds. Second, endpoint detection (i.e., determining whether the user has finished speaking) must wait for a trailing silence (i.e., a period of silence after the speech stops, typically 1.2 seconds) to distinguish between "pauses in the sentence" and "end of speech." During this trailing silence waiting period, the user has actually stopped speaking, and all audio is already in the buffer. Speech recognition and large-scale model inference are fully ready to start, but the sequential link causes computing resources to idle, wasting more than 1 second of valuable time. On the other hand, simply shortening the trailing silence threshold will misjudge breathing in the sentence as the end of speech, causing truncation errors. Secondly, the whole-sentence synthesis scheme only speaks after the entire audio sentence is synthesized. Even with streaming synthesis segmented by punctuation, the first clause of a Chinese response often consists of more than ten characters, and the synthesis time is approximately proportional to the number of characters (measured at around 0.1 seconds per character). This results in the first block still requiring more than one second. During the silent window before speaking, the robot has no response, making it impossible for the user to determine whether the robot has heard them, often leading to repeated questions and chaotic dialogue. To address the above issues, patent document CN119993150A discloses a real-time voice interaction method, system, computer device, and storage medium. It deploys speech recognition, language processing, and speech conversion modules on a local device to reduce data transmission latency and uses methods such as text streaming segmentation, pipelined parallelism between modules, and asynchronous audio generation to shorten response time. However, all calculations in this patented technology still start from endpoint confirmation, and the parallelism of each link only occurs within the processing chain "after the user finishes speaking." The long trailing silent waiting period of up to one second before endpoint confirmation is still wasted.

[0007] Fifth, multi-robot co-location failures. Simply co-locating multiple standalone dialogue systems when allowing two or more robot heads to converse or run simultaneously can lead to a series of failures. The following example illustrates this with two robot heads conversing. First, shared computing power contention. When two (side) robot heads simultaneously request the same large model inference service on a single GPU server, the contention for GPU memory and computing power will cause both sides' inference latency to deteriorate simultaneously, and may even trigger a memory overflow, causing the service to crash. Furthermore, a large model deployed on a single GPU essentially serves only one side at a time, lacking serialization guarantees. Second, acoustic interference: When two robot heads are in the same sound field, without coordinated speech, they may speak simultaneously, suppressing each other's voices, or one side's microphone may pick up the other's speaker sound and treat it as "user input," resulting in irrelevant answers and each speaking their own language. Secondly, there's the issue of repetitive loops: long dialogues driven by large models are prone to falling into a formulaic repetition pattern where "the surface-level vocabulary changes every round, and the sentence structure is repeated." Simple text duplication detection fails to detect this, and after ten or more rounds, the dialogue often gets stuck in a vicious cycle of endless repetition. Thirdly, there's the issue of failures causing deadlocks: if any proxy process on one side crashes, loses network connection, or becomes unresponsive for an extended period, the dialogue will stop on that side, leaving the other side in an infinite wait. To address these issues, in the area of ​​multi-device voice collaboration, patent document CN110442319A discloses a competing device that responds to voice triggers. When multiple devices simultaneously detect the same wake-up phrase, the device that meets the activation conditions sends a stop command to nearby devices, causing them to abandon responding to the same audio input. However, this technical solution only solves the problem of instantaneous answering of "which device responds to a wake-up call". It cannot guarantee the continuous mutual exclusion and orderly handover of speaking rights in multi-round dialogues, nor can it solve the unique problems of multi-robot continuous dialogues, such as the exclusive serialization of shared large model inference resources, the binding of speaking rights with speaker usage rights and microphone listening rights, and round timeout takeover and repeat detection.

[0008] In summary, while introducing large models into humanoid head robots has led to some improvements, existing technologies still have many technical shortcomings, resulting in humanoid head robots exhibiting unnatural performance and untimely responses in voice interaction and facial expression linkage. Summary of the Invention

[0009] This invention provides a robot voice-expression linkage method, system, device, and storage medium. Addressing the aforementioned deficiencies in the prior art, it adopts a novel approach to enable the robot head to not only understand human speech but also to respond while thinking. Simultaneously, the robot's mouth shape and facial expressions are synchronized and natural when speaking, and it can immediately stop when interrupted, thereby achieving autonomous conversation.

[0010] The robot voice-expression linkage method provided in this embodiment of the invention includes the following steps: S1. Within the fixed block length audio callback period, the speech synthesis block just written to the speaker output buffer is used as the far-end reference signal and aligned with the near-end audio signal sampled by the microphone. The echo-cancelled audio signal is obtained after acoustic echo cancellation. S2. The echo-cancelled audio signal is detected frame by frame using a neural network speech activity detection model. When the duration of trailing silence reaches the endpoint threshold, the user's speech is determined to have ended and a segment of the user's speech is captured. S3. Perform speech recognition on the user's voice to obtain transcribed text; S4. The transcribed text, image frames captured and encoded by the binocular camera, personality system prompts, and multi-turn dialogue history are constructed into a multimodal message; S5. The multimodal message is fed into a visual language big model with its own weights. The visual language big model performs a single inference and outputs the response text and discrete sentiment tags. S6. Before speech synthesis begins, based on the discrete emotion tags, generate first servo control information to drive the facial servo group to make a gesture corresponding to the discrete emotion tags. The first servo control information is used to control the facial expression. S7. Based on the reply text, the voice information is synthesized by streaming speech and played sentence by sentence by the speaker; S8. Generate second servo control information based on the voice information and its voice waveform, wherein the second servo control information includes information for generating lip movements. S9. Based on the first servo control information and the second servo control information, and taking continuous emotion coordinates as a condition, generate third servo control information to drive continuous changes in facial expressions.

[0011] Optionally, step S9 may be followed by: S10. During playback, interruption detection is performed on the audio signal after echo cancellation. When the user's voice meets the interruption condition, playback is interrupted and the process returns to step S2 to enter the listening state to obtain a segment of user voice.

[0012] Optionally, the discrete emotion tags in step S6 include a set of 13 emotions, such as smiling, gratified, happy, coquettish, shy, aggrieved, confused, thinking, serious, angry, frustrated, sleepy, and surprised, and each emotion tag corresponds to a set of preset first servo control information.

[0013] Optionally, step S2 includes the following specific steps: S201. Before the trailing silence duration reaches the endpoint threshold, the following two conditions are simultaneously met: the current trailing silence duration of the echo-cancelled audio signal reaches the speculation threshold, and the accumulated speech duration is not less than the minimum speech duration. These conditions are used as the judgment conditions to trigger speculation. The speculation threshold is preset to be less than the endpoint threshold, so that the judgment conditions are met before the endpoint is confirmed. When the judgment conditions are met, the following actions are performed: the current speculation generation count value is read as the generation marker of this speculation, and speech recognition is started in advance on the cached echo-cancelled audio signal to obtain the speculation transcribed text. Then, a speculation response is generated based on the speculation transcribed text. The speculation transcribed text, the speculation response, and the generation marker together constitute the speculation result carrying the generation marker. S202. When the duration of trailing silence falls below the speculation threshold, it is determined that the user resumes speaking and the increased speculation generation count value is submitted, thus invalidating the speculation result carrying the old generation mark. S203. When the duration of the trailing silence reaches the endpoint threshold and the user's speech is deemed to have ended, the generational marker of the speculation result is verified against the current generational count of the speculation, and the following three-way arbitration is executed: First, if the generational marker matches the current generational count and the speculation result is ready, the speculation transcript and speculation reply are directly adopted, skipping the speech recognition in step S3 and the reasoning in step S5, and steps S6 to S9 are continued with the adopted speculation result, that is, generating the first servo control information based on the discrete emotion tags in the speculation result, performing streaming speech synthesis and sentence-by-sentence playback on the speculation reply, and generating the second and third servo control information; the speculation result being ready refers to the speech recognition and... First, if the inference has been completed, its generational verification has passed, and the speculative transcription text and speculative response have been written into the speculative result, it can be directly read. Second, if the generational marker is consistent with the current speculative generational count value but the speculative thread is still executing and the speculative result has not yet been written, the speculative thread is waited for a limited time to complete. If it is completed within the waiting limit, it is handled according to the first method mentioned above. If it is still not ready after exceeding the waiting limit, it is handled according to the third method below. Third, if it is still not ready after exceeding the waiting limit, or if the generational marker is inconsistent with the current speculative generational count value, the speculative result is discarded, and speech recognition and inference are re-executed according to steps S3 to S5. Among them, the speculative threshold is 0.4 seconds, the minimum speech duration is 0.4 seconds, and the waiting limit is 1.5 seconds.

[0014] Optionally, step S2, and especially step S203, may include the following: S204. After determining that the user has finished speaking, pre-synthesized and cached background audio is immediately played through the speaker to mask the synthesis delay of the first speech block of the response. The background audio refers to a piece of spoken language that is played before the synthesis of the first speech block of the response is ready to fill the silence window. It does not carry the semantics of the response and only allows the robot to speak immediately after determining that the user has finished speaking. The background audio comes from a preset word pool containing spoken words. When the system starts, the background thread pre-synthesizes and caches the word pool word by word. Whether to play it in each round is determined according to a preset playback probability. The preset playback probability is 0.4, and the duration of playback from the word pool is between 0 and 4. One candidate is randomly selected from those between 0.2 seconds and 0.85 seconds, and the duration interval between 0.2 seconds and 0.85 seconds is used as a duration guard to filter out candidates whose rendering is dragged out by the synthesis engine, so that the playback end time of the background audio is no later than the ready time of the first speech block of the response, thereby preventing the overlap of the background audio and the first speech block of the response playing on the speaker at the same time; in step S7, the streaming speech synthesis limits the maximum number of characters in the first clause to 5 Chinese characters. If the first clause has more than 5 Chinese characters and there are no punctuation marks within 5 Chinese characters, it is forcibly segmented at the 5th Chinese character; in low latency mode, the endpoint threshold is 0.7 seconds.

[0015] Optionally, this method further includes a model warming step: a background daemon thread sends lightweight warming requests to the visual language large model and the speech recognition model used in step S3 at preset intervals. The lightweight warming requests sent to the visual language large model are word-level requests, and the lightweight warming requests sent to the speech recognition model are silent audio requests. This keeps the model weights of both models resident in GPU memory and the inference engine in a hot state, thereby eliminating the cold start delay of the first round of dialogue after idle switching. The preset interval is 30 to 60 seconds. The lightweight warming requests and dialogue requests share the same interface and queue. When there is a dialogue request in transit, the lightweight warming requests of this interval are skipped.

[0016] Optionally, when the method is applied to a robot system including at least two robot heads, the method further includes a speaking rights coordination step: a central coordinator maintains an authoritative state snapshot containing the running status, speaking rights stage, speaking rights ownership, round number, and coordination generation count value. The speaking rights stage alternates between a handover stage indicating a round and an occupation stage indicating a round in progress. Each robot head's corresponding agent periodically reads the authoritative state snapshot. When it reads that the running status is running, the speaking rights stage is a handover stage, and the speaking rights belong to itself, it sends a speaking rights request to the central coordinator, carrying the read coordination generation count value and round number. The central coordinator verifies within a mutex whether the running status is running, whether the speaking rights stage is a handover stage, and whether the requester is the party that owns the speaking rights. If the coordination generation count and round number are both true, the speaking right phase is set to the occupied phase, the coordination generation count is incremented, and the application is returned as successful. If either is false, an expired rejection is returned with the latest authority state snapshot, thus structurally rejecting the speaking right application based on the expired snapshot. The robot head that successfully applied exclusively takes over the inference of the visual language big model and the playback of the speaker and executes steps S3 to S9. The robot head that did not obtain the speaking right pauses its microphone listening and keeps its speaker silent. After the robot head finishes playing, it initiates a round submission to the central coordinator. After the same verification, the central coordinator flips the speaking right ownership to another robot head, sets the speaking right phase back to the handover phase, increments the round number, and increments the coordination generation count again.

[0017] This invention also provides a robot voice-expression linkage system, comprising: The echo cancellation module is used to take the speech synthesis block that has just been written into the speaker output buffer as the far-end reference signal within the fixed-block-length audio callback, align it with the near-end audio signal sampled by the microphone, and obtain the echo-cancelled audio signal after acoustic echo cancellation. The endpoint detection module is used to perform frame-by-frame detection on the echo-cancelled audio signal using a neural network speech activity detection model. When the duration of trailing silence reaches the endpoint threshold, it determines that the user's speech has ended and intercepts the user's speech, and continuously outputs the current duration of trailing silence and the accumulated speech duration. The speech recognition module is used to perform speech recognition on the user's speech to obtain transcribed text; The message construction module is used to construct a multimodal message from the transcribed text, binocular camera image frames, personality system prompts, and multi-turn dialogue history. The large model inference module is used to send the multimodal messages into a locally deployed, autonomously trained and fine-tuned visual language large model with its own weights, and output the response text and discrete sentiment tags in a single inference. The expression-first module is used to query the emotion-attitude mapping table based on the discrete emotion tags and drive the facial servo group to make full-face expressions before the speech synthesis begins. The synthesis and lip-syncing module is used to perform streaming speech synthesis on the reply text and play it sentence by sentence. At the same time, the neural lip-syncing model outputs joint angle sequences frame by frame based on the synthesized speech waveform to drive the facial servo group to produce lip-syncing movements. An interruption detection module is used to detect interruptions in the echo-cancelled audio signal during playback, and to interrupt playback and switch to listening mode when a user interruption is detected. The speculation execution module is used to read the current speculation generation count value as a generation marker when the current trailing silence duration reaches the speculation threshold and the accumulated speech duration is not less than the minimum speech duration. The background thread starts speech recognition in advance on the cached echo-cancelled audio signal and cascades the inference of the visual language model in advance to obtain the speculation result carrying the generation marker. When the user resumes speaking, the speculation generation count value is incremented to invalidate the old speculation result. The generation marker is checked when the user finishes speaking. If it is consistent and ready, the speculation result is adopted. If it is in execution, it is waited for a time limit. Otherwise, it is rolled back to the normal path. The pre-synthesized and cached pre-synthesized audio is played through the speaker immediately after the user finishes speaking. The maximum number of characters in the first clause of the streaming speech synthesis is limited, and forced segmentation is performed if the maximum number of characters is exceeded. The coordinator module is used to maintain the ownership of speaking rights, speaking rights stage, coordination generation count value and round number when the system includes at least two robot heads. It arbitrates the speaking rights application of the corresponding agent of each robot head by comparison and exchange, ensuring that at most one robot head has exclusive control over the inference and speaker playback of the visual language big model at any given time. It forcibly transfers speaking rights when the round times out or the heartbeat is interrupted, and performs repeat detection and prompt injection on the most recent reply text.

[0018] This invention also provides an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements any of the aforementioned robot voice-expression linkage methods.

[0019] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the aforementioned robot voice-expression linkage methods.

[0020] The beneficial effects achieved by the robot voice-expression linkage method, system, device, and storage medium provided in this invention are: (1) Thorough self-hearing cancellation. In the embodiment, the far-end reference signal and the near-end audio signal are aligned sample by sample within the same 20-millisecond audio callback, the echo estimation is accurate, and the measured self-hearing cancellation rate is greater than 99%. The robot can continue to listen to the user while it is speaking without having to turn off the microphone.

[0021] (2) Fast interruption response. Based on the interruption detection of the audio signal after echo cancellation, it is triggered after 250 to 400 milliseconds of continuous speech. The actual test showed that the reaction time from the user speaking to the robot stopping playback is less than 1 second, and the interaction is close to the natural rhythm of people talking to each other.

[0022] (3) Expression first, natural linkage. The single inference outputs the reply text and discrete emotion tags, eliminating the hundreds of milliseconds of serial delay in secondary emotion classification; the discrete emotion tags drive the whole face expression before the first speech synthesis block is played, realizing "expression first, then speech"; during playback, the lip movements are aligned with the speech frame by frame at 30 frames per second, and the whole face expression changes smoothly with the continuous emotion coordinates, eliminating the stiffness of "mouth moving but face not moving".

[0023] (4) Autonomous and controllable. The visual language big model is a self-weighted model that continues to be trained autonomously and deployed locally for inference. There is no network round-trip latency fluctuation. The dialogue audio and video data does not leave the local network. Moreover, the model natively has the robot's first-person embodied alignment and joint output capabilities.

[0024] (5) Speculative execution hides speech recognition and large model inference in the tailing silence waiting period. The result is usually ready when the endpoint is confirmed. The actual test saves 0.3 to 1.0 seconds per round. The intergenerational speculation mechanism ensures that speculation is correct in any time sequence: when the user continues to speak, the old speculation is automatically invalidated and a new speculation is made. The invalidated speculation has zero pollution to the shared multi-round dialogue history. The adopted speculation is recorded exactly once. When arbitration fails, it reverts to the normal path. In the worst case, it is equivalent to the original serial link. The benefits only increase and do not decrease.

[0025] (6) The pre-emptive speech allows the robot to speak immediately after the endpoint confirmation (the pre-emptive audio is pre-buffered and has zero synthesis overhead during playback). The user-perceived "no response window" is shortened from several seconds to close to zero. The random word pool and 0.4 playback probability avoid the mechanical feeling of the same spoken words in each round. The first block hard cut stabilizes the synthesis time of the first speech block at 0.6 seconds, decoupling it from the total length of the response.

[0026] (7) The delay was significantly reduced. The actual measured end-to-end "user finishes speaking - robot speaks" delay was reduced from more than 4.5 seconds to 1.5 to 2.5 seconds after the flow synthesis, model insulation, speculative execution, padding opening and first hard cut were superimposed. The insulation mechanism reduced the first-round delay from more than 13 seconds to the same as the thermal state.

[0027] (8) Multi-robot structural serialization and acoustic interference reduction. The comparison and exchange of speaking rights requests make "at most one head is reasoning and at most one head is speaking" a protocol invariant rather than a convention. The reasoning latency of the large visual language model deployed on a single card is consistent with that of the single-head scenario in the dual-robot head scenario, without the risk of concurrent contention and memory overflow. The speaking rights are bound to "speaker usage rights and microphone listening rights". The silent party pauses listening, which eliminates the erroneous cascading of one party recognizing the speaker sound of the other party from the mechanism.

[0028] (9) Multi-robot fault self-healing and repetition elimination. Two-level monitoring of 120-second round timeout and 8-second heartbeat timeout ensures that the speaking rights are automatically transferred after any agent crashes or disconnects from the network, and the dialogue can continue without manual intervention; sentence skeleton-level verbal slang detection makes up for the blind spots of word literal detection, and with the help of character binary similarity and periodic director prompts, in actual tests, long dialogues of more than 20 rounds no longer fall into the dead loop of semantic repetition. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of the robot voice-expression linkage method in an embodiment of the present invention; Figure 2 This is an overall block diagram of a robot multimodal voice interaction system for implementing a robot voice-expression linkage method, provided in an embodiment of the present invention. Figure 3 A flowchart of a dialogue loop provided in an embodiment of the present invention; Figure 4 This is a timing diagram for implementing a robot voice-expression linkage method, which includes emotion prioritization and lip-sync linkage, as provided in an embodiment of the present invention. Figure 5 This is a block diagram of a full-duplex echo cancellation and interruption detection method for implementing robot voice-expression linkage, provided in an embodiment of the present invention. Figure 6 This is a schematic diagram illustrating the construction and joint output of multimodal messages for implementing a robot voice-expression linkage method, provided in an embodiment of the present invention. Figure 7 A schematic diagram of the serial and parallel timing comparison and the speculative execution state machine for speculative execution of a robot voice-expression linkage method provided in an embodiment of the present invention; Figure 8This is a schematic diagram of the topology and speaking right state machine of a dual-robot dialogue system for implementing a robot voice-expression linkage method, provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0033] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a communication connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. The technical solutions of this invention are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0034] like Figure 1 As shown, the robot voice-expression linkage method provided in this embodiment of the invention includes the following steps: S1. Within the fixed block length audio callback period, the speech synthesis block just written to the speaker output buffer is used as the far-end reference signal and aligned with the near-end audio signal sampled by the microphone. The echo-cancelled audio signal is obtained after acoustic echo cancellation. S2. The echo-cancelled audio signal is detected frame by frame using a neural network speech activity detection model. When the duration of trailing silence reaches the endpoint threshold, the user's speech is determined to have ended and a segment of the user's speech is captured. S3. Perform speech recognition on the user's voice to obtain transcribed text; S4. The transcribed text, image frames captured and encoded by the binocular camera, personality system prompts, and multi-turn dialogue history are constructed into a multimodal message; S5. The multimodal message is fed into a visual language big model with its own weights. The visual language big model performs a single inference and outputs the response text and discrete sentiment tags. S6. Before speech synthesis begins, based on the discrete emotion tags, generate first servo control information to drive the facial servo group to make a gesture corresponding to the discrete emotion tags. The first servo control information is used to control the facial expression. S7. Based on the reply text, the voice information is synthesized by streaming speech and played sentence by sentence by the speaker; S8. Generate second servo control information based on the voice information and its voice waveform, wherein the second servo control information includes information for generating lip movements. S9. Based on the first servo control information and the second servo control information, and taking continuous emotion coordinates as a condition, generate third servo control information to drive continuous changes in facial expressions.

[0035] It should be noted that in step S9, the continuous emotion coordinates are two-dimensional coordinates composed of a pleasure component and an excitement component, with both components ranging from -1 to 1. They are obtained by mapping the discrete emotion labels and their emotion intensities as described in step S5. The condition of using continuous emotion coordinates means that the continuous emotion coordinates are used as a condition vector and injected into the expression generation model that generates the third servo control information using a feature linear modulation method. That is, the scaling coefficient and bias coefficient are predicted from the continuous emotion coordinates by the multilayer perceptron, and the intermediate features of the expression generation model are linearly transformed channel by channel. Interpolation is performed frame by frame between the continuous emotion coordinates of adjacent sentences to make the facial expression transition smoothly with the content of the sentence.

[0036] Preferably, the robot voice-expression linkage method provided in this embodiment of the invention further includes the following after step S9: S10. During playback, interruption detection is performed on the audio signal after echo cancellation. When the user's voice meets the interruption condition, playback is interrupted and the process returns to step S2 to enter the listening state to obtain a segment of user voice.

[0037] Preferably, the discrete emotion tags in step S6 include a set of 13 emotions, such as smiling, gratified, happy, coquettish, shy, aggrieved, confused, thinking, serious, angry, frustrated, sleepy, and surprised, and each emotion tag corresponds to a set of preset first servo control information.

[0038] Preferably, step S2 of the robot voice-expression linkage method provided in this embodiment of the invention includes the following specific steps: S201. Before the trailing silence duration reaches the endpoint threshold, the following two conditions are simultaneously met: the current trailing silence duration of the echo-cancelled audio signal reaches the speculation threshold, and the accumulated speech duration is not less than the minimum speech duration. These conditions are used as the judgment conditions to trigger speculation. The speculation threshold is preset to be less than the endpoint threshold, so that the judgment conditions are met before the endpoint is confirmed. When the judgment conditions are met, the following actions are performed: the current speculation generation count value is read as the generation marker of this speculation, and speech recognition is started in advance on the cached echo-cancelled audio signal to obtain the speculation transcribed text. Then, a speculation response is generated based on the speculation transcribed text. The speculation transcribed text, the speculation response, and the generation marker together constitute the speculation result carrying the generation marker. S202. When the duration of trailing silence falls below the speculation threshold, it is determined that the user resumes speaking and the increased speculation generation count value is submitted, thus invalidating the speculation result carrying the old generation mark. S203. When the duration of the trailing silence reaches the endpoint threshold and the user's speech is deemed to have ended, the generational marker of the speculation result is verified against the current generational count of the speculation, and the following three-way arbitration is executed: First, if the generational marker matches the current generational count and the speculation result is ready, the speculation transcript and speculation reply are directly adopted, skipping the speech recognition in step S3 and the reasoning in step S5, and steps S6 to S9 are continued with the adopted speculation result, that is, generating the first servo control information based on the discrete emotion tags in the speculation result, performing streaming speech synthesis and sentence-by-sentence playback on the speculation reply, and generating the second and third servo control information; the speculation result being ready refers to the speech recognition and... First, if the inference has been completed, its generational verification has passed, and the speculative transcription text and speculative response have been written into the speculative result, it can be directly read. Second, if the generational marker is consistent with the current speculative generational count value but the speculative thread is still executing and the speculative result has not yet been written, the speculative thread is waited for a limited time to complete. If it is completed within the waiting limit, it is handled according to the first method mentioned above. If it is still not ready after exceeding the waiting limit, it is handled according to the third method below. Third, if it is still not ready after exceeding the waiting limit, or if the generational marker is inconsistent with the current speculative generational count value, the speculative result is discarded, and speech recognition and inference are re-executed according to steps S3 to S5. Among them, the speculative threshold is 0.4 seconds, the minimum speech duration is 0.4 seconds, and the waiting limit is 1.5 seconds.

[0039] Preferably, the robot voice-expression linkage method provided in this embodiment of the invention further includes, after step S2, and particularly step S203: S204. After determining that the user has finished speaking, pre-synthesized and cached background audio is immediately played through the speaker to mask the synthesis delay of the first speech block of the response. The background audio refers to a piece of spoken language that is played before the synthesis of the first speech block of the response is ready to fill the silence window. It does not carry the semantics of the response and only allows the robot to speak immediately after determining that the user has finished speaking. The background audio comes from a preset word pool containing spoken words. When the system starts, the background thread pre-synthesizes and caches the word pool word by word. Whether to play it in each round is determined according to a preset playback probability. The preset playback probability is 0.4, and the duration of playback from the word pool is between 0 and 4. One candidate is randomly selected from those between 0.2 seconds and 0.85 seconds, and the duration interval between 0.2 seconds and 0.85 seconds is used as a duration guard to filter out candidates whose rendering is dragged out by the synthesis engine, so that the playback end time of the background audio is no later than the ready time of the first speech block of the response, thereby preventing the overlap of the background audio and the first speech block of the response playing on the speaker at the same time; in step S7, the streaming speech synthesis limits the maximum number of characters in the first clause to 5 Chinese characters. If the first clause has more than 5 Chinese characters and there are no punctuation marks within 5 Chinese characters, it is forcibly segmented at the 5th Chinese character; in low latency mode, the endpoint threshold is 0.7 seconds.

[0040] Preferably, the robot voice-expression linkage method provided in this embodiment of the invention further includes a model warming step: a background daemon thread sends lightweight warming requests to the visual language large model and the speech recognition model used in step S3 at a preset period, wherein the lightweight warming request sent to the visual language large model is a word-level request and the lightweight warming request sent to the speech recognition model is a silent audio request, so that the model weights of both are resident in the video memory and the inference engine is kept in a hot state to eliminate the cold start delay of the first round of dialogue after idle switching; the preset period is 30 seconds to 60 seconds; the lightweight warming request and the dialogue request share the same interface and queue, and the lightweight warming request of the current period is skipped when there is a dialogue request in transit.

[0041] Preferably, when the robot voice-expression linkage method provided in this embodiment of the invention is applied to a robot system including at least two robot heads, it further includes a speaking right coordination step: a central coordinator maintains an authoritative state snapshot containing the running status, speaking right stage, speaking right ownership, round number, and coordination generation count value. The speaking right stage alternates between a handover stage indicating a round and an occupation stage indicating a round in progress. The agent terminal corresponding to each robot head periodically reads the authoritative state snapshot. When it reads that the running status is running, the speaking right stage is a handover stage, and the speaking right belongs to itself, it sends a speaking right request to the central coordinator with the read coordination generation count value and round number. The central coordinator verifies within a mutex whether the running status is running, whether the speaking right stage is a handover stage, and whether the requester is... If the speaker's right, the coordination generation count value, and the round number are all equal to the current value, the speaker's right phase is set to the occupied phase, the coordination generation count value is incremented, and the application is returned as successful. If any one of these conditions is not met, an expired rejection is returned carrying the latest authority status snapshot, thus structurally rejecting the speaker's right application based on the expired snapshot. The robot head that successfully applied exclusively takes over the inference of the visual language big model and the playback of the speaker and executes steps S3 to S9. The robot head that did not obtain the right to speak pauses its microphone listening and keeps its speaker silent. After the robot head finishes playing, it initiates a round submission to the central coordinator. After the same verification, the central coordinator flips the speaker's right to another robot head, sets the speaker's right phase back to the handover phase, increments the round number, and increments the coordination generation count value again.

[0042] To explain the above robot voice-expression linkage method in detail, the following section provides a detailed explanation of each step using practical, implementable hardware.

[0043] Example 1: Implementation of Basic Dialogue Loop (I) System Overall Architecture like Figure 2 As shown, the robot system provided in this embodiment, which can apply the aforementioned robot voice-expression linkage method, is divided into three layers: robot head 101, host computer 102, and GPU server 103. The robot head 101 is equipped with a microphone 104, a binocular camera 105, a speaker 106, a facial servo motor group 107, and a control box 108. The control box 108 receives the entire face joint angle dictionary at port 8765 using a ZMQ (Inter-Process Network Message Library) request-response method, and distributes it to each servo motor control board via Modbus serial bus at a 20-millisecond frame rate, driving more than twenty joints of the facial servo motor group 107. The image frames from the binocular camera 105 and the audio from the microphone 104 are sent to the host computer 102 via the network, and the speaker 106 receives and plays PCM16 (16-bit uncompressed raw digital audio format) audio data via port 4321. On GPU server 103, a visual language large model service 109 (listening on port 8770, providing / asr, / asr_stream, / chat, / reason interfaces), a streaming speech synthesis service 110 (listening on port 8762, providing / tts and / tts_stream interfaces), and a lip-reading inference service 111 (listening on port 8771, providing the / face interface) are deployed. The host computer 102 runs a dialogue master program, connecting the complete loop of "acquisition—echo cancellation—endpoint detection—recognition—inference—expression—synthesis—lip-reading—interruption."

[0044] In another embodiment, the functions of the host computer 102 can also be directly undertaken by the Raspberry Pi inside the robot head 101 (Raspberry Pi direct connection mode). The Raspberry Pi directly accesses ports 8770, 8762, and 8771 of the GPU server 103 via HTTP, eliminating the need for an intermediate computer. The method described in this invention is valid in both deployments.

[0045] (ii) Step S1: Full-duplex echo cancellation with sample-level alignment like Figure 4As shown, preferably, the audio callback 401 can operate with a block length of 20 milliseconds and a sampling rate of 16kHz (kHz means kilohertz, here referring to a sampling rate of 16,000 sample points per second): Each callback first pops the next 20-millisecond speech synthesis block from the output buffer 402 as a far-end reference signal and writes it to the sound card output (writes a zero signal when there is no speech to be played), then takes the microphone near-end audio signal 403 sampled in this callback, and sends both to the echo canceller 404. Since the far-end reference signal is "the block that was just written out in this callback", the far end and near end are naturally aligned sample by sample, without the need for additional alignment search. Here, the far-end reference signal (the sound to be played by the speaker) and the near-end audio signal (the sound sampled by the microphone) are both common terms in the field of acoustic echo cancellation. Acoustic echo cancellation (AEC) is a conventional technique in this field and is widely used in scenarios such as hands-free calling and video conferencing to eliminate sound feedback from one's own speaker. The echo canceller 404 in this embodiment can directly adopt the WebRTC AEC3 algorithm from Google's open source, with a sampling rate of 16kHz. Preferably, the echo path delay parameter stream_delay_ms is set to 150 milliseconds (covering the total delay of the sound card output buffer, the air path from the speaker to the microphone, and the input buffer).

[0046] It should be noted that the full-duplex echo cancellation with sample-level alignment is the core of this invention, which distinguishes it from existing technologies. The key point is that it uses a single 20-millisecond audio callback as the atomic unit, and directly registers the "TTS voice block that was just pushed to the speaker in this callback" as the far-end reference signal (registered as an equal-length all-zero block when there is no voice to be played). It shares the same sampling time base with the near-end signal sampled by the microphone in the same callback and corresponds one-to-one with each sample. Alignment can be completed without cross-correlation search or time delay estimation, and the whole block is sent to AEC to cancel its own sound. Therefore, the echo estimation is accurate and the measured self-hearing cancellation rate is greater than 99%, so that the robot can still listen to the user uninterruptedly while it is speaking.

[0047] This core idea fundamentally differs from the existing "mute / half-duplex" avoidance approach—existing technologies rely on the robot shutting down or ignoring the microphone when speaking to avoid self-hearing, at the cost of user interruption, interaction degenerating into one-way alternation, and its full-duplex availability being limited to a predefined dialogue domain. This invention, by adopting the above-mentioned technical approach, does not rely on mute or limit topics, thus creating conditions for full-duplex listening under any topic and sub-second interruption in step S8. Actual testing shows that its output, the echo-cancelled audio signal 405, has a cancellation rate of over 99% for the robot's own playback sound. The callback only performs deterministic, rapid operations (blocking, writing, echo cancellation, enqueueing), with all time-consuming processing completed in a thread outside the callback, ensuring uninterrupted audio flow. When echo cancellation fails, the original microphone signal is directly transmitted according to the failure-open strategy, ensuring system availability. The playback side maintains a "number of actually output samples" counter, serving as the playback clock for lip-syncing later. Output buffer 402 also retains the most recent 4 seconds of remote history, which can be used to rewind the user's initial speech after an interruption is established.

[0048] (III) Step S2: Neural Network Endpoint Detection After echo cancellation, the audio signal 405 is sent to the endpoint detector 406. Voice activity detection (VAD, i.e., determining the presence of human voices in the audio frame by frame, a technique already existing in the field) and endpoint detection based on this to determine whether a sentence has been finished are performed. The endpoint detector 406 uses the open-source pre-trained Silero VAD version 6 neural network model (model size on the order of 2MB, which can be run in real time on a CPU). This model requires 512 samples per input (i.e., 32 milliseconds per frame) at a 16kHz sampling rate, and outputs a speech probability of 0 to 1 frame by frame.

[0049] Preferably, one possible determination logic is as follows: frames with a speech probability not less than a threshold of 0.5 are recorded as speech frames; when two consecutive frames are speech frames, it is determined that the user has started speaking (to eliminate jitter and prevent false triggering due to single-frame glitches); after the user starts speaking, when the cumulative speech frame duration is not less than 0.2 seconds and the trailing silence duration reaches the endpoint threshold, it is determined that the user has ended speaking. The endpoint threshold is set to 1.2 seconds in stable mode and 0.7 seconds in low-latency mode. After each frame update, the endpoint detector 406 simultaneously outputs the current trailing silence duration and the accumulated speech duration. These two continuously readable measurements are the triggering basis for speculative execution in Embodiment 2. Compared with the scheme that determines silence based on a fixed energy threshold, the neural network endpoint detection significantly reduces the false judgment rate under interference from environmental noise, keyboard sounds, and air conditioning sounds.

[0050] (iv) Step S3: Speech Recognition After the speech ends, the echo-cancelled audio signal (mono PCM16, 16kHz) from "start speaking" to "end speaking" is extracted, encoded in base64 (converting binary data into a string that can be transmitted with text), and submitted via HTTP to port 8770 / asr interface of GPU server 103, and the transcribed text is returned.

[0051] Preferably, in this embodiment, the speech recognition model is the FunASR Chinese model, and the actual time for single-sentence recognition is 0.15 seconds to 0.3 seconds; in another embodiment, the Whisper model half-precision inference can be used, with a time of 0.2 seconds to 0.4 seconds.

[0052] In the multilingual implementation, the recognition request is redirected to the multilingual routing service on port 8766. According to the language configuration table, it supports the recognition of 13 languages ​​and dialects. The recognized language and the synthesized language can be set independently.

[0053] (v) Step S4: Multimodal message construction like Figure 5 As shown, preferably, the multimodal message 505 is constructed from four parts: transcribed text 501, image frame 502, personality system prompt 503, and multi-turn dialogue history 504.

[0054] Image frame 502 is taken from the binocular camera 105: each of the left and right channels is 640×480 pixels and is stitched together to form a single frame of 1280×480 pixels. After being encoded as JPEG, it is converted to base64 data URL. Several frames (preferably 2 frames) are extracted from the recent frame buffer at the minimum interval, so that the model can see the current scene and also have short-term dynamic information.

[0055] The personality system prompt 503 is the robot's persona text, declaring its identity, speech style, and specific settings such as "you have binocular vision and can make facial expressions," and specifying the output format as a JSON object containing two fields: response text and emotion tag. The multi-turn dialogue history 504 retains the most recent 8 rounds of "user-robot" question-and-answer pairs to maintain contextual coherence. These four parts are assembled into a multimodal message 505 in the order of "system prompt + history + (image frame + current user text)."

[0056] (vi) Step S5: Self-training of the visual language large model for single-step joint inference Multimodal message 505 is sent to the visual language large model 506. Preferably, the visual language large model in this embodiment is based on the Qwen3-30B-A3B hybrid expert architecture, and through autonomous training and instruction fine-tuning, a robot embodied dialogue model with its own weights can be obtained.

[0057] Among them, Mixture of Experts (MoE) is a known large model architecture type in this field. It means that multiple groups of "expert" subnetworks are arranged in parallel within the model. Each inference is only activated by the routing module to participate in the calculation. In this way, while the total number of parameters is very large, the number of activated parameters and the amount of computation in a single inference are small, thus balancing capacity and speed. In this example, the total number of base parameters is about 30 billion, and only about 3 billion parameters are activated for each word.

[0058] Through actual testing, the applicant autonomously continued training and fine-tuning on this platform, obtaining a self-weighted embodied dialogue model and its joint output mechanism capable of jointly outputting "response text + discrete emotion tags" in a single inference. Its training objectives included three aspects: Firstly, joint output format alignment (a non-public core concept of this invention) – During the training phase, the target output of each sample is uniformly constructed as a JSON object containing two fields: reply and emotion. The reply field is the reply text, and the emotion field is the emotion tag. The emotion field value is limited to a fixed enumeration of 13 discrete emotions, as described later. Fine-tuning is used to ensure that the model generates the emotion field during the same forward decoding process when generating the reply, thus binding "what to say" and "what emotion to convey" to the same autoregressive probability chain, resulting in natural semantic consistency. During the inference phase, the model output is parsed according to JSON to separate the reply text 507 and the discrete emotion tag 508, obtaining both simultaneously in a single inference, eliminating the need for a second emotion classification inference. Compared to the existing two-model serial scheme of "dialogue model first outputs reply, then independent emotion classification model performs secondary analysis," the robot voice-expression linkage method provided in this embodiment does not require a second model, has no secondary inference delay, and the emotion and reply are of the same origin and are not easily separated.

[0059] Secondly, the robot's first-person embodied alignment—the training corpus uses "I am a bionic robot head with binocular vision and the ability to make facial expressions" as the first-person setting, covering image-based responses, greetings while being looked at, and referential understanding of objects and people in the camera.

[0060] Third, the short response style in spoken Chinese—the length distribution of responses converges towards short spoken sentences, which is suitable for the rhythm of voice broadcasting.

[0061] After testing, the trained model was deployed on GPU server 103 and used the vLLM inference framework, which supports tensor parallelism and continuous batch processing, for local inference. The entire inference process did not call any third-party cloud interfaces, and the dialogue text and image data did not leave the local network.

[0062] In actual testing, the single-round inference time under hot conditions was 3 to 5 seconds, with the pre-filling stage taking 2 seconds and the decoding speed at 18 words per second. The first round of cold start could take up to 13 seconds. To eliminate cold start, the host computer 102 sends a lightweight word-level warm-up request to port 8770 at a fixed period (preferably 30 to 60 seconds) through a background thread, keeping the model weights and inference cache resident in GPU memory, and the first-round latency is consistent with that under hot conditions. The model output is parsed to obtain the response text 507 and discrete sentiment label 508; when parsing fails, it is processed as a plain text response with a neutral sentiment, a downgraded path.

[0063] (vii) Step S6: Emotion-driven facial expressions like Figure 4 As shown, after the large model output time 301 on the timing axis and before the first voice block of the voice channel 303 is played, the facial expression channel 302 completes the full face expression distribution, forming an expression advance time difference 305 (measured to be 0.5 seconds to 1.5 seconds, depending on the synthesis time of the first block).

[0064] Specifically, the discrete emotion label 508 is taken from a set of 13 emotions: smile, pleased, happy, act cute, shy, wronged, confused, pondering, serious, angry, dejected, drowsy, and surprised. The emotion-posture mapping table maps each emotion to a set of preset facial servo postures (covering the target angles of the joints of the eyebrows, eyelids, cheeks, and corners of the mouth); after obtaining the discrete emotion label, it is immediately looked up in the table and sent to the control box 108 via the 8765 port to drive the facial servo group 107 first.

[0065] Since speech synthesis has not yet begun, the observer first sees the robot's "expression change" before hearing it speak, which aligns with the human intuition that "emotion precedes speech," significantly enhancing the anthropomorphic feel. Furthermore, it should be noted that this "emotion precedes speech" timing design is the core difference between the robot's speech-expression linkage method proposed in this invention and existing technologies. Its feasibility stems from the joint output of step S5: since the response text and discrete emotion tags are given simultaneously by the large model in the same inference, the emotion tags are available before the response text begins streaming synthesis and before the first speech block is played. Based on this, the system completes the emotion lookup and full-face expression delivery within the time window when the speech channel is still silent, forming an expression-priority time difference of 3. 05 (measured from 0.5 to 1.5 seconds), achieving "expression first, speech later" in terms of timing. The robot voice-expression linkage method proposed in this invention differs from the existing serial post-processing approach of "generating reply text first → secondary analysis by an independent emotion classification model → finally matching expression": the latter's emotions come from a separate link from the reply generation, which not only adds hundreds of milliseconds of secondary inference delay, but also often switches expressions only after the voice has been broadcast, presenting a lag feeling of "saying half a sentence and then the face changes"; this invention eliminates the secondary delay by joint output and actively prioritizes the distribution of expressions before the voice, so that the robot's emotions conform to the human-like rule of "emotions preceding speech", which is the key component of the anthropomorphic communication timing of this invention.

[0066] (viii) Step S7: Streaming Synthesis Playback and Frame-by-Frame Lip Movement Drive The reply text 507 is segmented into clauses according to punctuation. Each clause is called via the streaming speech synthesis interface on port 8762 to synthesize a 24kHz mono PCM16 speech block. Synthesis and playback occur simultaneously: the first clause begins playing as soon as it is synthesized, and subsequent clauses are synthesized in parallel while playing. The time it takes for the user to "hear the first sound" is decoupled from the overall sentence length. There are two playback output implementations: one is local playback via the sound card (corresponding to a full-duplex echo cancellation path); the other is resampling the 24kHz speech into 16kHz stereo and then sending it to the speaker 106 of the robot head 101 via port 4321 for playback.

[0067] Lip-sync driven, such as Figure 4As shown in lip-sync channel 304: Lip-sync inference service 111 resides on the GPU (service memory usage 1.7GB), receives 16kHz floating-point speech waveforms and emotion conditions (interface format: POST / face?emotion=happy&fps=30), and returns a joint angle sequence of 30 frames per second, including frame-by-frame angles of lip-sync related joints, including the mandibular joint. The playback side selects the current lip-sync frame for distribution using a playback clock calculated from the "number of actually output samples," ensuring frame-by-frame alignment between the lip-sync and the heard speech, unaffected by synthesis jitter. The mandibular angle range is configured according to the individual robot: in the first embodiment, the mandibular travel is 0 to 33 degrees; in the second embodiment, the mandibular travel is 0 to 28 degrees. The same lip-sync inference service 111 can serve two robot heads simultaneously. In the streaming embodiment, whenever a new synthesized audio exceeds 200 milliseconds, the lip-sync frame sequence is recalculated using the growing speech prefix, achieving synchronous progress between lip-sync and streaming synthesis. When lip-reading inference service 111 is unavailable, activate the amplitude backup path: map the mandibular opening to 10 + 18 × normalized energy (degrees) according to the speech energy envelope to ensure that "there is mouth movement for every sound".

[0068] It should be noted that the full-face expression is continuously driven by a two-dimensional emotion coordinate system of valence-arousal, allowing the intensity of the expression to smoothly transition with the content of the sentence rather than switching abruptly. The valence-arousal two-dimensional emotion coordinate system is a commonly used continuous emotion representation method in the field of affective computing: valence represents the positive or negative emotion and the degree of enjoyment (e.g., from frustration to happiness), and arousal represents the intensity of the emotion and the degree of arousal (e.g., from calm to excitement). Each emotional state corresponds to a point on this two-dimensional plane. Compared to the "jumps" of the aforementioned 13 discrete emotion labels, using this continuous coordinate system allows the intensity of the full-face expression to change smoothly with the sentence.

[0069] The specific implementation method for "conditional on continuous emotion coordinates" in step S9 can be found as follows: First, the definition and values ​​of continuous emotion coordinates: The continuous emotion coordinates are two-dimensional coordinates of pleasure and arousal, denoted as (v, a). The pleasure component v represents the positive or negative emotion and the degree of pleasure, and the arousal component a represents the strength of the emotion and the degree of arousal. The values ​​of both components range from -1 to 1. Any emotional state corresponds to a point on the two-dimensional plane. The origin (0, 0) corresponds to a neutral expression. The distance from the point to the origin represents the intensity of the emotion, and the direction of the point relative to the origin represents the type of emotion.

[0070] Secondly, the source of continuous emotion coordinates: In step S5, the discrete emotion labels 508 jointly output by the visual language big model 506 in a single inference are first mapped to a direction on the plane according to the emotion ring model—for example, happy and smiling are mapped to the quadrant where the pleasure and excitement are both positive, frustrated and aggrieved are mapped to the quadrant where both components are negative, and serious and thoughtful are mapped to the region where the pleasure is close to zero and the excitement is low—then the emotional intensity is radially scaled along this direction to obtain the target emotion coordinates of this sentence; when the upstream only provides discrete emotion labels without providing intensity, it is scaled according to the preset medium intensity of 0.7, that is, this method is backward compatible with the conditional granularity of only discrete labels. In the extended embodiment, the personality system prompt 503 can request the visual language big model to provide an emotion intensity field with a value of 0 to 1 in the jointly output JSON object. At this time, radial scaling is performed according to the value of this field, and the emotion type and emotion intensity become two independent and continuously adjustable degrees of freedom.

[0071] Thirdly, the specific injection method of "conditioning with continuous emotion coordinates": The target emotion coordinates are used as a two-dimensional condition vector. This vector is injected into the expression generation model to generate the third servo control information using Feature-wise Linear Modulation (FiLM, a conditional injection method that calculates a pair of scaling and bias coefficients from the condition vector and scales and translates the intermediate features of the network channel by channel). A multilayer perceptron takes these two-dimensional emotion coordinates as input and outputs a set of scaling and bias coefficients of the same length as the number of feature channels in the intermediate layer of the backbone network. Each channel of the intermediate layer feature is multiplied by the corresponding scaling coefficient and then added to the corresponding bias coefficient. After completing the linear transformation, the model continues forward propagation, ultimately outputting the target angles of the entire face joints, such as eyebrows, eyelids, cheeks, and corners of the mouth. The scaling and bias coefficients are initialized with identity transformations (initial value of scaling coefficient is 1, initial value of bias coefficient is 0), making the model equivalent to a baseline model without emotion conditions in the early stages of training. During training, the model gradually learns emotion modulation capabilities, avoiding the destruction of existing expression generation quality by conditional injection in the early stages of training. In a preferred embodiment, the emotion condition parameters of the lip-reading inference service 111 are expanded from discrete emotion tags to two components of continuous emotion coordinates (interface in the form of POST / face?valence=0.7&arousal=0.5&fps=30). When only discrete emotion tags are passed in, the server converts them into coordinates according to the aforementioned emotion ring model, and the interface is backward compatible.

[0072] Fourth, the realization of continuous changes in facial expressions: Since the emotion coordinates are continuous quantities rather than discrete categories, there is no hard switching of expressions between adjacent sentences. Let the emotion coordinates of the previous sentence be (v1, a1) and the emotion coordinates of the current sentence be (v2, a2). In the transition interval between sentences, the emotion coordinates of the current frame are obtained by linear interpolation between the two points according to the frame number. The above conditions are injected frame by frame along a transition trajectory on the emotion plane, and the expression generation model outputs the third servo control information frame by frame. Compared with the attitude jump caused by directly looking up the 13 discrete emotion tags, this method makes the types and strengths of facial expressions evolve smoothly with the content of the sentence. Under the condition that the text and audio of the same sentence remain unchanged, the expression can be continuously adjusted by only changing the target emotion coordinates.

[0073] Fifth, the synthesis and distribution of the third servo control information: The third servo control information starts with the first servo control information distributed in step S6 based on discrete emotion tags. On this basis, the expression generation model based on continuous emotion coordinates modulates the angles of the upper facial joints such as eyebrows, eyelids, and cheeks frame by frame. This information is then combined with the mouth shape-related joint angles contained in the second servo control information obtained in step S8 according to joint affiliation to form a complete dictionary of whole-face joint angles. This dictionary is then distributed to the control box 108 via the 8765 port and executed by the facial servo group 107 at a pace of 20 milliseconds per frame. The joints under the control of the two servo control information do not overlap and the merging does not cause any conflict.

[0074] Sixth, technical qualitative aspects: The robot voice-expression linkage method proposed in this invention maps the discrete emotion tags that are semantically related to the single inference output of the large model in step S5 to continuous emotion coordinates in real time. Then, it uses these coordinates as conditions to perform frame-by-frame continuous modulation and cross-sentence interpolation transition on the whole face expression, so that the types and strengths of expressions are both semantically related to the dialogue and evolve continuously with the sentences. This eliminates the abruptness inherent in the switching of discrete emotion categories while retaining the temporal advantage of "emotion first".

[0075] (ix) Step S8: Interruption detection and rotation In the full-duplex embodiment, the interruption detector 407 continuously receives the echo-cancelled audio signal 405 during playback and calculates the speech probability for each of the 512 sample frames: a speech frame is recorded as a speech frame if the speech probability is not less than a threshold (configurable from 0.6 to 0.8, 0.6 for quiet environments and 0.8 for loud speaker volumes); the cumulative silence between frames does not exceed 100 milliseconds (tolerating pauses between words); an interruption is determined when the cumulative continuous speech reaches the trigger duration (configurable from 250 milliseconds to 400 milliseconds). After an interruption is determined, the playback controller 408 immediately clears the output buffer 402, stops speech transmission, resets lip movements and facial expressions, and enters the listening state. The measured reaction time from the user speaking to the end of playback is less than 1 second. Since the input for interruption detection has already undergone echo cancellation, the robot's own voice will not trigger a false interruption.

[0076] In the half-duplex implementation (the downgrade path when a full-duplex sound card is not available), after playback ends, the system waits for the duration of the audio block plus a 0.4-second guard time, and then continuously monitors the root mean square amplitude of the ambient audio. If the amplitude drops below the set threshold by 0.3 seconds (the maximum wait time is 1.5 seconds), the system switches to listening mode. This can also achieve more than 99% self-listening exclusion, at the cost of not being able to interrupt playback.

[0077] (x) Variations of Example 1 In an optional embodiment, when the binocular camera 105 is unavailable, the multimodal message degenerates into a plain text message and is sent to the same model's text interface, while the remaining steps of the method remain unchanged. In an optional embodiment, the discrete emotion tag set can be expanded or pruned, and entries in the emotion-pose mapping table can be added or removed accordingly without affecting the joint output mechanism. In one embodiment, the base of the visual language large model can be replaced with other hybrid experts or dense architectures, as long as they have a joint output format after further training.

[0078] Example 2: Low-latency implementation (xi) Speculative execution: Hiding identification and reasoning within the tailing and silent waiting period In the dialogue loop of Example 1, the endpoint detection in step S2 must wait until the endpoint threshold (1.2 seconds in stable mode) is reached before confirming that the user has finished speaking. During this trailing silence waiting period, the user has stopped speaking, all audio has been cached in the host computer 102, and speech recognition and large model inference are fully ready to start. However, the serial execution causes the computing resources to idle. Example 2 introduces a speculative execution mechanism based on Example 1. It applies the idea of ​​"branch prediction / speculative execution" in processor architecture (pre-calculating along the most likely path before the branch condition is determined, and directly using the correct guess or discarding and rolling back if the guess is wrong) to the turn processing of human-computer dialogue for the first time. "Has the user finished speaking?" corresponds to the pending branch condition. The trailing silence waiting period of up to 1 second before endpoint confirmation is the "branch undetermined" window. This invention initiates speech recognition and large-scale model inference in advance within the window of uncertainty, following the most likely branch of "the user has finished speaking." If the guess is correct, the entire hidden calculation is performed, gaining a period of waiting time. If the guess is incorrect, the subsequent speculative generation mechanism invalidates the calculation and rolls back for recalculation. In the worst case, it is no worse than the original serial link. In contrast, existing technologies such as streaming recognition, streaming synthesis, and pipelined parallel technology all use endpoint confirmation as the starting point for calculation, never utilizing the waiting period before endpoint confirmation, and lacking the guarantee of invalidation and rollback when "pre-calculation is overturned by the user's continued speech."

[0079] like Figure 7As shown in the upper part, the upper part is a conventional serial link: after the user voice segment 601 ends, the system still has to wait for the following silent waiting segment 602 (1.2 seconds) before confirming the endpoint. Then, the speech recognition segment 603 (0.15 seconds to 0.4 seconds), the large model inference segment 604 (hot state 3 seconds to 5 seconds, of which the first clause is available earlier), and the speech synthesis first block segment 605 (0.6 seconds to 1.7 seconds) are executed in sequence, with each segment strictly connected end to end. The following describes the speculative parallel chain of this invention: After each frame update, the endpoint detector 406 outputs the current trailing silence duration and the accumulated speech duration. When the silence reaches the speculative trigger moment 606 (tailing silence 0.4 seconds and the accumulated speech duration not less than the minimum speech duration of 0.4 seconds), the speech recognition segment 603 and the large model inference segment 604 are initiated in advance by the background thread using the cached echo-cancelled audio signal 405, running in parallel with the remaining 0.8 seconds of silence waiting. When the endpoint confirmation moment 607 arrives, the recognition result and even the reply text are usually ready, and the synthesis playback can proceed directly. The difference between the two timelines is the delay saving amount 608, which is measured to be 0.3 to 1.0 seconds. The speculative threshold of 0.4 seconds is only one-third of the endpoint threshold of 1.2 seconds, and the remaining waiting window can be fully utilized in most real-world speech ending scenarios. Since the input for speculative input is the echo-cancelled audio signal 405, the sound played by the robot itself will not mistakenly trigger speculative input.

[0080] (xii) Speculative execution state machine and speculative intergenerational mechanism like Figure 7As shown in the lower part, the speculative execution operates according to a state machine: In the idle state 609, at the start of each round of dialogue, the speculative generation counter 615 is incremented once and the speculative context of the previous round is cleared; after entering the audio reception state 610, each audio block updates the endpoint detector 406 and reads the current trailing silence duration silence_sec and the accumulated speech duration speech_sec; when speech_sec≥0.4 seconds and silence_sec≥0.4 seconds and the current generation has not yet emitted a speculative signal, it transitions to the speculative execution state 611: records the emitting generation fired_gen=the current generation gen, and hands over the cached audio snapshot to the background speculative thread. After the speculative thread completes recognition and inference, it checks its own generation within the lock to ensure it matches the current generation before writing the speculative result, transitioning to the result-ready state 612. If the reception status detects that silence_sec drops below 0.4 seconds (indicating the user has resumed speaking), it immediately transitions to the invalid state 613: the speculative generation counter 615 increments, causing the executing speculative thread to fail naturally during its generation verification, the result is discarded, and the state machine returns to the reception state 610, waiting for the next silence threshold to be met before retransmitting. Upon endpoint confirmation, it enters the submission state 614, performing three-way arbitration: if the result is ready and generation matches → adopted, skipping speech recognition and large model inference after the endpoint; if the speculative thread is still alive → waits for 1.5 seconds before judging; if still not ready or generation does not match → discarded and backtracked to the normal path, the worst case being equivalent to the original serial link.

[0081] The subsequent process of the "adopted" route in the above three arbitrations is as follows: the main thread first places the speculative transcribed text and speculative reply into the one-time service slots mentioned in the following submission bypass, and then continues to execute according to steps S6 to S9 of Embodiment 1 as usual—driving the whole face expression according to the discrete emotion tag, streaming the reply text and playing it sentence by sentence, and driving the lip shape and whole face expression frame by frame; that is, "skipping" here only refers to skipping the two calculations of speech recognition and large model inference after endpoint confirmation. The subsequent expression, synthesis, lip shape and playback links are all executed as usual, and the behavior observed by the user side is completely consistent with the rounds in which no speculation occurred.

[0082] Regarding the three parameters—the speculation threshold of 0.4 seconds, the minimum speech duration of 0.4 seconds, and the waiting limit of 1.5 seconds—the following explanations should be provided: These are all preferred values ​​obtained through actual machine tuning, not the only optimal solution. They are all available as configuration items and can be adjusted according to hardware performance, network conditions, and ambient noise levels. Small variations around their respective values ​​do not alter the implementation effect or beneficial effects of this method. The selection criteria are as follows: the speculation threshold must be significantly less than the endpoint threshold to leave a usable parallel window. A stable endpoint threshold of 1.2 seconds, reduced to 0.4 seconds, is exactly one-third of this threshold, leaving 0.8 seconds before endpoint confirmation for parallel processing with speech recognition and large-scale model inference. If this value is too small, frequent misinterpretations of breaths during sentences will be mistaken for the end of speech, leading to a large number of invalid speculations. If it is too large, the remaining window will be insufficient to hide computations, and the benefit will approach zero. When the endpoint threshold is changed to a low-latency setting of 0.7 seconds, the speculation threshold should be proportionally reduced to maintain the above proportional relationship. A minimum speech duration of 0.4 seconds is used to filter out excessively short sounds such as coughs, single-word interjections, and environmental transients, avoiding speculation for meaningless audio. Its value only needs to cover the shortest duration of a valid syllable. The selection of the waiting upper limit of 1.5 seconds is based on the measured time (1.0 to 2.0 seconds) of the large model inference to the first usable clause under hot conditions: if the value is too small, speculation that is close to completion will be wasted and rolled back to full recalculation; if the value is too large, the time limit waiting itself becomes a new source of latency. Since exceeding the upper limit will backtrack to the normal path, this parameter only affects the size of the gain and not the correctness. In the worst case, it is still equivalent to the original serial link. In noisy environments, the speculation threshold can be adaptively increased to 0.5 to 0.6 seconds according to the variation of Example 2 to reduce invalid transmissions, while the rest of the logic remains unchanged.

[0083] The correctness of the speculative generational mechanism hinges on two key aspects: First, lock-free submission—the speculative result carries its own generational marker, and submission only requires a single comparison of "mark == current generation," eliminating the need for locking on the audio callback path. Second, zero historical pollution—when the speculative thread starts, a thread-local suppression flag (suppress) is set. The dialogue system's recording functions (writing multi-turn dialogue history 504, marking as greeted) operate null on threads with this flag set. Therefore, speculative inference calls the exact same complete inference function as the formal path (including personality system prompt 503, 8-turn history, and face context) without any side effects. Only when the submission state adopts the result is the main thread (without the suppression flag set) writing "user text - reply text" to the shared multi-turn dialogue history in one go. Thus, discarded speculations leave no trace, and adopted speculations are recorded exactly once, eliminating cross-thread history contention.

[0084] The specific implementation of the submission bypass is as follows: A bypass wrapper is installed on both the speech recognition entry function and the large model inference entry function, and a one-time recognition service slot and a one-time response service slot are set up. When the submission status uses a speculative result, the speculative transcribed text is first placed into the recognition service slot, and the speculative response is placed into the response service slot. Then, the main loop calls the recognition and inference entry points as usual according to the serial link. When the bypass wrapper finds that the corresponding service slot is not empty, it directly returns the result in the slot and clears the slot; the original function body is no longer executed. Since each service slot is consumed only once, the speculative result will not remain across rounds. In rounds where no speculation occurs, both slots are empty, the bypass wrapper transparently passes directly to the original function, and the behavior of the serial link remains unchanged byte by byte. The entire speculative mechanism is controlled by a master switch configuration item. When turned off, no wrappers or hooks are installed, and the system completely degenerates into the original serial link, facilitating comparison verification and troubleshooting.

[0085] In a preferred embodiment, the emotion-preceding facial expression accompanying the speculation allows the robot's face to be driven (the face reacts first) during the speculation phase, and the face to be reset to a neutral posture when the speculation is invalidated. This behavior is consistent with the interaction design of step S6, "emotion precedes speech".

[0086] (xiii) Streaming Implementation of Speculative Speech Recognition For speculative speech recognition, the preferred method is to call the / asr_stream streaming interface on port 8770. This returns partial and final transcriptions incrementally using a JSON line protocol. The partial transcription is used for log observation, and the final transcription serves as the speculative transcription text. If the streaming interface fails on the first call, the streaming binding is automatically removed within the same session, and the system is downgraded to the full / asr interface to avoid retrying failed interfaces in each round. Speculative speech recognition terminates silently when the recognition result is empty (the user only makes meaningless sounds), without generating any speculative results. Large model inference cascading can be independently switched on and off: when off, only speculative speech recognition is performed (inference still occurs after the endpoint), suitable for scenarios where inference service load is sensitive; when on, recent image frames from the binocular camera 105 are prioritized for use with the multimodal interface, and if no image frames are available, a plain text interface is used. Even if large model inference fails, a speculative result containing only the speculative transcription text is still submitted; only inference needs to be performed after the endpoint, saving recognition time.

[0087] (xiv) Padding opening and first hard cut There is still a synthesis delay between endpoint confirmation and the playback of the first voice block. In Implementation Example 2, a gap is covered by a pre-emptive speech interruption: after endpoint confirmation, the system immediately plays the pre-emptive speech audio from the cache and starts the synthesis of the first block in parallel; after the synthesis of the first block is completed, the pre-emptive speech ends and plays, and subsequent clauses are synthesized and connected simultaneously. Both the pre-emptive speech interruption and the hard cut of the first block are non-publicly known perceptual delay compression mechanisms proposed in this invention. The two work together to compress the gap between endpoint confirmation and the user hearing the first sound to near zero: the pre-emptive speech interruption fills the gap immediately with a pre-cached spoken word with zero synthesis overhead, while the hard cut of the first block compresses the synthesis time of the actual response to the first audible block from "related to the length of the whole sentence" (more than ten words, 1 to 1.7 seconds) to a fixed 0.6 seconds (5 words). The two are seamlessly connected one after the other. This differs from the existing approach of stream segmentation based solely on punctuation marks—the latter often has a first clause of more than ten characters, the first block still takes more than one second to synthesize, the silent window before the robot speaks still exists, and users are prone to repeatedly asking questions due to the robot's "lack of response," leading to chaotic dialogue.

[0088] The specific implementation of the filler phrase mechanism is as follows: The word pool contains four default spoken phrases: "um," "that," "let me think," and "let me see," which can be expanded through configuration options. At system startup, a background thread calls the streaming speech synthesis service 110 word by word and caches the 24kHz PCM16 audio as a content hash on disk. Subsequent startups directly read from the cache, resulting in zero synthesis overhead during runtime playback. After each endpoint confirmation, a 0.4 probability roll is used to determine whether to add a filler phrase in that round—avoiding mandatory filler phrases in every round to prevent a mechanical feel. When deciding on a filler phrase, a phrase is randomly selected from candidates in the word pool with a tested duration between 0.2 and 0.85 seconds. This 0.2 to 0.85 second range acts as a duration guard: if the synthesis engine occasionally renders a word longer than 0.85 seconds, that word is excluded, ensuring that the filler phrase ends no later than the first block's readiness time, and that the filler phrase audio does not overlap with or gap between the first speech block and the response.

[0089] Two terms and one value need further explanation: Firstly, the aforementioned filler audio refers to a spoken word segment played before the first speech block of the response is synthesized, used to fill the silence gap. Its content does not carry response semantics (i.e. it is not an answer to the user's question) and is only used to enable the robot to speak immediately after the endpoint confirmation. Therefore, it can be pre-synthesized and cached when the system starts up, with zero synthesis overhead during playback.

[0090] Secondly, the "overlap of the background audio and the first voice block of the reply" refers to the phenomenon where the background audio has not finished playing while the first voice block of the reply has been synthesized and started playing, and the two audio segments are played simultaneously on the same speaker, overlapping each other and making it difficult for the user to hear clearly; the duration guard is designed to prevent this phenomenon - limiting the candidate to a duration range of 0.2 seconds to 0.85 seconds, so that the playback time of the background audio ends no later than the ready time of the first block, without overlapping or leaving gaps.

[0091] Thirdly, regarding the playback probability of 0.4: This value is also configurable, not a critical value or the only optimal solution—a probability of 1 means that filler words are used in every round, and the repeated use of the same phrases creates a mechanical feeling; a probability of 0 means that filler words are not used, and the silent gaps after endpoint confirmation are exposed again; a probability of 0.4 means that filler words fill the gaps in about two-fifths of the rounds, and the rest of the rounds are played directly, which is an engineering compromise between "most gaps being covered" and "poems not being repeated in every round". This method can implement any value of this probability between 0 and 1. Small changes around 0.4 have no substantial impact on user perception and measured end-to-end latency. Implementers can adjust it themselves according to the robot's personality settings and word pool size.

[0092] The specific implementation of the first block hard cut is as follows: In step S7, the streaming speech synthesis divides the response text into a sequence of clauses according to punctuation. When the length of the first clause exceeds the maximum number of characters (5), punctuation is prioritized for segmentation within the first 5 characters. If there is no punctuation within the first 5 characters, a forced segmentation (hard cut) is performed at the 5th character. The actual synthesis time is approximately proportional to the number of characters (on the order of 0.1 seconds per character), and the synthesis time for the first 5-character block is stable at 0.6 seconds. Since subsequent clauses are synthesized in parallel during the playback of the first block, hard cutting does not cause subsequent interruptions. In the control embodiment without hard cutting, the default upper limit for the first clause is 18 characters, and segmentation is only performed at punctuation marks. The synthesis time for the first block is 1 to 1.7 seconds.

[0093] (xv) Extension of model insulation Building upon the heat preservation of the visual language model in step S5, Example 2 extends the heat preservation to the speech recognition model: The host computer 102 periodically (preferably every 30 to 60 seconds) sends lightweight heat preservation requests to the GPU server 103 via a background daemon thread. These requests send word-level requests to the visual language model and 0.5-second silence audio requests to the speech recognition model (each processing time is less than 50 milliseconds), ensuring that the weights of both models remain resident in memory and the inference engine remains in a hot state. Actual testing showed that without heat preservation, the initial latency after idle period exceeded 13 seconds; with heat preservation enabled, the initial latency was consistent with the hot state, and the additional memory and computing power overhead was negligible. Heat preservation requests share the same interface and queue as normal dialogue requests. When a real request is in transit, the heat preservation cycle is automatically skipped to avoid competing for computing power with user requests.

[0094] (xvi) End-to-end delay decomposition and configuration tiering The end-to-end "user finishes speaking - robot speaks" latency can be broken down into five stages: trailing silence wait (1200 ms for stable mode / 700 ms for low latency mode), speech recognition (150 ms to 400 ms), large model inference (1000 ms to 2000 ms for the first clause), speech synthesis first block (600 ms after hard cut), and playback output (sound card buffering for tens of ms). In Example 2, the mechanisms function as follows: speculative execution hides recognition and inference within the trailing silence wait; the pre-emptive speech opening advances the user's perception of the speaking moment to immediately after endpoint confirmation; the first block hard cut compresses and synthesizes the first block; insulation eliminates the cold start component of inference; and the low latency mode directly shortens the trailing silence wait.

[0095] Typical configurations include three levels: Stable level (endpoint threshold 1.2 seconds, no speculation, no hard cut), with a measured end-to-end time of 4.5 seconds; Low latency level 1 (endpoint threshold 0.7 seconds, hard cut of 5 words, padding probability 0.4, heat preservation enabled), with a measured time of 2.0 to 2.5 seconds; Low latency level 2 (speculative execution and streaming recognition enabled on top of level 1), with a measured time of 1.5 to 2.0 seconds. Each level is switched via configuration items (speculation switch, speculation threshold, minimum voice duration, waiting limit, padding switch, word pool, playback probability, first block word count, hard cut switch, endpoint threshold). The default is the conservative level, which is optimized on-site according to hardware and noise conditions. The system also includes two types of measurement tools: a latency benchmark tool that drives the entire link with a fixed WAV file, and a speculation gain probe that counts the number of speculative launches, submissions, and rejections and the net gain, used for regression verification and threshold tuning.

[0096] (XVII) Single-round timing example The collaboration of each mechanism is illustrated by taking a round of actual measured conversation under the low-latency second gear as an example. The user pauses 2.6 seconds after the question starts, and the time of the last speech frame is recorded as t=0: at t=0.40 seconds, the trailing silence reaches the speculative threshold, the accumulated speech duration of 2.6 seconds meets the minimum speech duration condition, speculation is transmitted with generation 3, and the background thread takes away the cached 3.0 seconds of audio; at t=0.62 seconds, the streaming recognition returns the first partial transcription; at t=0.78 seconds, the final transcription is ready, the cascade starts large model inference, and the emotion-first expression drives the robot's face at the same time; at t=1.20 seconds, the trailing silence reaches the endpoint threshold, the endpoint is confirmed, at this time the speculative inference is still being executed, and arbitration is submitted to enter a time-limited wait; at t=1.52 seconds, the speculative reply is ready and the generation verification is consistent, the submission is adopted, the main thread writes to the shared multi-turn conversation history, and the speech recognition and large model inference after the endpoint are skipped entirely; at t=1.53 seconds, the draw hits the filler phrase, the word "Hmm" (cached duration 0.42 seconds) is randomly selected from the word pool and played immediately, and at the same time the 5-character clause of the first hard-cut block is sent to streaming synthesis; at t=1.95 seconds, the filler phrase finishes playing; at t=2.13 seconds, the first block synthesis is completed and played immediately after, and subsequent clauses are synthesized and continued in sequence. The utterance time perceived by the user in this round is t=1.53 seconds, and the real reply utterance time is t=2.13 seconds; compared with the stable gear serial link on the same hardware (endpoint waiting 1.2 seconds, speech recognition 0.3 seconds, inference to the availability of the first clause 1.8 seconds, first block synthesis 1.5 seconds accumulated in sequence), the utterance time is t=4.8 seconds, and the present invention shortens it by more than 2.6 seconds. If the user resumes speaking at t=0.9 seconds in this example, the generation increments to 4, the speculative result of generation 3 is discarded during submission verification, the shared multi-turn conversation history and facial state are not affected at all, and the system re-speculates with generation 4 when the next silence meets the standard.

[0097] (18) Variation of the second embodiment In one embodiment, the speculation only covers speech recognition and does not cascade large model inference, which is suitable for scenarios where the inference service is shared by multiple robots. In one embodiment, the speculation threshold is adaptively increased with the ambient noise level (increased to 0.5 seconds to 0.6 seconds when the noise is loud) to reduce invalid transmissions. In one embodiment, the filler phrase word pool is customized according to the robot's personality (different personas use different colloquialisms), and repeated word selection is prohibited for the same round. In one embodiment, the speculative execution is used in combination with the interruption detection of step S8, and the speculative execution also takes effect in the new round of audio collection after the interruption is established.

[0098] Third Embodiment: Dual-Robot Dialogue Implementation Mode In the following, the extended operation mode of the present invention in a multi-robot scenario is illustrated by taking two robot heads as an example, and this embodiment can be extended to scenarios where more robot heads speak in turn.

[0099] (19) System Topology and Central Coordinator like Figure 7 As shown in the upper part, the system of this embodiment includes a first robot head 701 (black head) and a second robot head 702 (white head), which are driven by a first agent 703 and a second agent 704 respectively (the agent is the host computer process that runs the dialogue master editing program of Embodiment 1); a central coordinator 706 (HTTP service listening on port 8772) and a large model inference service 707 (listening on port 8770, single card deployment, i.e., the visual language large model service that provides speech recognition interface and visual language large model inference in Embodiment 1; there is also a streaming speech synthesis service on port 8762 and a lip-sync inference service on port 8771) are deployed on the GPU server 705; the operator console 708 performs manual control by accessing the central coordinator 706 via HTTP. The Central Coordinator 706 is a multi-threaded HTTP service implemented purely with the Python standard library. It does not rely on GPUs or robot hardware and can be deployed on the same machine as the large model inference service 707. The service's request queue depth is 128 to accommodate sudden short-connection polling. All state changes are completed within a single mutex lock, and the reading and writing of state snapshots are atomic operations. It provides the following interfaces: GET / state (state snapshot), POST / claim (speaking rights request), POST / speaking (response pre-publishing), POST / done (round completion), POST / heartbeat (heartbeat), POST / control (control panel management), GET / transcript (incremental retrieval of dialogue transcription), and GET / health (health check).

[0100] (20) Speaking Rights State Machine like Figure 7The lower half shows the running states, including idle state 709, running state 710, paused state 711, and stopped state 712. These states are driven by the start, pause, resume, and stop actions of the control panel 708 via the / control interface. The speaking rights phase alternates between the handover phase 713 and the occupying phase 714: the handover phase 713 indicates that the party holding the speaking rights can apply between rounds; the occupying phase 714 indicates that the party holding the speaking rights is in the middle of a round (reasoning or playback). The coordination generation counter 715 increments once for each authority state transition (successful application, round completion, forced transfer, running state change, topic change). All fields of the state machine—running state, speaking rights phase, speaking rights holder, round number, coordination generation counter value, topic, previous reply text, and reply to be published—together constitute a state snapshot, which is atomically read out by the / state interface. When a dialogue is initiated, the control panel 708 sets the topic, the starting speaker (default first robot head 701) and the maximum number of rounds (default 20) with the start action. The central coordinator 706 places the speaking right on the starting speaker and sets the stage to handover stage 713.

[0101] (XXI) Comparison and exchange of speaking rights requests and round submission A typical round's interaction sequence is as follows: The first agent 703 polls / state 8 times per second; upon finding "running, handover phase, speaking rights belong to its own side," it initiates a speaking rights request carrying the coordinated generation count value (generation) and round number (turn) from the snapshot (using a POST call to the / claim interface, with the request body carrying three fields: robot head identifier, coordinated generation count value, and round number (turn)). The central coordinator 706 performs five checks within the mutex lock: running status is running, phase is handover, the requester equals the speaking rights holder, generation equals the current generation, and turn equals the current round number; if all are true, the phase is set to occupied, the round start time is recorded, the occupied count is incremented by 1, the generation is incremented, and a successful request response carrying the new generation is returned; if any one is false, an expired rejection response carrying the latest state snapshot is returned (HTTP status code 409), and the requester returns to polling based on the snapshot. In a competitive scenario, the application issued by the second agent based on an expired snapshot will inevitably receive an expired rejection response due to intergenerational inconsistency, and the authoritative state will not be affected in any way—this is the structural guarantee of the comparison and exchange semantics for "old observations cannot rewrite new states".

[0102] It should be noted that the technical solution adopted by the robot voice-expression linkage method proposed in this invention is not limited to the transfer of CAS primitives, but rather a complete set of speaking rights protocols built around multi-robot continuous dialogue: binding speaking rights with the exclusive right to reason in the large model, the right to use the speaker, and the right to listen to the microphone (the occupying party exclusively uses the reasoning service 707 and the speaker, and the silent party pauses microphone listening to eliminate acoustic interference). The claim (application) / speaking (pre-release) / done (round completion) / heartbeat interfaces are all driven by the authority state transfer with triple verification of "attribution + generation + round number", and superimposed with two-level fault-tolerant transfer of 120 seconds round timeout and 8 seconds heartbeat (see XXIII) as well as repeat detection of the count of catchphrases not less than 3, the similarity of character tuples Jaccard, and the cycle rotation of director's prompts (see XXIV) - these constraints for dialogue rounds, acoustic interference suppression and repeat elimination are not included in the general distributed primitives such as CAS, mutex locks, leases, and heartbeats.

[0103] After the application is successful, the first agent 703 exclusively calls the large model inference service 707 and generates the response text and discrete sentiment tags according to steps S4 to S5 of Example 1. Before the playback starts, the calculated response text can be pre-published to the pending publication field of the central coordinator 706 via response pre-publishing (POST / speaking). This interface only performs attribution and generation verification, does not flip speaking rights or increment generation, and its function is to allow the counterpart agent to obtain the content in advance during its own playback and pre-calculate the next round of responses, further compressing the gap between rounds. After playback and lip-syncing are completed, the first agent 703 initiates a round submission (calling the / done interface via POST, with the request body carrying six fields: robot head identifier (head), coordination generation count value (generation), round number (turn), reply text (reply), emotion tag (emotion), and the heard text (heard_text)). The central coordinator 706 verifies the occupied stage, owner, generation, and round number, then writes the reply text to the transcription, updates the previous reply text, clears the pending publication field, decrements the occupied count by 1, increments the round number by 1, and executes a speech right flip: the stage returns to the handover stage 713, the owner is changed to the second robot head 702, and the coordination generation count value is incremented again. When the round number reaches the maximum number of rounds, the running state automatically transitions to a stopped state 712. The central coordinator 706 maintains the occupied count and ensures it does not exceed 1, ensuring that at any given time, at most one agent calls the large model inference service 707, and the visual language large model deployed on a single card experiences no concurrent contention in a dual-head scenario.

[0104] (22) Acoustic isolation and facial expression linkage within rounds During the occupation phase, the speaking agent fully executes the "emotion-first—streaming synthesis—frame-by-frame lip-sync" chain of steps S6 to S7 in Implementation Example 1: Before speech synthesis begins, discrete emotion tags query the emotion-gesture mapping table of 13 emotions to drive the whole-face expression; the reply text is played sentence by sentence through streaming speech synthesis; the lip-sync inference service outputs a joint angle sequence including the jaw joint at 30 frames per second based on the synthesized speech waveform, aligned with the frame-by-frame playback. The silent agent immediately pauses the microphone listening thread and keeps the speaker silent after polling for "speaking right not belonging to itself," maintaining only the heartbeat and polling; it resumes listening only after successfully applying for speaking right, thus preventing the cascading error of one party treating the other party's speaker sound as input.

[0105] The acquisition of dialogue content supports two modes (switched via set_content_mode on console 708): In acoustic mode, the speaker recognizes the words just spoken by the other party through the microphone; in relay mode, the previous reply text forwarded by the central coordinator 706 is directly retrieved. The former demonstrates a true "listening and speaking" closed loop, while the latter ensures lossless content even in noisy environments. The default is a hybrid mode of acoustic recognition first, and reverting to relay if it fails. The specific implementation of acoustic mode is as follows: After the other party's turn ends, the speaker opens a directional audio reception window, uses the neural network endpoint detection in step S2 to capture the other party's speech, and calls speech recognition. If the recognition result is empty or has too few words, it automatically reverts to relay text to ensure that the content chain is not broken.

[0106] (23) Fault Tolerance: Round Timeout and Heartbeat Each agent reports its working status (thinking, playing, standby) to / heartbeat at a fixed interval (preferably 2 to 4 seconds). The watchdog thread of the central coordinator 706 checks once per second: first, if the duration of the occupancy phase exceeds the round timeout threshold of 120 seconds (covering the worst-case inference plus playback time), the round is considered dead; second, if the last heartbeat of the speaker is more than 8 seconds ago, the agent is considered dead. Upon any successful transfer, a forced transfer is executed: the occupancy count is decremented by 1, the recovery round count is incremented by 1, the current round is completed with an empty reply (this round is marked as a recovery round in the transcription and does not overwrite the previous valid reply text), the speaker rights are flipped to the other side, and the coordinating generation count is incremented—even if the original speaker subsequently "revives," any submissions carrying the old generation will be rejected, without disrupting the transferred state. The watchdog also maintains the invariant "running state: there must be a speaker at runtime." If, due to an anomaly, there is no speaker, the ownership is immediately reset to the opposite side of the previous speaker (or the starting speaker). In actual testing, after a single process is forcibly killed, the dialogue will automatically continue within 8 seconds (heartbeat-based death) to 120 seconds (turn timeout if there is no heartbeat record).

[0107] (24) Two-level repeat detection and prompt injection Whenever the speaking party submits a reply text via / speaking or / done, the central coordinator 706 updates the sliding window (the latest 5 transcribed replies plus the current reply), and sequentially performs three levels of detection and injection.

[0108] The first level is pet phrase pattern detection: take the latest 4 replies, count the number of rhetorical question hits (hitting one of the marker words "have you tried", "you try", "have you tried it", "have you ever tried", "you try to", or the reply ends with "?") and the number of self-deprecation hits (hitting one of the marker words "I have to learn first", "then I have to", "I tried", "directly fail", "directly crash"); if any count is not less than 3, it is determined that the corresponding pet phrase pattern is entered. This detection targets structural repetition where "the surface wording changes every round, but the sentence pattern skeleton and role division are fixed" — for example, one party uses rhetorical questions to respond every round, and the other party uses a self-deprecation template to respond every round — this type of repetition is exactly the blind spot of literal similarity detection. After the determination is established, a targeted correction prompt is selected from the preset correction prompt table according to the identity of the next speaking party: inject "Do not use rhetorical questions to respond anymore, instead directly state your opinion, describe specific experiences, or first agree then supplement" for the party who is used to using rhetorical questions; inject "Do not use self-deprecation templates to respond anymore, instead make a positive statement, put forward different opinions, or steer the topic to a new direction" for the party who is used to using self-deprecation.

[0109] The second level is character bigram similarity calculation: remove blanks from the reply text and take the set of all bigrams of two adjacent characters. When the Jaccard similarity (a commonly used set similarity metric in the field of information retrieval, equal to the size of the intersection of the two sets divided by the size of the union, with a value ranging from 0 to 1, the larger the value the more similar) between the current reply and any historical reply in the sliding window is not less than the threshold of 0.5, it is determined as literal repetition, and a general correction prompt is injected: "Do not repeat the meaning you have just said, talk about brand new content or change the direction of the topic". The character bigram does not require word segmentation for Chinese text, the calculation is a set operation, and a single comparison is completed on the microsecond scale.

[0110] The third level is the director's prompt rotation: If the above two levels of checks are not met, and the next round's sequence number is an integer multiple of the director's cycle 6, then one prompt is selected from the director's prompt table (5 categories: discussing unmentioned aspects from a new perspective, actively raising new questions or giving specific examples, proposing different viewpoints to create back-and-forth, associating with relevant specific details, and naturally shifting to a relevant new direction) according to the rotation pointer; when there is a dialogue topic, the rotation pointer, on an odd number of cycles, changes to injecting the anchoring prompt "the topic has diverged too far, naturally returning to the topic." The three levels of prompts (targeted correction, general correction, director / anchoring) are selected according to priority and written into the director's prompt field of the status snapshot; the agent of the next speaker reads and appends this field when constructing the personality system prompt in step S4, thus realizing prompt injection. Prompt injection only affects the generation conditions of the next round and does not modify the multi-round dialogue history itself.

[0111] (25) Variations of Example 3 In one embodiment, the number of robot heads is expanded to three or more, and the speaking right flipping strategy is changed from "giving to the other side" to rotating in a preset order or having the visual language big model nominate the next speaker, while the comparison and exchange verification logic remains unchanged. In one embodiment, the central coordinator and the big model inference service are deployed in separate units, and each agent accesses across network segments, with the interface and timing remaining unchanged. In one embodiment, the console dynamically modifies the topic, maximum number of rounds, or content mode via / control during the dialogue, and modifies the intergenerational coordination count value as an increment of the authority state transition, automatically invalidating expired requests that are in flight. In one embodiment, the round timeout threshold is dynamically adjusted according to the reply length (long replies are relaxed to 180 seconds), and the heartbeat cycle and timeout are tuned according to network quality. In one embodiment, reply pre-publishing allows the silent party to pre-calculate its next round reply using the pre-published reply text as input during the speaker's playback, compressing the gap between rounds from "inference time" to "playback tail difference"; the pre-calculated result also carries the intergenerational coordination count value, which can only be used after the intergenerational verification passes after the speaking right flip, and the pre-calculated result is automatically discarded due to intergenerational mismatch when the topic is modified by the console before the flip. In one embodiment, each agent simultaneously enables the speculative execution and padding of Embodiment 2, which take effect independently within their respective radio rounds.

[0112] (26) Summary of measured results The actual test data on the deployment of dual bionic robot heads and a single GPU server are as follows.

[0113] Example 1 (Basic Dialogue Loop): Self-hearing cancellation rate greater than 99%; interruption response time less than 1 second; facial expression advance time difference 0.5 to 1.5 seconds; speech recognition time 0.15 to 0.4 seconds; large model hot inference 3 to 5 seconds; streaming synthesis reduces the end-to-end "speaking-speaking" delay from 4.5 seconds in the whole sentence synthesis scheme to less than 2.5 seconds; the heat preservation mechanism reduces the first-round delay from more than 13 seconds to the same as the hot state.

[0114] Example 2 (Low Latency): Speculation execution saves 0.3 to 1.0 seconds per round, and discarded speculations have zero pollution to the shared multi-round dialogue history; the padding of the call reduces the user-perceived "no response window" to near zero; the first block hard cut stabilizes the first block synthesis time at 0.6 seconds; the end-to-end latency of stable, low latency level 1, and low latency level 2 are 4.5 seconds, 2.0 to 2.5 seconds, and 1.5 to 2.0 seconds, respectively.

[0115] Example 3 (Dual Robot Dialogue): The occupancy count never exceeds 1 throughout the entire process; there is no concurrent contention for large model inference; the single-round inference latency is consistent with the single-head scenario; there are no instances of mutual voice misidentification in dual-head simultaneous dialogue; after manually killing one side of the agent process, the dialogue automatically resumes after an 8-second heartbeat timeout (120 seconds for round timeout when there is no heartbeat record); the 20-round long dialogue did not experience any more verbal loops under the effect of two-level repeat detection and director prompts; the cost of retrying expired requests is 125 milliseconds per polling cycle; the processing overhead of the central coordinator itself is negligible in the entire round (speaking right application and verification, round submission and speaking right flipping are all less than 1 millisecond), and the coordination layer does not constitute a new latency bottleneck.

[0116] Furthermore, embodiments of the present invention also provide a robot voice-expression linkage system, comprising: The echo cancellation module is used to take the speech synthesis block that has just been written into the speaker output buffer as the far-end reference signal within the fixed-block-length audio callback, align it with the near-end audio signal sampled by the microphone, and obtain the echo-cancelled audio signal after acoustic echo cancellation. The endpoint detection module is used to perform frame-by-frame detection on the echo-cancelled audio signal using a neural network speech activity detection model. When the duration of trailing silence reaches the endpoint threshold, it determines that the user's speech has ended and intercepts the user's speech, and continuously outputs the current duration of trailing silence and the accumulated speech duration. The speech recognition module is used to perform speech recognition on the user's speech to obtain transcribed text; The message construction module is used to construct a multimodal message from the transcribed text, binocular camera image frames, personality system prompts, and multi-turn dialogue history. The large model inference module is used to send the multimodal messages into a locally deployed, autonomously trained and fine-tuned visual language large model with its own weights, and output the response text and discrete sentiment tags in a single inference. The expression-first module is used to query the emotion-attitude mapping table based on the discrete emotion tags and drive the facial servo group to make full-face expressions before the speech synthesis begins. The synthesis and lip-syncing module is used to perform streaming speech synthesis on the reply text and play it sentence by sentence. At the same time, the neural lip-syncing model outputs joint angle sequences frame by frame based on the synthesized speech waveform to drive the facial servo group to produce lip-syncing movements. An interruption detection module is used to detect interruptions in the echo-cancelled audio signal during playback, and to interrupt playback and switch to listening mode when a user interruption is detected. The speculation execution module is used to read the current speculation generation count value as a generation marker when the current trailing silence duration reaches the speculation threshold and the accumulated speech duration is not less than the minimum speech duration. The background thread starts speech recognition in advance on the cached echo-cancelled audio signal and cascades the inference of the visual language model in advance to obtain the speculation result carrying the generation marker. When the user resumes speaking, the speculation generation count value is incremented to invalidate the old speculation result. The generation marker is checked when the user finishes speaking. If it is consistent and ready, the speculation result is adopted. If it is in execution, it is waited for a time limit. Otherwise, it is rolled back to the normal path. The pre-synthesized and cached pre-synthesized audio is played through the speaker immediately after the user finishes speaking. The maximum number of characters in the first clause of the streaming speech synthesis is limited, and forced segmentation is performed if the maximum number of characters is exceeded. The coordinator module is used to maintain the ownership of speaking rights, speaking rights stage, coordination generation count value and round number when the system includes at least two robot heads. It arbitrates the speaking rights application of the corresponding agent of each robot head by comparison and exchange, ensuring that at most one robot head has exclusive control over the inference and speaker playback of the visual language big model at any given time. It forcibly transfers speaking rights when the round times out or the heartbeat is interrupted, and performs repeat detection and prompt injection on the most recent reply text.

[0117] The modules described above cooperate with each other to implement the robot voice-expression linkage method provided in the embodiments of the present invention, which has been described in detail in the foregoing embodiments and will not be repeated here. Furthermore, when the processes described in the embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted from the content as required by the legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium may not include electrical carrier signals and telecommunication signals.

[0118] In this invention, unless otherwise explicitly specified and limited, the first feature being "on" or "below" the second feature can mean that the first feature and the second feature are in direct contact, or that the first feature and the second feature are in indirect contact through an intermediate medium.

[0119] Furthermore, "above," "on top of," and "above" the first feature in relation to the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. Similarly, "below," "under," and "beneath" the first feature in relation to the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0120] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot voice-expression linkage method, characterized in that, Includes the following steps: S1. Within the fixed block length audio callback period, the speech synthesis block just written to the speaker output buffer is used as the far-end reference signal and aligned with the near-end audio signal sampled by the microphone. The echo-cancelled audio signal is obtained after acoustic echo cancellation. S2. The echo-cancelled audio signal is detected frame by frame using a neural network speech activity detection model. When the duration of trailing silence reaches the endpoint threshold, the user's speech is determined to have ended and a segment of the user's speech is captured. S3. Perform speech recognition on the user's voice to obtain transcribed text; S4. The transcribed text, image frames captured and encoded by the binocular camera, personality system prompts, and multi-turn dialogue history are constructed into a multimodal message; S5. The multimodal message is fed into a visual language big model with its own weights. The visual language big model performs a single inference and outputs the response text and discrete sentiment tags. S6. Before speech synthesis begins, based on the discrete emotion tags, generate first servo control information to drive the facial servo group to make a gesture corresponding to the discrete emotion tags. The first servo control information is used to control the facial expression. S7. Based on the reply text, the voice information is synthesized by streaming speech and played sentence by sentence by the speaker; S8. Generate second servo control information based on the voice information and its voice waveform, wherein the second servo control information includes information for generating lip movements. S9. Based on the first servo control information and the second servo control information, and taking continuous emotion coordinates as a condition, generate third servo control information to drive continuous changes in facial expressions.

2. The voice-expression linkage method according to claim 1, characterized in that, Step S9 is followed by: S10. During playback, interruption detection is performed on the audio signal after echo cancellation. When the user's voice meets the interruption condition, playback is interrupted and the process returns to step S2 to enter the listening state to obtain a segment of user voice.

3. The voice-expression linkage method according to claim 2, characterized in that, The discrete emotion tags in step S6 include a set of 13 emotions: smiling, gratified, happy, coquettish, shy, aggrieved, confused, thinking, serious, angry, frustrated, sleepy, and surprised. Each emotion tag corresponds to a set of preset first servo control information.

4. The voice-expression linkage method according to claim 3, characterized in that, Step S2 includes the following specific steps: S201. Before the trailing silence duration reaches the endpoint threshold, the following two conditions are simultaneously met: the current trailing silence duration of the echo-cancelled audio signal reaches the speculation threshold, and the accumulated speech duration is not less than the minimum speech duration. These conditions are used as the judgment conditions to trigger speculation. The speculation threshold is preset to be less than the endpoint threshold, so that the judgment conditions are met before the endpoint is confirmed. When the judgment conditions are met, the following actions are performed: the current speculation generation count value is read as the generation marker of this speculation, and speech recognition is started in advance on the cached echo-cancelled audio signal to obtain the speculation transcribed text. Then, a speculation response is generated based on the speculation transcribed text. The speculation transcribed text, the speculation response, and the generation marker together constitute the speculation result carrying the generation marker. S202. When the duration of trailing silence falls below the speculation threshold, it is determined that the user resumes speaking and the increased speculation generation count value is submitted, thus invalidating the speculation result carrying the old generation mark. S203. When the duration of the trailing silence reaches the endpoint threshold and the user's speech is deemed to have ended, the generational marker of the speculation result is verified against the current generational speculation count, and the following three-way arbitration is executed: First, if the generational marker matches the current generational speculation count and the speculation result is ready, the speculation transcript and speculation reply are directly adopted, skipping the speech recognition in step S3 and the reasoning in step S5, and steps S6 to S9 are continued with the adopted speculation result, that is, the first servo control information is generated based on the discrete emotion tags in the speculation result, and the speculation reply is processed. The process involves streaming speech synthesis and sentence-by-sentence playback to generate second and third servo control information. The "speculation result is ready" means that the speech recognition and reasoning of the speculation thread have been completed, its generational verification has passed, and the speculation transcription text and speculation response have been written into the speculation result, which can be directly read. Secondly, if the generational marker matches the current speculation generational count value but the speculation thread is still executing and the speculation result has not yet been written, the process waits for the speculation thread to complete within a limited time. If it completes within the waiting limit, it is handled according to the first method mentioned above; if it is still not ready after exceeding the waiting limit, it is handled according to the third method below. Third, if the waiting limit is exceeded and the result is still not ready, or if the generational marker is inconsistent with the current speculative generational count, the speculative result is discarded, and speech recognition and reasoning are re-executed according to steps S3 to S5; wherein, the speculative threshold is 0.4 seconds, the minimum speech duration is 0.4 seconds, and the waiting limit is 1.5 seconds.

5. The voice-expression linkage method according to claim 4, characterized in that, Step S2, and especially step S203, also includes: S204. After determining that the user has finished speaking, pre-synthesized and cached background audio is immediately played through the speaker to mask the synthesis delay of the first speech block of the response. The background audio refers to a piece of spoken language that is played before the synthesis of the first speech block of the response is ready to fill the silence window. It does not carry the semantics of the response and only allows the robot to speak immediately after determining that the user has finished speaking. The background audio comes from a preset word pool containing spoken words. When the system starts, the background thread pre-synthesizes and caches the word pool word by word. Whether to play it in each round is determined according to a preset playback probability. The preset playback probability is 0.4, and the duration of playback from the word pool is between 0 and 4. One candidate is randomly selected from those between 0.2 seconds and 0.85 seconds, and the duration interval between 0.2 seconds and 0.85 seconds is used as a duration guard to filter out candidates whose rendering is dragged out by the synthesis engine, so that the playback end time of the background audio is no later than the ready time of the first speech block of the response, thereby preventing the overlap of the background audio and the first speech block of the response playing on the speaker at the same time; in step S7, the streaming speech synthesis limits the maximum number of characters in the first clause to 5 Chinese characters. If the first clause has more than 5 Chinese characters and there are no punctuation marks within 5 Chinese characters, it is forcibly segmented at the 5th Chinese character; in low latency mode, the endpoint threshold is 0.7 seconds.

6. The voice-expression linkage method according to claim 1, characterized in that, The process also includes a model warming step: a background daemon thread sends lightweight warming requests to the visual language large model and the speech recognition model used in step S3 at preset intervals. The lightweight warming requests sent to the visual language large model are word-level requests, and the lightweight warming requests sent to the speech recognition model are silent audio requests. This keeps the model weights of both models resident in GPU memory and the inference engine in a hot state, eliminating the cold start delay of the first round of dialogue after idle switching. The preset interval is 30 to 60 seconds. The lightweight warming requests and dialogue requests share the same interface and queue. When there is a dialogue request in transit, the lightweight warming request for this period is skipped.

7. The voice-expression linkage method according to claim 1, characterized in that, When the method is applied to a robotic system comprising at least two robot heads, it further includes a speaking rights coordination step: a central coordinator maintains an authoritative state snapshot containing the operating status, speaking rights phase, speaking rights ownership, round number, and coordination generation count value. The speaking rights phase alternates between a handover phase indicating a round and an occupying phase indicating a round in progress. Each robot head's corresponding agent periodically reads the authoritative state snapshot. When it reads that the operating status is running, the speaking rights phase is a handover phase, and the speaking rights belong to its own party, it initiates a speaking rights request to the central coordinator, carrying the read coordination generation count value and round number. The central coordinator coordinates the... The system checks whether the running status is running, whether the speaking right stage is in the handover stage, whether the applicant is the party that owns the speaking right, and whether the coordination generation count value and the round number carried are equal to the current value. If all of these conditions are met, the speaking right stage is set to the occupation stage, the coordination generation count value is incremented, and the application is returned as successful. If any condition is not met, an expired rejection is returned carrying the latest authority status snapshot, so that the speaking right application based on the expired snapshot is structurally rejected. The robot head that successfully applied exclusively takes over the reasoning of the visual language big model and the playback of the speaker and executes steps S3 to S9. The robot head that did not obtain the speaking right pauses its microphone listening and keeps its speaker silent. After the robot head finishes playing, it submits a round to the central coordinator. After the same verification, the central coordinator flips the speaking right to another robot head, sets the speaking right stage back to the handover stage, increments the round number by one, and increments the coordination generation count value again.

8. A robot voice-expression linkage system, characterized in that, include: The echo cancellation module is used to take the speech synthesis block that has just been written into the speaker output buffer as the far-end reference signal within the fixed-block-length audio callback, align it with the near-end audio signal sampled by the microphone, and obtain the echo-cancelled audio signal after acoustic echo cancellation. The endpoint detection module is used to perform frame-by-frame detection on the echo-cancelled audio signal using a neural network speech activity detection model. When the duration of trailing silence reaches the endpoint threshold, it determines that the user's speech has ended and intercepts the user's speech, and continuously outputs the current duration of trailing silence and the accumulated speech duration. The speech recognition module is used to perform speech recognition on the user's speech to obtain transcribed text; The message construction module is used to construct a multimodal message from the transcribed text, binocular camera image frames, personality system prompts, and multi-turn dialogue history. The large model inference module is used to send the multimodal messages into a locally deployed, autonomously trained and fine-tuned visual language large model with its own weights, and output the response text and discrete sentiment tags in a single inference. The expression-first module is used to query the emotion-gesture mapping table based on the discrete emotion tags and drive the facial servo group to make full-face expressions before the speech synthesis begins. The synthesis and lip-syncing module is used to perform streaming speech synthesis on the reply text and play it sentence by sentence. At the same time, the neural lip-syncing model outputs joint angle sequences frame by frame based on the synthesized speech waveform to drive the facial servo group to produce lip-syncing movements. An interruption detection module is used to detect interruptions in the echo-cancelled audio signal during playback, and to interrupt playback and switch to listening mode when a user interruption is detected. The speculation execution module is used to read the current speculation generation count value as a generation marker when the current trailing silence duration reaches the speculation threshold and the accumulated speech duration is not less than the minimum speech duration. The background thread starts speech recognition in advance on the cached echo-cancelled audio signal and cascades the inference of the visual language model in advance to obtain the speculation result carrying the generation marker. When the user resumes speaking, the speculation generation count value is incremented to invalidate the old speculation result. The generation marker is checked when the user finishes speaking. If it is consistent and ready, the speculation result is adopted. If it is in execution, it is waited for a time limit. Otherwise, it is rolled back to the normal path. The pre-synthesized and cached pre-synthesized audio is played through the speaker immediately after the user finishes speaking. The maximum number of characters in the first clause of the streaming speech synthesis is limited, and forced segmentation is performed if the maximum number of characters is exceeded. The coordinator module is used to maintain the ownership of speaking rights, speaking rights stage, coordination generation count value and round number when the system includes at least two robot heads. It arbitrates the speaking rights application of the corresponding agent of each robot head by comparison and exchange, ensuring that at most one robot head has exclusive control over the inference and speaker playback of the visual language big model at any given time. It forcibly transfers speaking rights when the round times out or the heartbeat is interrupted, and performs repeat detection and prompt injection on the most recent reply text.

9. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the robot voice-expression linkage method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot voice-expression linkage method according to any one of claims 1 to 7.

Images