A bionic robot visual line driving system and method with audio-visual fusion and human-like interactive gestures

Abstract

The application discloses a bionic robot visual line driving system and method with sound-visual fusion and humanoid interactive gestures, and belongs to the technical field of robots. The method sequentially completes sound-visual space-time fusion, multi-target priority arbitration, dialogue time sequence state determination, humanoid eye movement parameter matching and smooth trajectory generation. The system adopts a decoupled topological structure with multiple modules in one-way series connection. The application adopts a serial decision link, four-dimensional weighted hard interlocking, dialogue time sequence binding strength and weak closed loop, human-like micro eye movement arrangement and a perception quality self-adaptive mechanism. The application can improve the visual line stability in a multi-person interactive scene, realize hierarchical humanoid gestures in different interactive stages, enhance the environmental robustness and system operation reliability, and improve the naturalness and immersion of human-machine interaction of the bionic robot.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a bionic robot gaze-driven system and method that integrates sound and vision and possesses anthropomorphic interactive expressions. Background Technology

[0002] Existing technologies mainly refer to the gaze-driven technology system applied to bionic robots. This type of technology usually uses monocular vision face detection as the core perception method. It uses a camera to collect real-time images to identify the spatial position of the target face and directly drives the eye servo motor to continuously lock onto and track the target. Some improved solutions introduce a microphone array to collect sound source orientation information as a supplement to visual perception to assist gaze steering and achieve preliminary multimodal perception linkage.

[0003] However, in real-world open interactive scenarios, the aforementioned existing technologies generally suffer from multiple performance shortcomings and experience defects: insufficient spatiotemporal alignment accuracy of multimodal perception data; disordered target priority management in multi-person interaction scenarios; frequent ping-pong switching of gaze due to personnel obstruction and movement, resulting in poor interaction stability; complete disconnect between gaze-driven logic and dialogue sequence; the use of a homogenized face-locking strategy throughout; undifferentiated eye behavior at different interaction stages; overall stiffness and mechanicalness; lack of human-like expression; significant degradation in perception performance in far-field, high-noise, and low-light environments; easy loss of targets and significant jitter during tracking; weak environmental adaptability; coarse eye movement trajectory planning; abrupt and rigid servo execution; lack of micro-motion detail design, further enhancing the mechanical feel; the system architecture often adopts a flat and fusion mode; multiple data sources can easily trigger control commands simultaneously, causing oscillations; insufficient reliability for long-term continuous operation; and high algorithm iteration and maintenance costs. Summary of the Invention

[0004] The purpose of this invention is to provide a bionic robot gaze-driven system and method that integrates sound and vision and has anthropomorphic interactive expressions, thereby solving the above-mentioned technical problems.

[0005] To achieve the above objectives, the present invention provides a bionic robot gaze-driven method that integrates sound and vision and possesses anthropomorphic interactive expressions, comprising the following steps: S1. Based on the audio signal collected by the microphone array and the video signal collected by the visual camera, after parallel preprocessing and timestamp synchronization and alignment, a serial fusion decision is performed to obtain the multi-target spatial coordinates and environmental density labels with confidence. S2. Based on the spatial coordinates of multiple targets and environmental density labels, a dynamic priority arbitration algorithm is used to perform weighted sorting and locking determination to obtain the main interactive person ID and the corresponding locking flag. S3. Based on the real-time interactive data stream composed of ASR streaming results, LLM inference state, and TTS broadcast state, the state is determined by the dialogue timing state machine to obtain a unique interactive state code. S4. Based on the unique interaction status code, the human-like eye-tracking strategy library is used for index matching to obtain the four-dimensional parameter set of the gaze center point, offset amplitude, saccade cycle and blink interval in the corresponding state. S5. Based on the four-dimensional parameter set and the spatial coordinates of the main interactive person, a smooth trajectory is generated through low-pass filtering and S-shaped velocity planning to obtain the servo drive control signal.

[0006] Preferably, the timestamp synchronization alignment and spatial coordinate fusion in S1 specifically include the following steps: S11. Based on the azimuth data with global system timestamp output from the microphone array and the face detection box data with exposure center timestamp output from the visual camera, the predicted azimuth and predicted pixel coordinates at the same system scheduling time are obtained by aligning the time axis cubic spline interpolation. S12. Based on the predicted azimuth angle and predicted pixel coordinates, perform rigid body transformation of the spatial coordinate system through a pre-calibrated spatial transformation matrix to obtain the spatial coordinates of multiple targets in a unified polar coordinate system.

[0007] Preferably, the serial fusion decision in S1 also includes a dynamic computing power allocation step, specifically: S13. Based on the number of face detection boxes and the number of effective sound source trajectories in the field, the front-end computing power attention weight is dynamically adjusted by scene density classification to obtain the audio perception update frequency and visual perception update frequency in the corresponding scene. When the count is ≥2 and it is determined to be a scene with many people, the microphone array azimuth update frequency is increased to 100Hz, while the visual detection box update frequency is reduced to 15Hz; when the count is 1 and it is determined to be a scene with a single person, the visual detection frame rate is increased to 30Hz and facial key point precision localization is enabled, and the audio is only used as an auxiliary for VAD silent judgment.

[0008] Preferably, the dynamic priority arbitration algorithm in S2 specifically includes the following steps: S21. Based on four dimensions—sound source VAD activity flag, face distance, face orientation Euler angles, and ASR streaming confidence cumulative moving average—the final total score S for each target is calculated through weighted summation, and the target ID corresponding to the highest score is identified. The calculation formula is as follows: ; in, This is the source activity weighting coefficient. For face distance weighting coefficients, These are the weighting coefficients for the Euler angles. The confidence level weighting coefficient; As a marker of sound source activity, For facial distance, The human face is facing the Euler angle. The cumulative moving average of ASR streaming confidence scores; S22. Based on the locked target ID, the hard interlock flag is activated and the interlock shielding window duration is configured. The interlock shielding window duration is the effective duration of the hard interlock flag, and its value is the sum of the continuous sounding duration of the locked target and the preset holding tail duration. The azimuth angle jump interruption requests of other targets in the field other than the locked target are also blocked to obtain an exclusive locking result with anti-misoperation switching capability.

[0009] Preferably, after obtaining the main user ID and the hard interlock lock flag, the method further includes a step of dynamically switching the target binding strength, specifically: S23. Based on the current interaction status code, establish a target binding connection of corresponding strength through a closed-loop intensity dynamic switching mechanism to obtain the gaze output result in the corresponding state; When the interaction status code is in the listening state, the deviation of the target center point of each frame is used as an error term to participate in the closed-loop PID correction through a strong closed-loop negative feedback connection, so as to obtain a gaze deviation convergence result within ±0.5°. When in thinking or broadcasting mode, a weak open-loop binding mode is adopted, which allows the line of sight to drift slowly within a preset dead zone around the target origin. A global re-search is only triggered when the target misses the target for more than a preset time.

[0010] Preferably, the smooth trajectory generation in S5 includes micro-eye movement timing arrangement and dynamic correction of offset amplitude steps, specifically: S51. Based on two independent phase offset trigger pulses generated by the MCU hardware timer, and according to the asynchronous timing arrangement and conflict avoidance rules, micro-sagging and blinking actions are superimposed on the main gaze trajectory to obtain eye movement timing results with human-like micro-movement characteristics. S52, Based on Dynamically Perceived Quality Assessment Factors The offset amplitude in the four-dimensional parameter set is corrected in real time by mapping the offset amplitude coefficient to obtain eye-tracking offset parameters adapted to the current perception quality; dynamic perception quality assessment factor. The calculation formula is: ; in, CRLB normalized confidence for microphone array azimuth estimation Scoring the stability of the visual face detection bounding box; when Time offset amplitude coefficient ,when Time offset amplitude coefficient ,when Time offset amplitude coefficient And forcibly downgrade to a safe gaze state; This is the azimuth confidence weighting coefficient; It is the weighting coefficient for the stability of face detection.

[0011] A biomimetic robot gaze-driven system that integrates sound and vision and has anthropomorphic interactive expressions includes: a multimodal perception fusion module, a dynamic priority arbitration module, a dialogue timing state machine module, an anthropomorphic eye-tracking strategy library module, and a smooth trajectory driving module, which are connected in a unidirectional series. The multimodal perception fusion module is used to output multi-target spatial coordinates and environmental density labels with confidence levels; The dynamic priority arbitration module is used to receive a multi-target list and output the main interactive user ID and hard interlocking lock flag. The dialogue timing state machine module is used to monitor the entire interaction state and output a unique interaction status code. The anthropomorphic eye-tracking strategy library module is used to retrieve and output the corresponding four-dimensional eye-tracking parameter group using the interaction status code as the unique primary key. The smooth trajectory drive module is used to generate smooth servo control signals.

[0012] Preferably, the multimodal perception fusion module includes a microphone array unit, a visual camera unit, and a fusion processing unit; The microphone array unit is installed at the acoustic openings on both sides of the robot's head and is electrically connected to the fusion processing unit. It is used to collect audio signals and output azimuth data with a global system timestamp. The visual camera unit is installed in the center of the robot's forehead between its two eyes and is electrically connected to the fusion processing unit. It is used to collect video signals and output face detection box data with exposure center timestamp. The fusion processing unit has a built-in Flash storage medium and a spatial transformation matrix from the camera pixel coordinate system to the microphone polar coordinate system, which is used to perform timestamp alignment, spatial transformation and scene density determination. The fusion processing unit is also equipped with an environmental noise monitoring unit, which is used to calculate the signal-to-noise ratio of speech segments in real time, and dynamically switch between sound-visual fusion mode, visual priority mode and pure visual emergency mode according to multi-level signal-to-noise ratio thresholds.

[0013] Preferably, the dialogue timing state machine module is connected to the external ASR engine, LLM service, and TTS engine signals respectively, and is used to monitor the ASR streaming intermediate results, the LLM inference in progress flag, and the TTS broadcast status flag. The dialogue timing state machine module has built-in interactive states including listening, thinking, broadcasting, and hesitant states, and each state transition is configured with an anti-shaking lag mechanism; the listening state activation requires that the voice activity be continuous and effective and the speech recognition output the first valid result; the thinking state activation requires that the voice activity be silent and the large language model return to the inference state; the broadcasting state activation requires that the speech synthesis output valid audio data; and the hesitant state activation requires that the speech recognition confidence is insufficient or the semantic classification uncertainty is too high.

[0014] Preferably, the anthropomorphic eye-tracking strategy library module has a built-in parameter storage unit and a dynamic coefficient mapping unit; The parameter storage unit is used to store the four-dimensional eye-tracking reference parameters corresponding to each interaction state. The dynamic coefficient mapping unit is connected to the output of the multimodal perception fusion module and is used to receive dynamic perception quality evaluation factors. And output offset magnitude correction coefficient This allows for real-time adjustment of the reference offset.

[0015] Therefore, the present invention employs the above-mentioned bionic robot gaze-driven system and method with sound-visual fusion and anthropomorphic interactive expressions, which has the following beneficial effects: 1. The system adopts a five-module unidirectional serial architecture consisting of a multimodal perception fusion module, a dynamic priority arbitration module, a dialogue timing state machine module, a humanoid eye-tracking strategy library module, and a smooth trajectory driving module. It follows a data flow topology of parallel preprocessing → serial fusion decision → conditional branch mapping, and limits the humanoid eye-tracking strategy library to using only the interaction status code as the unique retrieval key, without accessing the original perception data. This architecture eliminates the logical race condition caused by the parallel triggering of multi-source data, ensuring that the trigger source of each eye movement command is unique and traceable, thus avoiding the command oscillation problem of traditional flat architectures. At the same time, the logic of each module is decoupled, allowing for independent iterative upgrades and exception resets, improving the reliability and code reusability of the system during long-term continuous operation, and reducing the cost of secondary development and maintenance.

[0016] 2. By aligning audio and video data using global timestamp cubic spline interpolation, the temporal dimension is synchronized. Combined with a pre-calibrated spatial transformation matrix, the coordinate system rigid body transformation is completed. Simultaneously, the scene density is determined based on the number of faces and sound sources in the field, and the computing power weight and update frequency of the audiovisual perception branch are dynamically adjusted. This solves the "sound mismatch" problem caused by differences in sampling frequency and transmission delay jitter between audio and video, improves the accuracy of fusion decision, eliminates the lag and stuttering of eye-tracking, and achieves on-demand dynamic allocation of perception resources under the constraint of limited embedded main control computing power. This avoids missed detections caused by visual computing power overload in multi-person far-field scenes, and ensures sub-pixel stability of gaze in single-person near-field scenes, compressing the perception latency of the main interactive object in the entire scene to the hard real-time requirements of real-time interaction.

[0017] 3. A target priority ranking algorithm with four-dimensional weighting of sound source activity, face distance, face facing angle, and speech recognition confidence was constructed, and a hard-interlocked shielding window that maintains a trailing effect as the speech duration is set up. This solves the problem of ping-pong switching of gaze caused by multiple people interrupting in open scenes, occasional voices from the side, and brief appearances of passersby. It reduces the false gaze switching rate in densely interactive scenarios, enabling the robot's social gaze to have the focus and exclusivity that conforms to human interaction habits, and improves the interaction stability and natural experience in multi-person interaction scenarios.

[0018] 4. By linking the entire interaction state chain of ASR, LLM, and TTS through the dialogue temporal state machine, four types of interaction states are divided: listening, thinking, broadcasting, and hesitating. Differentiated target binding strength and gaze offset rules are matched accordingly. The listening state adopts strong closed-loop negative feedback to achieve high-precision locking, while the thinking and broadcasting states are downgraded to weak open-loop binding to allow for relaxed gaze drift. This aligns the robot's eye behavior with the temporal depth of dialogue semantics, giving different interaction stages a clear and distinguishable facial expression level. It ensures high precision and stability of gaze locking in the listening state, while giving the thinking and broadcasting states a natural sense of relaxation in the eyes. This enables flexible coexistence and real-time switching of different interaction intentions on the same hardware platform.

[0019] 5. Using two independent phase random trigger pulses, micro-saccades and blinking movements are asynchronously superimposed on the main gaze trajectory, and conflict avoidance rules are configured. At the same time, a dynamic perception quality assessment factor that integrates acoustic and visual confidence is introduced to map and adjust the eye movement deviation amplitude coefficient in real time. On the one hand, this scheme superimposes secondary micro-movements that conform to human physiological characteristics on the basic gaze behavior, giving the robot's eyes a vivid rhythm and improving the user's subjective perception score of the robot "having internal consciousness". On the other hand, it can flexibly reduce the eye movement amplitude according to the degree of environmental noise and light interference, maintain restrained and reasonable eye behavior in low perception quality environments, avoid abnormal actions such as jumping or staring into space due to erroneous perception data, and ensure the trust in the interaction process.

[0020] 6. A three-level perceptual modal routing strategy based on the instantaneous signal-to-noise ratio of speech segments is set up, along with a large model inference timeout watchdog timer and a forced rollback mechanism for abnormal states. This design enables the robot to adapt to acoustic environments with varying levels of noise, maintaining stable and non-erratic basic eye-tracking interaction capabilities even in extremely noisy scenarios, thus broadening the application boundaries of bionic robots in open commercial scenarios. Simultaneously, it avoids prolonged "crash staring" due to network fluctuations and computing power blockages in upstream intelligent agents, ensuring that the robot can maintain a minimum level of natural facial expression output even when upstream services are abnormal, thereby improving the fault tolerance resilience of the product's actual deployment and user safety.

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] Figure 1 The overall structural diagram of a bionic robot gaze-driven system with sound and vision fusion and anthropomorphic interactive expressions provided by the present invention; Figure 2 A flowchart of the audiovisual data timestamp synchronization and spatial fusion process for a bionic robot gaze-driven method with anthropomorphic interactive expressions provided by the present invention. Figure 3 The flowchart of multi-target dynamic priority arbitration and main interaction target locking of a bionic robot gaze-driven method with sound and vision fusion and anthropomorphic interactive expressions provided by the present invention; Figure 4 A schematic diagram of the dialogue timing state machine and state switching conditions for a bionic robot gaze-driven method with sound and vision fusion and anthropomorphic interactive expressions provided by the present invention. Figure 5 A schematic diagram showing the gaze offset range under different interactive states of a bionic robot gaze-driven method with sound-visual fusion and anthropomorphic interactive expressions provided by the present invention. Figure 6 A schematic diagram illustrating the timing arrangement of micro-scanning, blinking, and main gaze trajectories in a bionic robot gaze-driven method that integrates sound and vision and possesses anthropomorphic interactive expressions, provided by the present invention. Figure 7 The flowchart shows the adaptive degradation process of gaze offset driven by the perception quality Q in a bionic robot gaze-driven method that integrates sound and vision and has anthropomorphic interactive expressions, as provided by the present invention. Figure 8 A flowchart of the audio-visual modal switching based on instantaneous signal-to-noise ratio for a gaze-driven method for a bionic robot with sound-visual fusion and anthropomorphic interactive expressions provided by the present invention. Figure 9A schematic diagram of an acoustic-visual joint calibration device and coordinate transformation for a bionic robot gaze-driven method with acoustic-visual fusion and anthropomorphic interactive expressions provided by the present invention. Figure 10 The flowchart illustrates the LLM inference timeout and system anomaly safety rollback process for a gaze-driven method for a bionic robot with sound-visual fusion and anthropomorphic interactive expressions, as provided in this invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of the present invention and are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.

[0024] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as a process, method, system, product, or server that includes a series of steps or units, not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or device.

[0025] The following is in conjunction with the appendix Figures 1-10 The embodiments of the present invention will be described in detail below.

[0026] Existing bionic robot gaze-driven technologies mostly employ monocular vision face tracking schemes, directly driving eye servos to continuously lock onto the target by detecting the face position through a camera. While some improved schemes introduce microphone arrays to supplement sound source location information, they still have many shortcomings in practical applications: In multi-person interaction scenarios, gaze switching is prone to frequent ping-pong due to occlusion and target ID changes, lacking effective priority arbitration and exclusive locking mechanisms; gaze-driven logic is completely disconnected from dialogue sequence, using a homogeneous face locking strategy throughout, failing to distinguish the differences in expression at different interaction stages, resulting in stiff and mechanical eye movements; perception performance degrades significantly in far-field and high-noise environments, lacking flexible modal switching and perception quality adaptation mechanisms, easily leading to target loss, tracking jitter, or even abnormal scanning; eye movement planning is crude, with servos using uniform linear motion and lacking human-like micro-motion design, further enhancing the mechanical feel; the system architecture is mostly a flat and fused mode, with multi-source data parallel triggering easily causing logic race conditions and command oscillations, resulting in insufficient reliability during long-term operation and difficulty in meeting the natural interaction requirements of real-world scenarios.

[0027] Based on the above analysis, specific implementation methods are designed, including: Example 1: This embodiment provides a bionic robot gaze-driven system and corresponding driving method that integrates sound and vision and possesses anthropomorphic interactive expressions. The overall system architecture is as follows: Figure 1 As shown, a decoupled topology structure with five modules connected in unidirectional series is adopted, and the driving method steps are as follows: S1. Based on the audio signal collected by the microphone array and the video signal collected by the visual camera, after parallel preprocessing and timestamp synchronization and alignment, a serial fusion decision is performed to obtain the multi-target spatial coordinates and environmental density labels with confidence.

[0028] The execution flow of this step's audio-visual spatiotemporal fusion is as follows: Figure 2 As shown, the microphone array outputs azimuth data at a sampling rate of 16kHz and a frame length of 10ms, and binds it to a hardware global timestamp. The visual camera outputs face detection bounding boxes at 30fps and binds them to a global timestamp of the exposure center moment. The two data streams are sent in parallel to the fusion node for unified processing, which specifically includes the following sub-steps: S11. Time axis synchronization alignment: Based on the azimuth data with global system timestamp output by the microphone array and the face detection box data with exposure center timestamp output by the visual camera, the predicted azimuth and predicted pixel coordinates at the same system scheduling time are obtained through time axis cubic spline interpolation alignment processing.

[0029] See Figure 2 As shown, the fusion node uses the current system scheduling time as a benchmark, backtracks 50ms into the time window, selects the two most recent sets of acoustic and visual data within the window, and calculates the predicted azimuth angle and predicted pixel coordinates at the current scheduling time through cubic spline interpolation. This eliminates the misalignment problem caused by differences in sampling frequency and transmission delay in acoustic and visual data from the time dimension, ensuring the spatiotemporal consistency of the fused data and improving the accuracy of the fusion decision by more than 40% compared to the direct synchronization method.

[0030] S12. Unified transformation of spatial coordinates: Based on the predicted azimuth angle and predicted pixel coordinates, a rigid body transformation of the spatial coordinate system is performed through a pre-calibrated spatial transformation matrix to obtain the spatial coordinates of multiple targets in a unified polar coordinate system.

[0031] The spatial transformation matrix is ​​solved in advance through a joint audio-visual calibration process. For the calibration device and process, please refer to [link / reference needed]. Figure 9As shown: An integrated calibration fixture is used, with an AprilTag visual marker of known spatial coordinates fixed at the center of the fixture. Four audio signal transmitters are evenly arranged around the marker along a circle with a radius of 10cm. During calibration, the robot faces the calibration fixture, the camera captures the center pixel coordinates of the QR code, and the microphone array receives the four audio signals. The azimuth angles of the four sound sources are calculated using the TDOA algorithm, and the geometric mean is taken to obtain the acoustic center azimuth angle. The system solves the spatial transformation matrix H from the camera pixel coordinate system to the microphone polar coordinate system using the least squares method, and the matrix H is fixed in the Flash storage medium. The entire process takes no more than 30 seconds, which can eliminate system-level coupling errors caused by microphone misalignment and camera tilt during production assembly, ensuring high-precision and uniform acoustic-visual spatial pointing of the equipment leaving the factory, and providing a precise input reference for the upper-level fusion algorithm.

[0032] S13. Dynamic allocation of scene computing power: Based on the number of face detection boxes and the number of effective sound source trajectories in the scene, the attention weight of the front-end computing power is dynamically adjusted by the scene density classification to obtain the audio perception update frequency and visual perception update frequency in the corresponding scene.

[0033] When the detected face and sound source count is ≥2, it is determined to be a multi-person, dense scene. The front-end computing power is actively shifted towards the audio branch, increasing the microphone array azimuth update frequency to 100Hz, while the visual detection box update frequency is reduced to 15Hz. When the count is 1, it is determined to be a single-person, clean scene. The computing power is shifted towards the visual branch, increasing the visual detection frame rate to 30Hz and enabling precise facial landmark localization. The audio branch only serves as a silent auxiliary for VAD (Visual Analogue Detection). Under the constraint of limited embedded main control computing power, this mechanism enables on-demand allocation of perception resources, compressing the perception latency of the main interactive object across the entire scene to within 80ms, meeting the hard real-time requirements of real-time interaction.

[0034] S2. Based on the spatial coordinates of multiple targets and the environmental density labels, a dynamic priority arbitration algorithm is used to perform weighted sorting and locking determination to obtain the main interactive person ID and the corresponding locking flag.

[0035] For the complete execution process of multi-target arbitration and locking, please refer to Figure 3 As shown, this illustrates the end-to-end logic from inputting a multi-target list to outputting the main target ID, specifically including the following sub-steps: S21. Four-dimensional weighted scoring and ranking: Based on four dimensions—voice source VAD activity marker, face distance, face orientation Euler angles, and ASR streaming confidence cumulative moving average—the final total score for each target is calculated through weighted summation. Lock onto the target ID corresponding to the highest score.

[0036] The calculation formula is as follows: ; in, The sound source activity weighting coefficient is 0.6, which serves as a veto condition. If a sound source is identified, it is directly promoted to the first position in the candidate queue. This is a face distance weighting coefficient, with a value of 0.2; the closer the face, the higher the score. The Euler angle weighting coefficient is 0.1, and the higher the degree of face alignment, the higher the score. This is the confidence level weighting coefficient, with a value of 0.1; As a marker of sound source activity, For facial distance, The human face is facing the Euler angle. The cumulative moving average of ASR streaming confidence scores; S22. Hard Interlock Exclusive Locking: Based on the locked target ID, by activating the hard interlock flag and configuring the interlock shielding window duration (i.e., the effective duration of the hard interlock flag), the value is the continuous sounding duration of the locked target plus a preset 300ms hold tail duration. This shields azimuth angle jump interruption requests from targets other than the locked target within the field, resulting in an exclusive locking result with anti-misoperation switching capability. See also Figure 3 As shown, after locking the highest-scoring target, the hardware lock flag is immediately activated to block the azimuth angle jumps of other targets and keep the trailing window at the sound duration + 300ms. This mechanism can reduce the false eye switching rate in multi-person dense interaction scenarios to less than 3%, and can avoid the ping-pong switching of eye contact caused by multiple people interrupting, passing people briefly entering the frame, and people speaking from the side. This makes the eye direction have the focused exclusivity that conforms to human social habits.

[0037] S23. Target binding strength dynamic switching: Based on the current interaction status code, a target binding connection of corresponding strength is established through a closed-loop strength dynamic switching mechanism to obtain the gaze output result in the corresponding state.

[0038] When the interaction status code is in the listening state, a strong closed-loop negative feedback connection is established, and the deviation of the target center point in each frame is used as an error term to participate in the closed-loop PID correction, so that the gaze deviation converges within ±0.5°, ensuring the high accuracy and stability of gaze locking and preventing target loss due to attitude disturbances. When the interaction status code is in the thinking or broadcasting state, it is downgraded to a weak open-loop binding mode. The system locks the origin of the original target spatial coordinate system, allowing the line of sight to drift freely and slowly within a dead zone of ±8° of the field of view around the target origin. The drifting process does not trigger the re-locking logic, and only triggers the global re-search when the target misses the target for more than 2 seconds, giving the gaze a natural sense of relaxation and freedom during the thinking and broadcasting stages.

[0039] S3. Based on the real-time interactive data stream composed of ASR streaming results, LLM inference state, and TTS broadcast state, the state is determined by the dialogue timing state machine to obtain a unique interactive state code.

[0040] See the state transition logic and jump relationships of the dialogue sequence state machine. Figure 4 As shown, the triggering conditions and transition paths for four core states—listening, thinking, broadcasting, and hesitating—are clearly presented. The dialogue sequence state machine module is connected to the external ASR engine, LLM service, and TTS engine signals, respectively, to monitor ASR streaming intermediate results, LLM inference in progress flags, and TTS broadcasting status flags in real time. Each state transition is configured with a Schmitt trigger-type debouncing hysteresis mechanism, with the specific activation conditions as follows: Listening state: VAD voice activity detection is continuously effective for ≥300ms, and ASR outputs the first timestamp token; the silence threshold for exiting the listening state is set to 500ms, and a 200ms insensitive zone is set to suppress state jitter; Thinking state: The VAD silent detection has been continuously valid for ≥500ms, and the LLM service returns the "Inference in progress" status code; Broadcast status: The TTS engine outputs the "First frame of audio data has been enqueued" flag, and the current state is non-silent; Hesitant state: After the user finishes speaking, the moving average confidence score returned by ASR is <0.6, or the domain classification entropy value returned by the semantic parser is >0.8.

[0041] The state machine only outputs a unique state code as the retrieval key for the downstream policy library, without transmitting raw perception data. This ensures the uniqueness and traceability of the decision-making chain's trigger source, avoiding instruction oscillation. Furthermore, the state machine is configured with a safe rollback mechanism to handle LLM inference timeout exceptions. For the complete exception handling process, please refer to [link to full example]. Figure 10 As shown: Define the maximum safe dwell time window for the thinking state. From the moment the system detects that the VAD has been silent for ≥500ms and calls the LLM interface, a hardware watchdog timer is started to count. If the count reaches 8s and no incremental token or end flag is received from the LLM, it is determined to be an inference timeout exception. The state machine is forced to exit the thinking state and enter the standby gentle scanning state, controlling the gaze axis to make a slow sinusoidal motion within the horizontal ±15° and vertical ±10° range. At the same time, an asynchronous liveness detection thread is started in the background, sending a lightweight heartbeat detection to the LLM service every 2s. After the service recovers, the unfinished inference results are retrieved again and cached in the local queue, waiting for the next round of interaction. This avoids the robot from falling into a long period of stagnation due to upstream service anomalies, improving the system's fault tolerance and user safety.

[0042] S4. Based on the unique interaction status code, index matching is performed through the anthropomorphic eye-tracking strategy library to obtain the four-dimensional parameter set of the gaze center point, offset amplitude, saccade period, and blink interval in the corresponding state.

[0043] See gaze offset range under different interaction states. Figure 5As shown, with the center O of the basic gaze visual axis as the origin, the differences in the expected deviation of the visual axis in each state are quantified in polar coordinates, and the specific calibration is as follows: Listening state: the deviation range is [0, 1°], the clamp is strictly located near the key points of the target face, and only micro-scanning and micro-movement are superimposed; Broadcast mode: The deviation range is [1°, 3°], using a gentle offset to simulate the natural and relaxed state of humans occasionally looking away when speaking; Thinking state: The deviation range is [5°, 10°]. A large-amplitude slow offset is used, with random directions and evenly distributed landing points within a disk centered at O. The offset speed does not exceed 2° / s to avoid abrupt movements. Hesitant state: A second-order underdamped damped oscillatory response is superimposed on the thinking state offset. The line of sight first deflects rapidly to one side by about 12°, and then at a rate of attenuation. Oscillation period The pattern of the movement decays and stabilizes near the center, simulating the micro-movement trajectory of the eyes when humans hesitate and repeatedly confirm.

[0044] By clearly distinguishing the difference in offset ratios between different states, each eye movement of the robot has an interpretable psychological state mapping, enhancing the semantic readability of eye behavior, and providing quantitative optimization space for subsequent reverse optimization of parameters based on user subjective evaluation.

[0045] S5. Based on the four-dimensional parameter set and the spatial coordinates of the main interactive person, a smooth trajectory is generated through low-pass filtering and S-shaped velocity planning to obtain the servo drive control signal.

[0046] The smooth trajectory generation process includes micro-eye movement timing arrangement and dynamic correction of offset amplitude: S51. Human-like Micro-Eye Movement Timing Arrangement: Based on two independent phase-shifted trigger pulses generated by the MCU hardware timer, micro-saccades and blinking movements are superimposed on the main gaze trajectory according to asynchronous timing arrangement and conflict avoidance rules, resulting in an eye movement timing result with human-like micro-movement characteristics. The temporal arrangement relationship of the three is shown in [reference needed]. Figure 6 As shown.

[0047] The micro-sagittal trigger pulse cycle is a random jitter of 3.5s to 5.5s, with a sagittal amplitude of ±2° to ±4° from the gaze center and a sagittal duration of 120ms to ±20ms. After a rapid jump, it returns to the gaze center. The blink trigger pulse uses an independent random sequence of 2.8s to 6.2s, with a closed-eye holding time of 150ms to 250ms. Both pulses are configured with conflict avoidance rules. When the time windows of the sagittal pulse and the blink pulse overlap, the blink pulse is forcibly delayed by 20ms to 80ms to ensure that they are executed sequentially in a time-sharing manner, avoiding mechanical jamming and visual abruptness. Human-like secondary micro-motion features are superimposed on the main gaze trajectory, giving the eyes a vibrant, breathing rhythm and enhancing the user's subjective perception that the robot "has intrinsic consciousness."

[0048] S52. Perceived Quality Adaptive Offset Correction: Based on Dynamic Perceived Quality Assessment Factors The offset magnitude in the four-dimensional parameter set is corrected in real time by mapping the offset magnitude coefficient to obtain eye-tracking offset parameters that adapt to the current perception quality. For the complete adaptive degradation process, please refer to [link to documentation]. Figure 7 As shown.

[0049] Dynamically perceived quality assessment factors The calculation formula is: ; in, CRLB normalized confidence for microphone array azimuth estimation Scoring the stability of the visual face detection bounding box; when Time offset amplitude coefficient ,when Time offset amplitude coefficient ,when Time offset amplitude coefficient And forcibly downgrade to a safe gaze state; This is the azimuth confidence weighting coefficient; It is the weighting coefficient for the stability of face detection.

[0050] See Figure 7 As shown, according to Value partitioning three-level offset strategy: when At that time, it was determined to be of high perceived quality, and the offset amplitude coefficient was... Perform a normal anthropomorphic offset of ±8° from the default thinking state; when At that time, it was judged to be of medium perceived quality. This reduces the thought state offset to ±4°, avoiding meaningless saccades caused by unreliable positioning; when When this is the case, it is judged as low perceived quality. It forcibly freezes all active offsets, retaining only minute tremors of ±0.5° at the center point, and forcibly degrades to a safe gaze state until perceptual quality is restored. This flexible degradation mechanism allows eye-movement behavior to adapt to environmental noise and light interference, avoiding abrupt actions such as large jumps or staring into space caused by abnormal perception, thus ensuring a sense of trust in the interaction.

[0051] In addition, the system is configured with an adaptive routing mechanism for environmental noise; see the three-level mode switching process. Figure 8 As shown: The system background starts a low-priority silent environment monitoring thread to calculate the long-term average power spectral density of the ambient noise floor in real time, dynamically evaluate the instantaneous signal-to-noise ratio in the 300Hz~3400Hz speech frequency band, and set three-level decision thresholds: When the signal-to-noise ratio is ≥15dB, in a quiet / slightly noisy environment, switch to the sound and vision fusion mode, and all sound and vision data participate in the fusion calculation; When 5dB≤Signal-to-Noise Ratio<15dB, in a moderately noisy environment, switch to visual priority mode. Audio data is only used for VAD liveness detection to trigger state switching. The azimuth data is multiplied by a weight attenuation coefficient of 0.3 and does not participate in the calculation of the main interactive human's spatial coordinates. When the signal-to-noise ratio is <5dB, in an extremely noisy environment, switch to pure vision emergency mode. The microphone array output data is isolated by software gating and does not enter the fusion node. The system is forced to switch to pure vision single-mode tracking. At the same time, the state machine silence judgment threshold is shortened from 500ms to 200ms to speed up the response speed under vision dominance.

[0052] This strategy enables robots to maintain stable basic eye-tracking interaction capabilities even in extremely harsh acoustic environments such as shopping malls and train stations, expanding the boundaries of open application scenarios for the product.

[0053] The overall architecture of the bionic robot gaze-driven system that implements the above method can be found in [reference needed]. Figure 1 As shown, it includes a multimodal perception fusion module, a dynamic priority arbitration module, a dialogue temporal state machine module, a human-like eye-tracking strategy library module, and a smooth trajectory driving module, all connected in a unidirectional series. Multimodal perception fusion module: Used to output multi-target spatial coordinates and environmental density labels with confidence, including a microphone array unit, a vision camera unit, and a fusion processing unit; the microphone array unit is installed at the acoustic openings on both sides of the robot's head and is electrically connected to the fusion processing unit, used to collect audio signals and output azimuth data with global system timestamps; the vision camera unit is installed at the center of the robot's forehead between the robot's eyes and is electrically connected to the fusion processing unit, used to collect video signals and output face detection box data with exposure center timestamps; the fusion processing unit has built-in Flash storage medium, which has a spatial transformation matrix from the camera pixel coordinate system to the microphone polar coordinate system, used to perform timestamp alignment, spatial transformation, and scene density determination; it also has an environmental noise monitoring subunit, used to calculate the signal-to-noise ratio of speech segments in real time, and dynamically switch between sound-visual fusion mode, vision priority mode, and pure vision emergency mode according to multi-level signal-to-noise ratio thresholds.

[0054] Dynamic Priority Arbitration Module: Used to receive a multi-target list, perform four-dimensional weighted scoring and hard interlocking, and output the main interactor ID and hard interlocking flag.

[0055] Dialogue Sequence State Machine Module: Used to monitor the full-link interaction status of ASR, LLM, and TTS, perform status judgment and exception rollback, and output a unique interaction status code.

[0056] Humanoid eye-tracking strategy library module: used to retrieve corresponding four-dimensional eye-tracking parameter sets by using the interaction status code as the unique primary key; it has a built-in parameter storage unit and a dynamic coefficient mapping unit. The parameter storage unit stores the four-dimensional eye-tracking baseline parameters corresponding to each interaction state. The dynamic coefficient mapping unit is connected to the output end of the multimodal perception fusion module to receive dynamic perception quality evaluation factors and output offset amplitude correction coefficients to adjust the baseline offset amplitude in real time.

[0057] Smooth trajectory drive module: Used to receive eye movement parameters and target coordinates, perform S-shaped velocity planning, micro-eye movement superposition and low-pass filtering, and output smooth servo control signals to drive the eyeball mechanism.

[0058] In summary, this embodiment eliminates multi-source data logic competition at its source through a layered and decoupled serial decision-making architecture. It combines precise spatiotemporal fusion of sound and vision with dynamic allocation of computing power to ensure both perception accuracy and real-time performance across all scenarios. It addresses the issue of accidental gaze switching in multi-person scenarios by relying on four-dimensional weighted arbitration and hard interlocking mechanisms. It achieves human-like facial expression layering at different interaction stages through a strong-weak closed-loop strategy of dialogue timing binding. It also incorporates human-like micro-eye movement choreography and an adaptive degradation mechanism for perception quality. In addition, it features environmental noise routing and anomaly safety backoff design, comprehensively improving the naturalness, stability, and environmental robustness of the bionic robot's gaze interaction, and meeting the immersive interaction needs in real open scenarios.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for driving the gaze of a biomimetic robot that integrates sound and vision and possesses anthropomorphic interactive expressions, characterized in that, Includes the following steps: S1. Based on the audio signal collected by the microphone array and the video signal collected by the visual camera, after parallel preprocessing and timestamp synchronization and alignment, a serial fusion decision is performed to obtain the multi-target spatial coordinates and environmental density labels with confidence. S2. Based on the spatial coordinates of multiple targets and environmental density labels, a dynamic priority arbitration algorithm is used to perform weighted sorting and locking determination to obtain the main interactive person ID and the corresponding locking flag. S3. Based on the real-time interactive data stream composed of ASR streaming results, LLM inference state, and TTS broadcast state, the state is determined by the dialogue timing state machine to obtain a unique interactive state code. S4. Based on the unique interaction status code, the human-like eye-tracking strategy library is used for index matching to obtain the four-dimensional parameter set of the gaze center point, offset amplitude, saccade cycle and blink interval in the corresponding state. S5. Based on the four-dimensional parameter set and the spatial coordinates of the main interactive person, a smooth trajectory is generated through low-pass filtering and S-shaped velocity planning to obtain the servo drive control signal.

2. The method for driving the gaze of a bionic robot with sound-visual fusion and anthropomorphic interactive expressions according to claim 1, characterized in that: The timestamp synchronization alignment and spatial coordinate fusion in S1 specifically include the following steps: S11. The azimuth data with global system timestamp output from the microphone array and the face detection box data with exposure center timestamp output from the visual camera are aligned by cubic spline interpolation on the time axis to obtain the predicted azimuth and predicted pixel coordinates at the same system scheduling time. S12. Based on the predicted azimuth angle and predicted pixel coordinates, perform rigid body transformation of the spatial coordinate system through a pre-calibrated spatial transformation matrix to obtain the spatial coordinates of multiple targets in a unified polar coordinate system.

3. The method for driving the gaze of a bionic robot with sound-visual fusion and anthropomorphic interactive expressions according to claim 2, characterized in that: The serial fusion decision in S1 also includes a dynamic computing power allocation step, specifically: S13. Based on the number of face detection boxes and the number of effective sound source trajectories in the field, the front-end computing power attention weight is dynamically adjusted by scene density classification to obtain the audio perception update frequency and visual perception update frequency in the corresponding scene. When the count is ≥2 and it is determined to be a scene with many people, the microphone array azimuth update frequency is increased to 100Hz, while the visual detection box update frequency is reduced to 15Hz; when the count is 1 and it is determined to be a scene with a single person, the visual detection frame rate is increased to 30Hz and facial key point precision localization is enabled, and the audio is only used as an auxiliary for VAD silent judgment.

4. The method for driving the gaze of a bionic robot with sound-visual fusion and anthropomorphic interactive expressions as described in claim 3, characterized in that: The dynamic priority arbitration algorithm in S2 specifically includes the following steps: S21. Based on four dimensions—voice source VAD activity marker, face distance, face orientation Euler angles, and ASR streaming confidence cumulative moving average—the final total score for each target is calculated through weighted summation. The target ID corresponding to the highest score is locked; the calculation formula is: ； in, This is the source activity weighting coefficient. For face distance weighting coefficients, These are the weighting coefficients for the Euler angles. The confidence level weighting coefficient; As a marker of sound source activity, For facial distance, The human face is facing the Euler angle. The cumulative moving average of ASR streaming confidence scores; S22. Based on the locked target ID, the hard interlock flag is activated and the interlock shielding window duration is configured. The interlock shielding window duration is the effective duration of the hard interlock flag, and its value is the sum of the continuous sounding duration of the locked target and the preset holding tail duration. The azimuth angle jump interruption requests of other targets in the field other than the locked target are also blocked to obtain an exclusive locking result with anti-misoperation switching capability.

5. The method for driving the gaze of a bionic robot with sound-visual fusion and anthropomorphic interactive expressions according to claim 4, characterized in that: After obtaining the main user ID and the hard interlock lock flag, the process also includes a dynamic switching step for the target binding strength, specifically: S23. Based on the current interaction status code, establish a target binding connection of corresponding strength through a closed-loop strength dynamic switching mechanism to obtain the gaze output result in the corresponding state; When the interaction status code is in the listening state, the deviation of the target center point of each frame is used as an error term to participate in the closed-loop PID correction through a strong closed-loop negative feedback connection, so as to obtain a gaze deviation convergence result within ±0.5°. When in thinking or broadcasting mode, a weak open-loop binding mode is adopted, which allows the line of sight to drift slowly within a preset dead zone around the target origin. A global re-search is only triggered when the target misses the target for more than a preset time.

6. The method for driving the gaze of a bionic robot with sound-visual fusion and anthropomorphic interactive expressions according to claim 1, characterized in that: The smooth trajectory generation in S5 includes micro-eye movement timing arrangement and dynamic correction of offset amplitude, specifically: S51. Based on two independent phase offset trigger pulses generated by the MCU hardware timer, and according to the asynchronous timing arrangement and conflict avoidance rules, micro-sagging and blinking actions are superimposed on the main gaze trajectory to obtain eye movement timing results with human-like micro-movement characteristics. S52, Based on Dynamically Perceived Quality Assessment Factors The offset amplitude in the four-dimensional parameter set is corrected in real time by mapping the offset amplitude coefficient to obtain eye-tracking offset parameters adapted to the current perception quality; dynamic perception quality assessment factor. The calculation formula is: ； in, CRLB normalized confidence for microphone array azimuth estimation Scoring the stability of the visual face detection bounding box; when Time offset amplitude coefficient ,when Time offset amplitude coefficient ,when Time offset amplitude coefficient And forcibly downgrade to a safe gaze state; This is the azimuth confidence weighting coefficient; It is the weighting coefficient for the stability of face detection.

7. A bionic robot gaze-driving system with acoustic-visual fusion and anthropomorphic interactive expressions, used to implement the bionic robot gaze-driving method with acoustic-visual fusion and anthropomorphic interactive expressions as described in any one of claims 1-6, characterized in that, include: The module consists of a multimodal perception fusion module, a dynamic priority arbitration module, a dialogue timing state machine module, a human-like eye-tracking strategy library module, and a smooth trajectory driving module, which are connected in a unidirectional series. The multimodal perception fusion module is used to output multi-target spatial coordinates and environmental density labels with confidence levels; The dynamic priority arbitration module is used to receive a multi-target list and output the main interactive user ID and hard interlocking lock flag. The dialogue timing state machine module is used to monitor the entire interaction state and output a unique interaction status code. The anthropomorphic eye-tracking strategy library module is used to retrieve and output the corresponding four-dimensional eye-tracking parameter group using the interaction status code as the unique primary key. The smooth trajectory drive module is used to generate smooth servo control signals.

8. A bionic robot gaze-driven system with acoustic-visual fusion and anthropomorphic interactive expressions as described in claim 7, characterized in that: The multimodal perception fusion module includes a microphone array unit, a visual camera unit, and a fusion processing unit; The microphone array unit is installed at the acoustic openings on both sides of the robot's head and is electrically connected to the fusion processing unit. It is used to collect audio signals and output azimuth data with a global system timestamp. The visual camera unit is installed in the center of the robot's forehead between its two eyes and is electrically connected to the fusion processing unit. It is used to collect video signals and output face detection box data with exposure center timestamp. The fusion processing unit has a built-in Flash storage medium and a spatial transformation matrix from the camera pixel coordinate system to the microphone polar coordinate system, which is used to perform timestamp alignment, spatial transformation and scene density determination. The fusion processing unit is also equipped with an environmental noise monitoring unit, which is used to calculate the signal-to-noise ratio of speech segments in real time, and dynamically switch between sound-visual fusion mode, visual priority mode and pure visual emergency mode according to multi-level signal-to-noise ratio thresholds.

9. A bionic robot gaze-driven system with acoustic-visual fusion and anthropomorphic interactive expressions as described in claim 8, characterized in that: The dialogue timing state machine module is connected to the external ASR engine, LLM service, and TTS engine signals respectively, and is used to monitor the ASR streaming intermediate results, the LLM inference in progress flag, and the TTS broadcast status flag. The dialogue timing state machine module has built-in interactive states including listening, thinking, broadcasting, and hesitant states, and each state transition is configured with an anti-shaking lag mechanism; the listening state activation requires that the voice activity be continuous and effective and the speech recognition output the first valid result; the thinking state activation requires that the voice activity be silent and the large language model return to the inference state; the broadcasting state activation requires that the speech synthesis output valid audio data; and the hesitant state activation requires that the speech recognition confidence is insufficient or the semantic classification uncertainty is too high.

10. A bionic robot gaze-driven system with acoustic-visual fusion and anthropomorphic interactive expressions as described in claim 9, characterized in that: The anthropomorphic eye-tracking strategy library module has a built-in parameter storage unit and a dynamic coefficient mapping unit. The parameter storage unit is used to store the four-dimensional eye-tracking reference parameters corresponding to each interaction state. The dynamic coefficient mapping unit is connected to the output of the multimodal perception fusion module and is used to receive dynamic perception quality evaluation factors. And output offset magnitude correction coefficient This allows for real-time adjustment of the reference offset.

Images