Head orientation method for auditory-visual collaborative humanoid robot

Abstract

The invention provides a head orientation method for a humanoid robot with synergy of audition and vision, and relates to the technical field of robots, and the method comprises the steps: S1, collecting an environment sound signal through a microphone array distributed at the head of the robot, and estimating the initial orientation of a sound source; s2, target image information is obtained through a visual sensor installed on the head of the robot, and candidate target areas are extracted; s3, performing joint processing on the sound source orientation and the visual information based on a time sequence modeling auditory and visual fusion algorithm to obtain a target direction; s4, when the confidence coefficient of the fusion result is insufficient, controlling the head of the robot to execute active detection so as to enhance the positioning reliability; and S5, according to the target direction, a multi-degree-of-freedom execution mechanism of the robot head is driven to complete natural and stable steering. When the comprehensive confidence coefficient is insufficient, an active detection mechanism can be triggered to drive the head of the robot to carry out small-amplitude scanning so as to confirm the target direction, and therefore the problem that the direction is wrong due to long-time staring due to inaccurate positioning is avoided.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method for head orientation of a humanoid robot that combines auditory and visual coordination. Background Technology

[0002] As humanoid robots are increasingly used in intelligent customer service, companionship, care, education, and entertainment, the ability for robots to quickly and accurately orient themselves towards the interaction target is a crucial prerequisite for achieving natural communication. Currently, robot head orientation primarily relies on either auditory or visual perception.

[0003] On the one hand, auditory-based orientation methods typically use microphone arrays to estimate the location of sound sources through algorithms such as time difference estimation and generalized cross-correlation. These methods are simple to implement and computationally inexpensive, but they are prone to location estimation errors or tracking loss in reverberant environments, strong noise interference, or when multiple sound sources coexist. On the other hand, vision-based orientation methods generally use cameras to acquire scene images and combine them with face detection, human detection, or target recognition algorithms to determine the direction of possible interactive objects. These methods perform well in well-lit, unobstructed environments, but they cannot provide reliable location information in dim lighting, backlighting, occlusion, or when the target temporarily leaves the frame.

[0004] However, existing technologies generally suffer from the following shortcomings: most methods only perform static weighting and do not consider the dynamic nature of time series, resulting in unstable response when the target moves rapidly or the sound source orientation changes abruptly; existing microphone arrays or cameras lack biomimetic design, resulting in insufficient acoustic direction resolution and visual depth perception accuracy, making it difficult to support accurate 3D positioning; existing methods lack active detection and feedback mechanisms, and when the robot's positioning is uncertain, it often stares in the wrong direction or remains stationary for a long time, affecting the naturalness of human-computer interaction. Summary of the Invention

[0005] Based on the technical problems existing in the background art, the present invention proposes a method for head orientation of a humanoid robot with auditory and visual coordination.

[0006] A method for head orientation of a humanoid robot with auditory and visual coordination includes: S1. Collect ambient sound signals by using a microphone array distributed on the robot's head to estimate the initial location of the sound source; S2. Acquire target image information through a vision sensor installed on the robot's head and extract candidate target regions; S3. The audiovisual fusion algorithm based on temporal modeling jointly processes the sound source location and visual information to obtain the target direction; S4. When the confidence level of the fusion result is insufficient, control the robot head to perform active detection to enhance the positioning reliability. S5. Based on the target direction, drive the multi-degree-of-freedom actuators of the robot head to complete a natural and smooth turn.

[0007] Preferably, step S1 includes the following steps: S1-1. Install dual-channel microphone arrays on both sides of the robot's head, with the microphone spacing... The microphone is designed to simulate the distance between human ears of 18-22cm, with a sampling frequency of 44.1kHz and a quantization accuracy of 16bit. A bionic auricle structure with a height of 50-70mm and an outward flare of 15°-25° is installed on the outside of the microphone to enhance the vertical positioning characteristics of the 3-7kHz frequency band. S1-2, Simultaneously acquire the left channel signal and right channel signal And store it as a data buffer for a period of not less than 100ms. Indicates the time sampling point; S1-3. Preprocess the left and right channel signals separately: Pre-emphasis filtering:

[0008] in, : The filtered audio signal; : Original sound signal; Pre-emphasis coefficient = 0.95, used to enhance high-frequency components; : The signal from the previous sampling point; Frame division: 25ms per frame, frame shift 10ms; Adding windows: Hanming windows were adopted. ; in, Window function; : Index of the current frame sampling point; Total number of sampling points per frame; S1-4. Perform a Fast Fourier Transform on each frame of signal to obtain the frequency domain signal. and And calculate the cross power spectrum:

[0009] in, Cross-power spectrum; The spectrum of the left channel signal; : The spectrum of the right channel signal; The complex conjugate of the right channel signal; Frequency index; S1-5. Estimating the time difference of arrival using generalized cross-correlation-phase transform:

[0010] in, Delayed correlation function; Assume a delay amount; Cross-power spectrum; Cross-power spectral amplitude, used for normalization; Imaginary unit; Phase rotation factor; By searching The maximum peak value is used to obtain the arrival time difference between the left and right channel signals. ; S1-6, Azimuth Calculation

[0011] in, : The azimuth angle relative to the sound source directly in front of the robot; Speed ​​of sound, approximately 343 m / s; Time difference between arrival of left and right channel signals; Microphone spacing; when If the result is invalid, discard it as an invalid estimate; finally, output the initial azimuth angle of the sound source. This is for subsequent use in audiovisual fusion.

[0012] Preferably, step S2 includes the following steps: S2-1. Install an RGB-D camera in front of the robot's head, with a horizontal field of view of 90°~110°, a vertical field of view of 60°~70°, a frame rate of 30fps, and a resolution of 1280×720 pixels; the depth camera has a maximum range of 3m and a depth accuracy of ±2cm. S2-2. Real-time acquisition of color images and depth maps, denoising of color images, and hole filling and error correction of depth maps; S2-3. Input the processed color image into a convolutional neural network object detection model (such as YOLOv5s) to identify bounding boxes of possible sound source objects in the image and output candidate target regions and their confidence scores. S2-4. Combining the depth map, map the candidate target regions onto a three-dimensional coordinate system and calculate the spatial orientation vector of each candidate target. And thus obtain the target's azimuth angle. With pitch angle ; S2-5. Output candidate target information: Use the bounding box, confidence score and three-dimensional orientation vector of the candidate target as input to the subsequent audiovisual fusion module.

[0013] Preferably, step S3 includes the following steps: S3-1, Input sound source azimuth angle The sequence (past 1 second, 10ms per frame) is used as auditory input; the three-dimensional orientation vector of the candidate target is input. Sequences and their confidence levels As visual input; the sound source angle and visual direction vectors are normalized, and the values ​​are mapped to... interval; S3-2, Sequence of sound source azimuth angles The auditory feature vector is encoded through a one-dimensional convolutional network. ; visual direction vector sequence and its confidence level The input is encoded into a visual feature vector by a multilayer perceptron. ; S3-3, Construct a cross-modal Transformer model, and Simultaneously, the input is fused into the attention layer; The formula for calculating attention is:

[0014] in: These are the auditory and visual feature mapping matrices, respectively; For visual confidence The weighted mask, when Increase the weight of the auditory channel when necessary, otherwise increase the weight of the visual channel; output This is the fused integrated feature sequence; S3-4. Features of the fused sequence Predicting the target direction vector using a fully connected layer Convert 3D direction vectors into azimuth angles. and pitch angle ; S3-5, Combining input visual confidence Combine acoustic signal strength to calculate overall confidence level ;like If it exceeds a set threshold (e.g., 0.75), it will... The output direction is used as the final target direction; if it is below the threshold, active detection is triggered to enhance positioning reliability.

[0015] Preferably, during the S3 cross-modal fusion, the acoustic features are averaged in the time dimension to make their frame rate consistent with the visual frame rate.

[0016] Preferably, the fusion of auditory and visual features in S3-3 adopts a weighted linear superposition method, and its calculation formula is as follows:

[0017] in, Auditory characteristics As a visual feature, , The confidence level of the visual candidate target.

[0018] Preferably, the overall confidence level of the target direction output in S3-5 It is obtained by weighting visual confidence with acoustic signal-to-noise ratio, and the calculation formula is as follows:

[0019] in, These are the fusion weights for visual and auditory perception, respectively, satisfying... .

[0020] Preferably, when the overall confidence level When N consecutive frames are below the threshold of 0.75, an active detection step is triggered, causing the robot's head to perform a small scan to search for the target direction, where N ranges from 3 to 5.

[0021] The auditory-visual coordinated head orientation method for humanoid robots proposed in this invention has the following beneficial effects: (1) This invention adopts a cross-modal fusion algorithm based on temporal modeling, and realizes dynamic weighted fusion of auditory and visual features through the Transformer attention mechanism. Compared with the existing static weighting or simple feature splicing methods, it can better adapt to the changes of sound source and target in time, and improve the stability and accuracy of target direction estimation. Dynamic weight mask is introduced in the fusion process, and the weight is adaptively adjusted by combining visual confidence and acoustic signal-to-noise ratio, so that the robot can maintain high robustness in low light or noise interference environments.

[0022] (2) The acoustic perception part of this invention adopts a biomimetic auricle structure with an outward flare angle of 15° to 25°, which effectively enhances the vertical sound source resolution and improves the shortcomings of simple planar microphone arrays in pitch angle estimation. The visual perception part combines depth information with candidate target detection, supports three-dimensional direction calculation, and is not limited to two-dimensional planar recognition, thus improving the robot's positioning accuracy in three-dimensional space.

[0023] (3) When the overall confidence level is insufficient, the present invention will trigger an active detection mechanism to drive the robot head to perform a small scan to confirm the target direction, thereby avoiding the problem of staring at the wrong direction for a long time due to inaccurate positioning, and significantly improving the naturalness of human-computer interaction. Attached Figure Description

[0024] Figure 1This is a flowchart of a humanoid robot head orientation method based on auditory and visual coordination proposed in this invention. Detailed Implementation

[0025] Reference Figure 1 A method for head orientation of a humanoid robot with auditory and visual coordination, comprising: S1. Collect ambient sound signals by using a microphone array distributed on the robot's head to estimate the initial location of the sound source; S1 includes the following steps: S1-1. Install dual-channel microphone arrays on both sides of the robot's head, with the microphone spacing... The microphone is designed to simulate the distance between human ears of 18-22cm, with a sampling frequency of 44.1kHz and a quantization accuracy of 16bit. A bionic auricle structure with a height of 50-70mm and an outward flare of 15°-25° is installed on the outside of the microphone to enhance the vertical positioning characteristics of the 3-7kHz frequency band. S1-2, Simultaneously acquire the left channel signal and right channel signal And store it as a data buffer for a period of not less than 100ms. Indicates the time sampling point; S1-3. Preprocess the left and right channel signals separately: Pre-emphasis filtering:

[0026] in, : The filtered audio signal; : Original sound signal; Pre-emphasis coefficient = 0.95, used to enhance high-frequency components; : The signal from the previous sampling point; Frame division: 25ms per frame, frame shift 10ms; Adding windows: Hanming windows were adopted. ; in, Window function; : Index of the current frame sampling point; Total number of sampling points per frame; S1-4. Perform a Fast Fourier Transform on each frame of signal to obtain the frequency domain signal. and And calculate the cross power spectrum:

[0027] in, Cross-power spectrum; The spectrum of the left channel signal; : The spectrum of the right channel signal; The complex conjugate of the right channel signal; Frequency index; S1-5. Estimating the time difference of arrival using generalized cross-correlation-phase transform:

[0028] in, Delayed correlation function; Assume a delay amount; Cross-power spectrum; Cross-power spectral amplitude, used for normalization; Imaginary unit; Phase rotation factor; By searching The maximum peak value is used to obtain the arrival time difference between the left and right channel signals. ; S1-6, Azimuth Calculation

[0029] in, : The azimuth angle relative to the sound source directly in front of the robot; Speed ​​of sound, approximately 343 m / s; Time difference between arrival of left and right channel signals; Microphone spacing; when If the result is invalid, discard it as an invalid estimate; finally, output the initial azimuth angle of the sound source. This is for subsequent use in audiovisual fusion.

[0030] S2. Acquire target image information through a vision sensor installed on the robot's head and extract candidate target regions; S2 includes the following steps: S2-1. Install an RGB-D camera in front of the robot's head, with a horizontal field of view of 90°~110°, a vertical field of view of 60°~70°, a frame rate of 30fps, and a resolution of 1280×720 pixels; the depth camera has a maximum range of 3m and a depth accuracy of ±2cm. S2-2. Real-time acquisition of color images and depth maps, denoising of color images, and hole filling and error correction of depth maps; S2-3. Input the processed color image into a convolutional neural network object detection model (such as YOLOv5s) to identify bounding boxes of possible sound source objects in the image and output candidate target regions and their confidence scores. Step 1: Input and Preprocessing Input: Image frames (RGB, common resolution 640×640) captured by the robot's head camera.

[0031] Preprocessing: The image is scaled to a fixed size, normalized to [0,1], and then fed into the YOLOv5s network.

[0032] Step 2: Network Structure (YOLOv5s) YOLOv5s is a lightweight object detection model that consists of three parts: Backbone (CSPDarknet): Responsible for extracting multi-scale features.

[0033] Neck (FPN+PAN): Performs multi-scale feature fusion to enable the detection of small, medium, and large targets.

[0034] Head: Prediction is performed on feature maps at three different scales, with output for each grid point: Bounding box regression parameters (position + size) Category probability distribution (belonging to predefined categories such as "face / body / mouth / phone / speaker") Target confidence level Step 3: Generation of candidate target regions The YOLOv5s Head section predicts a large number of candidate boxes on feature maps at different scales. Each candidate box contains:

[0035] in: : Coordinates of the bounding box center point (relative to image normalized coordinates) : Bounding box width and height Target confidence (the probability that there is an object inside the bounding box) Category probability distribution (e.g., person, mouth, speaker, mobile phone, etc.) Step 4: Candidate box selection and confidence calculation Candidate sound source object category selection: During training, the category set is limited to objects that may produce sound (e.g., human face / mouth, speaker, television, mobile phone, etc.).

[0036] Formula for calculating target confidence:

[0037] in: = Probability of "whether there is a target" = Probability of belonging to the most likely sound source category Non-maximum suppression (NMS): Removes redundant candidate boxes with high overlap and retains only the bounding boxes with the highest confidence.

[0038] Step 5: Output Results Ultimately, YOLOv5s will output a set of candidate target regions:

[0039] That is: the position, size, category label, and corresponding confidence level of each bounding box.

[0040] These candidate target regions are then passed to the S3 audiovisual fusion step in the patented solution for joint modeling with the sound source azimuth sequence.

[0041] S2-4. Combining the depth map, map the candidate target regions onto a three-dimensional coordinate system and calculate the spatial orientation vector of each candidate target. And thus obtain the target's azimuth angle. With pitch angle ; S2-5. Output candidate target information: Use the bounding box, confidence score and three-dimensional orientation vector of the candidate target as input to the subsequent audiovisual fusion module.

[0042] S3. The audiovisual fusion algorithm based on temporal modeling jointly processes the sound source location and visual information to obtain the target direction; S3 includes the following steps: S3-1, Input sound source azimuth angle The sequence (past 1 second, 10ms per frame) is used as auditory input; the three-dimensional orientation vector of the candidate target is input. Sequences and their confidence levels As visual input; the sound source angle and visual direction vectors are normalized, and the values ​​are mapped to... interval; S3-2, Sequence of sound source azimuth angles The auditory feature vector is encoded through a one-dimensional convolutional network. ; visual direction vector sequence and its confidence level The input is encoded into a visual feature vector by a multilayer perceptron. ; S3-3, Construct a cross-modal Transformer model, and Simultaneously, the input is fused into the attention layer; The cross-modal Transformer model is trained using a multimodal training dataset, which includes audio and video data with sound source location annotations. The model input feature dimension is set to 128, the learning rate is 0.001, the batch size is 64, and the optimizer is Adam.

[0043] The formula for calculating attention is:

[0044] in: These are the auditory and visual feature mapping matrices, respectively; For visual confidence The weighted mask, when Increase the weight of the auditory channel when necessary, otherwise increase the weight of the visual channel; output This is the fused integrated feature sequence; S3-4. Features of the fused sequence Predicting the target direction vector using a fully connected layer Convert 3D direction vectors into azimuth angles. and pitch angle ; S3-5, Combining input visual confidence Combine acoustic signal strength to calculate overall confidence level ;like If it exceeds a set threshold (e.g., 0.75), it will... The output direction is used as the final target direction; if it is below the threshold, active detection is triggered to enhance positioning reliability.

[0045] S4. When the confidence level of the fusion result is insufficient, control the robot head to perform active detection to enhance the positioning reliability. S4-1 Active Detection Trigger: When the overall confidence level If the current location result is determined to be unreliable when the number of consecutive frames below a set threshold (e.g., 0.75, N=3~5), an active detection procedure is triggered.

[0046] S4-2 Scanning Strategy Planning: The robot control module sets the scanning range and step size: Horizontal scanning range ±30°~45°; Vertical angle can be selected from ±10° to 15°; Use a step size of 5° to 10° to ensure coverage while avoiding excessive mechanical movement.

[0047] S4-3 Perform head movements: The robot's neck servo motor is controlled to rotate sequentially within a set range and step size, and held at each angular position for 100–300 ms to ensure the stability of sound and image acquisition.

[0048] S4-4 Data Reacquisition and Fusion: During the scanning process, the microphone array and the visual sensor simultaneously acquire new audio and video data; For each scanning angle position, a temporary positioning result is calculated and then weighted and averaged with the result of the previous frame.

[0049] S4-5 Optimal Direction Determination: After the entire scanning cycle is completed, the set of azimuth angles with the highest overall confidence is selected as the final positioning result.

[0050] If the highest confidence level is still below the threshold, the scanning range is expanded or active detection is retried.

[0051] S4-6 Active Detection Termination and Feedback: When the positioning result reaches the set reliability ( (≥0.85), stop active detection and drive the robot head to turn towards the final target direction.

[0052] S5. Based on the target direction, drive the multi-degree-of-freedom actuators of the robot head to complete a natural and smooth turn.

[0053] The S3 cross-modal fusion process performs average pooling of acoustic features over time to make their frame rate consistent with the visual frame rate.

[0054] The fusion of auditory and visual features in S3-3 adopts a weighted linear superposition method, and its calculation formula is as follows:

[0055] in, Auditory characteristics As a visual feature, , The confidence level of the visual candidate target.

[0056] The overall confidence level of the target direction output in S3-5 It is obtained by weighting visual confidence with acoustic signal-to-noise ratio, and the calculation formula is as follows:

[0057] in, These are the fusion weights for visual and auditory perception, respectively, satisfying... .

[0058] When the overall confidence level When N consecutive frames are below the threshold of 0.75, an active detection step is triggered, causing the robot's head to perform a small scan to search for the target direction, where N ranges from 3 to 5.

[0059] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for head orientation of a humanoid robot with auditory and visual coordination, characterized in that, include: S1. Collect ambient sound signals by using a microphone array distributed on the robot's head to estimate the initial location of the sound source; S2. Acquire target image information through a vision sensor installed on the robot's head and extract candidate target regions; S3. The audiovisual fusion algorithm based on temporal modeling jointly processes the sound source location and visual information to obtain the target direction; S4. When the confidence level of the fusion result is insufficient, control the robot head to perform active detection to enhance the positioning reliability. S5. Based on the target direction, drive the multi-degree-of-freedom actuators of the robot head to complete a natural and smooth turn.

2. The method for head orientation of a humanoid robot with auditory and visual coordination according to claim 1, characterized in that, S1 includes the following steps: S1-1. Install dual-channel microphone arrays on both sides of the robot's head, with the microphone spacing... The microphone is designed to simulate the distance between human ears of 18-22cm, with a sampling frequency of 44.1kHz and a quantization accuracy of 16bit. A bionic auricle structure with a height of 50-70mm and an outward flare of 15°-25° is installed on the outside of the microphone to enhance the vertical positioning characteristics of the 3-7kHz frequency band. S1-2, Simultaneously acquire the left channel signal and right channel signal And store it as a data buffer for a period of not less than 100ms. Indicates the time sampling point; S1-3. Preprocess the left and right channel signals separately: Pre-emphasis filtering: ； in, : The filtered audio signal; : Original sound signal; Pre-emphasis coefficient = 0.95, used to enhance high-frequency components; : The signal from the previous sampling point; Frame division: 25ms per frame, frame shift 10ms; Adding windows: Hanming windows were adopted. ; in, Window function; : Index of the current frame sampling point; Total number of sampling points per frame; S1-4. Perform a Fast Fourier Transform on each frame of signal to obtain the frequency domain signal. and And calculate the cross power spectrum: ； in, Cross-power spectrum; The spectrum of the left channel signal; : The spectrum of the right channel signal; The complex conjugate of the right channel signal; Frequency index; S1-5. Estimating the time difference of arrival using generalized cross-correlation-phase transform: ； in, Delayed correlation function; Assume a delay amount; Cross-power spectrum; Cross-power spectral amplitude, used for normalization; Imaginary unit; Phase rotation factor; By searching The maximum peak value is used to obtain the arrival time difference between the left and right channel signals. ; S1-6, Azimuth Calculation ； in, : The azimuth angle relative to the sound source directly in front of the robot; Speed ​​of sound, approximately 343 m / s; Time difference between arrival of left and right channel signals; Microphone spacing; when If the result is invalid, discard it as an invalid estimate; finally, output the initial azimuth angle of the sound source. This is for subsequent use in audiovisual fusion.

3. The method for head orientation of a humanoid robot with auditory and visual coordination according to claim 2, characterized in that, S2 includes the following steps: S2-1. Install a camera in front of the robot's head with a horizontal field of view of 90° to 110°, a vertical field of view of 60° to 70°, a frame rate of 30fps, and a resolution of 1280×720 pixels; the depth camera has a maximum range of 3m and a depth accuracy of ±2cm. S2-2. Real-time acquisition of color images and depth maps, denoising of color images, and hole filling and error correction of depth maps; S2-3. Input the processed color image into the convolutional neural network target detection model to identify the bounding boxes of possible sound source objects in the image and output the candidate target regions and their confidence scores. S2-4. Combining the depth map, map the candidate target regions onto a three-dimensional coordinate system and calculate the spatial orientation vector of each candidate target. And thus obtain the target's azimuth angle. With pitch angle ; S2-5. Output candidate target information: Use the bounding box, confidence score and three-dimensional orientation vector of the candidate target as input to the subsequent audiovisual fusion module.

4. The method for head orientation of a humanoid robot with auditory and visual coordination according to claim 3, characterized in that, S3 includes the following steps: S3-1, Input sound source azimuth angle The sequence is used as auditory input; the three-dimensional direction vector of the input candidate target is used. Sequences and their confidence levels As visual input; the sound source angle and visual direction vectors are normalized, and the values ​​are mapped to... interval; S3-2, Sequence of sound source azimuth angles The auditory feature vector is encoded through a one-dimensional convolutional network. ; visual direction vector sequence and its confidence level The input is encoded into a visual feature vector by a multilayer perceptron. ; S3-3, Construct a cross-modal Transformer model, and Simultaneously, the input is fused into the attention layer; The formula for calculating attention is: ； in: These are the auditory and visual feature mapping matrices, respectively; For visual confidence The weighted mask, when Increase the weight of the auditory channel when necessary, otherwise increase the weight of the visual channel; output This is the fused integrated feature sequence; S3-4. Features of the fused sequence Predicting the target direction vector using a fully connected layer Convert 3D direction vectors into azimuth angles. and pitch angle ; S3-5, Combining input visual confidence Combine acoustic signal strength to calculate overall confidence level ;like If it exceeds the set threshold, The output direction is used as the final target direction; if it is below the threshold, active detection is triggered to enhance positioning reliability.

5. The method for head orientation of a humanoid robot with auditory and visual coordination according to claim 4, characterized in that, The S3 cross-modal fusion process performs average pooling of acoustic features over time to make their frame rate consistent with the visual frame rate.

6. The method for head orientation of a humanoid robot with auditory and visual coordination according to claim 4, characterized in that, The fusion of auditory and visual features in S3-3 adopts a weighted linear superposition method, and its calculation formula is as follows: ； in, Auditory characteristics As a visual feature, , The confidence level of the visual candidate target.

7. The method for head orientation of a humanoid robot with auditory and visual coordination according to claim 4, characterized in that, The overall confidence level of the target direction output in S3-5 It is obtained by weighting visual confidence with acoustic signal-to-noise ratio, and the calculation formula is as follows: ； in, These are the fusion weights for visual and auditory perception, respectively, satisfying... .

8. The method for head orientation of a humanoid robot with auditory and visual coordination according to claim 4, characterized in that, When the overall confidence level When N consecutive frames are below the threshold, an active detection step is triggered, causing the robot's head to perform a small scan to search for the target direction, where N ranges from 3 to 5.

Images