Robot tracking and positioning method and system based on audio-visual fusion

Abstract

The invention relates to the field of robot system control, and particularly discloses a robot tracking and positioning method and system based on audio-visual fusion, and the method comprises the following steps: synchronous collection of audio-visual data, multi-modal fusion judgment, setting of a visual data continuity judgment threshold value, and setting of a visual data continuity judgment threshold value. When the visual detection equipment continuously detects a face target with confidence greater than a set value in multiple frames, the system preferentially enters a visual tracking mode; when the interruption time of the visual data exceeds the set time, the system is automatically switched to a sound source positioning mode; and driving element coordination control: according to the obtained image offset or the obtained target sound source angle change, the driving element performs angle coordination adjustment on the head of the robot at the horizontal and longitudinal angles. According to the method, the limitation of a single mode is overcome, smooth mode switching is realized, the positioning precision is improved, and motion control is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of system control, in particular to a robot control system. BACKGROUND

[0002] The existing head tracking positioning system of the robot is mostly based on single modal input, such as using only visual information or only using auditory information for target positioning. However, in a complex environment, single modal has significant limitations. For example, when the visual sensor field of view is blocked, the light is insufficient, or the image processing is delayed, the robot will not be able to accurately lock the target; while relying solely on hearing, the sound source positioning accuracy may be disturbed by noise, resulting in a large deviation in the rotation of the driving element, and this instability makes the robot lack of fluency and continuity when interacting with humans.

[0003] Although there have been attempts to fuse visual and auditory information for tracking, the existing solutions are mostly loosely coupled designs, with low coordination between modules, large mode switching delay, and lack of smooth transition strategies for failed modalities, resulting in action stalls or repeated seeking phenomena during multi-modal switching of the robot, thereby affecting the interactive experience.

[0004] The technical problem to be solved by the present application is how to improve the coordination of visual and auditory tracking of the robot. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a robot tracking positioning method and system based on audio-visual fusion.

[0006] The technical solution adopted by the present application is: a robot tracking positioning method based on audio-visual fusion, comprising the following steps:

[0007] S1, audio-visual data synchronous acquisition, acquiring sound source angle information through a plurality of audio input devices, and simultaneously performing face detection through a visual detection device to acquire the position and confidence information of the target in the image;

[0008] S2, multi-modal fusion judgment, setting a visual data continuity judgment threshold, when the visual detection device continuously detects a face target with a confidence greater than a set value for multiple frames, the system preferentially enters a visual tracking mode; when the visual data is interrupted for more than a set time, the system automatically switches to a sound source positioning mode;

[0009] S31, in the visual tracking mode, calculating the image offset of the face center relative to the image center, and adjusting the head angle of the robot;

[0010] S32, in the sound source positioning mode, the sound source information is obtained by the multi-channel audio input device, the sound source angle is calculated and mapped to the 180° working range of the driving element, and the driving element is triggered to rotate when the target sound source angle changes by more than a set angle;

[0011] S4, driving element coordination control: according to the image offset obtained in S31 or the target sound source angle change obtained in S32, the driving element adjusts the angle of the robot head in the horizontal and vertical directions.

[0012] In some embodiments, the multi-channel audio input device is arranged in an array, including one audio input device arranged at the center of the robot head and six audio input devices uniformly arranged at the outer edge of the robot head.

[0013] In some embodiments, in the S2 step, a vision data counter vision_data_count is set, when a face is detected in a continuous image frame with a confidence greater than 0.5, a vision data ready flag vision_data_ready is set to true, and the system switches to the visual tracking mode.

[0014] In some embodiments, in the S31 step, the image offset of the face center relative to the image center is calculated as follows:

[0015] When a face is detected, the face center coordinates are calculated as follows:

[0016] Face center X coordinate: face_cx = x1 + box_w / 2;

[0017] Face center Y coordinate: face_cy = y1 + box_h / 2;

[0018] Where x1, y1 are the coordinates of the top left corner of the detection box, and box_w, box_h are the width and height of the detection box.

[0019] Calculate the image offset of the face relative to the image center:

[0020] X-axis offset: offset_x = face_cx - center_x;

[0021] Y-axis offset: offset_y = face_cy - center_y;

[0022] Where center_x = w / 2, center_y = h / 2;

[0023] w, h are the image width and height respectively.

[0024] In some embodiments, in the step S32, the sound source distribution image imga is acquired, and the mic.get_dir(imga) is called to calculate the sound source direction intensity array b,

[0025] For each element b i in the array b i ≥ 2, the sound source angle calculation is involved:

[0026] X coordinate component: AngleX = b i × sin(i × π / 6);

[0027] Y coordinate component: AngleY = b i × sin(i × π / 6);

[0028] Wherein, i is the audio input device array index, and the value range is 0-5.

[0029] The sound source angle is calculated according to the X and Y coordinate components, specifically:

[0030] When AngleY < 0, AngleAddPi = 180;

[0031] When AngleX < 0 and AngleY > 0, AngleAddPi = 360;

[0032] When AngleY = 0, Angle = 90 (AngleX > 0) or 270 (AngleX < 0);

[0033] In other cases, Angle = AngleAddPi + round(atan(AngleX / AngleY) × 180 / π, 4);

[0034] The sound source intensity is calculated: .

[0035] In some embodiments, based on one of the above methods, the system includes a sound source positioning module, a visual detection module, a rudder control module and a communication interface module, and the sound source positioning module, the visual detection module and the rudder control module are connected through the communication interface module.

[0036] In some embodiments, the sound source positioning module includes a controller and a microphone array, the visual detection module includes a client and a camera, and the rudder control module includes a rudder controller and a three-axis rudder assembly.

[0037] In some embodiments, the system includes an exception handling measure, and the exception handling measure includes hardware fault detection, data validity verification and algorithm robustness design.

[0038] In some embodiments, the data validity verification includes:

[0039] Sound source angle data range check: 0° ≤ angle ≤ 360°, data outside the range is discarded;

[0040] Visual detection confidence filtering: detection results with confidence < 0.5 do not participate in control.

[0041] The beneficial effects of the present application are:

[0042] The robot tracking and positioning method and system based on audio-visual fusion achieve the following technical effects:

[0043] (1) Overcome the limitations of single modality: through the organic fusion of visual detection and sound source positioning, improve the robustness and continuity of target tracking;

[0044] (2) Realize smooth mode switching: establish a visual priority adaptive switching mechanism, seamlessly switch to sound source positioning mode when vision fails, avoid tracking interruption;

[0045] (3) Improve positioning accuracy: combine the accuracy of face detection and the omnidirectionality of sound source positioning to achieve high-precision target positioning;

[0046] (4) Optimize motion control: use PID control and sinusoidal acceleration algorithm to realize smooth and accurate control of the rudder, and improve the naturalness of interaction. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a method flowchart of the present application;

[0048] Figure 2 is a system structure schematic diagram of Example 3. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0050] The present application provides various technical solutions:

[0051] Example 1

[0052] Please refer to Figure 1 A robot tracking and positioning method based on audio-visual fusion, comprising the following steps:

[0053] S1, audio-visual data synchronous acquisition, obtaining sound source angle information through multi-channel audio input device, and simultaneously obtaining the position and confidence information of the target in the image through visual detection device; wherein the multi-channel audio input device realizes array distribution.

[0054] S2, multi-modal fusion judgment, setting a visual data continuity judgment threshold, when the visual detection device detects a face target with a confidence greater than 0.5 for multiple consecutive frames, the system preferentially enters a visual tracking mode; when the visual data is interrupted for more than a set time, the system automatically switches to a sound source positioning mode;

[0055] S31, in the visual tracking mode, calculating the image offset of the face center relative to the image center, and adjusting the robot head angle;

[0056] S32, in the sound source positioning mode, obtaining sound source information through the multi-channel audio input device to calculate the sound source angle mapping to the 180° working range of the driving element, and triggering the driving element to rotate when the target sound source angle change is greater than a set angle;

[0057] S4, driving element coordinated control: according to the image offset obtained in S31 or the target sound source angle change obtained in S32, the driving element adjusts the angle of the robot head in the horizontal and vertical directions.

[0058] In the visual detection process, the video stream is collected through the visual detection device, and the MediaPipe library is used for face detection.

[0059] Specifically, the BGR image is converted to RGB format, the model parameter model_selection=0 is set, and the minimum detection confidence min_detection_confidence=0.5.

[0060] When a face is detected, the face center coordinates are calculated:

[0061] Face center X coordinate: face_cx=x1+box_w / 2;

[0062] Face center Y coordinate: face_cy=y1+box_h / 2;

[0063] Wherein, x1, y1 are the coordinates of the top left corner of the detection box, and box_w, box_h are the width and height of the detection box.

[0064] Calculate the image offset of the face relative to the image center:

[0065] X-axis offset: offset_x=face_cx-center_x;

[0066] Y-axis offset: offset_y = face_cy - center_y;

[0067] where center_x = w / 2, center_y = h / 2;

[0068] w, h are image width and height respectively.

[0069] In the sound source positioning process, the multi-channel audio input device realizes array distribution, including 1 audio input device arranged at the center of the robot head and 6 audio input devices uniformly distributed at the outer edge of the robot head, wherein the 6 audio input devices at the periphery are distributed in a circular ring and two devices are spaced apart by 60°.

[0070] Specifically, the mic.get_map() function is called to obtain the sound source distribution image imga, the image is scaled to 160x160 pixels to obtain imgb, and the mic.get_dir(imga) function is called to calculate the sound source direction intensity array b.

[0071] For each element b of array b i When b i ≥ 2, participate in sound source angle calculation:

[0072] X coordinate component: AngleX = b i ×sin(i×π / 6);

[0073] Y coordinate component: AngleY = b i ×sin(i×π / 6);

[0074] where i is the array index of the audio input device, and the value range is 0-5.

[0075] According to the X and Y coordinate components, the sound source angle is calculated, specifically:

[0076] When AngleY < 0, AngleAddPi = 180;

[0077] When AngleX < 0 and AngleY > 0, AngleAddPi = 360;

[0078] When AngleY = 0, Angle = 90 (AngleX > 0) or 270 (AngleX < 0);

[0079] In other cases, Angle = AngleAddPi + round(atan(AngleX / AngleY) × 180 / π,4);

[0080] Calculate the sound source intensity: .

[0081] Pan axis horizontal rotation and Tilt axis vertical pitching are realized by a three-axis control system, and the angle of the driving element is accurately controlled by pulse width modulation, with an angle accuracy of ±1°.

[0082] A multi-modal fusion judgment strategy is provided, specifically, a vision data counter vision_data_count is set, when a face with a confidence greater than 0.5 is detected in a continuous image frame, a vision data ready flag vision_data_ready is set to true, and the system switches to a vision tracking mode.

[0083] In the vision tracking mode:

[0084] The current time is recorded as last_vision_data_time;

[0085] When no valid face is detected for more than 1000 milliseconds, the driving element drives the robot head to return to a 45° position, and the system switches to a sound source positioning mode;

[0086] In the sound source positioning mode:

[0087] The sound source angle data is received;

[0088] The angle difference is calculated: angle_diff=|servo_angle-current_angle|;

[0089] When angle_diff>15°, the driving element is triggered to rotate.

[0090] Driving element coordination control is realized by a three-axis control system, including:

[0091] No. 1 driving element: Pan axis, angle range 0°-180°, zero position 45°;

[0092] No. 2 driving element: Tilt axis, angle range 60°-120°, zero position 90°;

[0093] No. 3 driving element: reverse Tilt axis, angle range 60°-120°, zero position 90°;

[0094] Angle to pulse width conversion formula:

[0095] Forward driving element: PWM=500+(angle / 180)×2000 microseconds;

[0096] Reverse driving element: PWM=2500-(angle / 180)×2000 microseconds;

[0097] PID control algorithm is adopted, and the parameters are set as:

[0098] X-axis: Kp=0.03, Ki=0.0, Kd=0.001, output range ±15°;

[0099] Y-axis: Kp=0.04, Ki=0.0, Kd=0.001, output range ±8°;

[0100] PID calculation formula:

[0101] Integral term: integral = error × dt;

[0102] Derivative term: derivative = (error - prev_error) / dt;

[0103] Output: output = Kp × error + Ki × integral + Kd × derivative;

[0104] Dead zone processing is applied, and the dead zone range is 0.4 times the face frame width, and the error is zero in the dead zone;

[0105] Smooth control algorithm is adopted, and the smoothing factor α = 0.6: new_angle = α × target_angle + (1-α) × current_angle.

[0106] Example 2

[0107] On the basis of example 1, the sine wave acceleration rotation algorithm is optimized, and in the sound source positioning mode, when the detected angle difference is greater than 15°, the sine wave acceleration curve is used to realize smooth rotation:

[0108] Calculate the number of rotation steps: total_steps = |target_angle - start_angle|;

[0109] Determine the rotation direction: step = 1 (target angle is greater than the starting angle) or -1 (target angle is less than the starting angle);

[0110] Loop to execute rotation (i from 0 to total_steps-1), where i is the current step:

[0111] 1) Calculate the progress ratio: progress = i / total_steps;

[0112] 2) Calculate the speed factor: speed_factor = sin(progress × π);

[0113] 3) Calculate dynamic delay: dynamic_delay = base_delay / (0.5 + 0.5x speed_factor);

[0114] 4) Limit delay range: 3ms≤dynamic_delay≤base_delayx3;

[0115] 5) Perform single-step rotation: current_angle = start_angle + i x step;

[0116] 6) Delay wait: delay(dynamic_delay)

[0117] Wherein, base_delay is set to 25 milliseconds, and a smooth motion curve is realized by a sine wave function to avoid sudden changes and oscillations in the movement of the steering engine.

[0118] Compared with the traditional uniform rotation, the algorithm can reduce mechanical impact, improve positioning accuracy, and reduce system noise.

[0119] Example 3

[0120] Based on the method of example 1, please refer to Figure 2 An audio-visual fusion-based robot tracking and positioning system includes a sound source positioning module, a visual detection module, a steering engine control module, and a communication interface module. The sound source positioning module includes a controller and a microphone array, the visual detection module includes a client and a camera, and the steering engine control module includes a steering engine controller and a three-axis steering engine assembly. The sound source positioning module, the visual detection module, and the steering engine control module are connected through the communication interface module.

[0121] In this embodiment, the controller uses a K210 AI development board, the audio input device uses a microphone, and the driving element uses a steering engine. Other devices that can achieve the same function as the audio input device or driving element are also within the scope of protection of this application.

[0122] The microphone array includes 7 digital MEMS microphones with a sampling rate of 16 kHz and a dynamic range of ≥60dB, and is connected to the controller.

[0123] The camera has a resolution of ≥640x480 pixels, a frame rate of ≥30fps, and a viewing angle of ≥60°. The client runs a visual processing program based on Python3.

[0124] The steering engine controller provides ≥3 PWM outputs with a pulse width accuracy of 1 microsecond and is connected to the client. The three-axis steering engine assembly includes:

[0125] Pan axis servo: standard 180° servo, torque ≥ 3 kg·cm, accuracy ± 1°

[0126] Tilt axis servo group: two servos form differential pitch control, working angle 60°-120°, synchronization accuracy ± 0.5°;

[0127] Communication interface module, realize data transmission between modules. The controller is in communication connection with the client. The client is in communication connection with the servo controller.

[0128] The working process of the system in this embodiment is:

[0129] (1) Initialization stage: the client scans the serial port device, establishes communication connection with the controller and the servo controller, opens the camera, and the servo is centered to zero position.

[0130] (2) Data acquisition stage: the microphone array continuously acquires environmental sound, and the controller calculates the sound source angle; the camera acquires video stream, and the client performs face detection.

[0131] (3) Fusion control stage: the client switches modes and controls the servo according to the visual detection result and the sound source positioning data according to the visual priority strategy.

[0132] (4) Execution stage: the servo controller receives the control instruction, and drives the three-axis servo assembly to complete the target tracking action.

[0133] Further, the sound source positioning program run by the controller includes:

[0134] Microphone array driving module: read 7-way microphone data;

[0135] Signal processing module: realize FFT transform and sound source direction calculation algorithm;

[0136] Communication control module: process client control instruction and send angle data;

[0137] State indication module: display sound source direction and system state through LED;

[0138] The fusion control program run by the client adopts a multi-thread architecture:

[0139] Main thread: visual processing and user interface display;

[0140] Audio data thread: receive and process K210 sound source data;

[0141] Control task thread: realize fusion control logic and servo drive;

[0142] Further, the embodiment also has the following technical parameters: sound source positioning accuracy: ±5°; visual tracking accuracy: ±2°; system response delay: <200ms; mode switching time: <100ms; servo positioning accuracy: ±1°.

[0143] Embodiment 4

[0144] This embodiment optimizes the exception handling and fault tolerance mechanism based on embodiment three.

[0145] The system includes the following exception handling measures:

[0146] (1) Hardware fault detection:

[0147] When the serial communication is abnormal, the automatic reconnection mechanism is used, and the retry times do not exceed 3 times;

[0148] When the camera is disconnected, the user is prompted to reconnect and reinitialize;

[0149] When the servo controller does not respond, check the connection state and reestablish communication;

[0150] (2) Data validity verification:

[0151] Sound source angle data range check: 0° ≤ angle ≤ 360°, data outside the range is discarded;

[0152] Visual detection confidence filtering: detection results with confidence <0.5 do not participate in control;

[0153] Servo angle out-of-range protection: automatically limited within the set range;

[0154] (3) Algorithm robustness design:

[0155] Continuous abnormal data counter, restart the corresponding module after more than 10 times of abnormality;

[0156] Control command legality check to avoid sending incorrect pulse width parameters;

[0157] System state monitoring, regularly check the running state of each module;

[0158] Through the above fault tolerance mechanism, the system can run stably under various abnormal conditions, improving the reliability and practicality of the product.

[0159] Finally, it should be noted that the above only describes the preferred examples of the present application, and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that modifications can be made to the technical solutions described in the foregoing embodiments, or some of the technical features thereof can be replaced equivalently, without departing from the spirit and principle of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A robot tracking and positioning method based on audio-visual fusion, characterized by, The method comprises the following steps: S1, audio-visual data synchronous acquisition, obtaining sound source angle information through a plurality of audio input devices, and simultaneously performing face detection through a visual detection device to obtain the position and confidence information of the target in the image; S2, multi-modal fusion judgment, setting a visual data continuity judgment threshold, when the visual detection device continuously detects a face target with a confidence greater than a set value for multiple frames, the system preferentially enters a visual tracking mode; when the visual data interruption exceeds a set time, the system automatically switches to a sound source positioning mode; S31, in the visual tracking mode, calculating the image offset of the face center relative to the image center, and adjusting the robot head angle; S32, in the sound source positioning mode, obtaining sound source information through a plurality of audio input devices to calculate the sound source angle mapping to the 180° working range of the driving element, and triggering the driving element to rotate when the target sound source angle change is greater than a set angle; S4, driving element coordinated control: according to the image offset obtained in S31 or the target sound source angle change obtained in S32, the driving element adjusts the angle of the robot head in the horizontal and vertical directions.

2. The robot tracking and positioning method based on audio-visual fusion according to claim 1, characterized in that, The plurality of audio input devices are arranged in an array, including one audio input device arranged at the center of the robot head and six audio input devices uniformly arranged at the outer edge of the robot head. 3.The robot tracking and positioning method based on audio-visual fusion of claim 1, wherein, In the S2 step, a visual data counter vision_data_count is set, when a face with a confidence greater than 0.5 is detected in a continuous image frame, a visual data ready flag vision_data_ready is set to true, and the system switches to the visual tracking mode.

4. The robot tracking and positioning method based on audio-visual fusion according to claim 1, characterized in that, In the S31 step, the image offset of the face center relative to the image center is calculated as follows: When a face is detected, the face center coordinates are calculated as follows: Face center X coordinate: face_cx = x1 + box_w / 2; Face center Y coordinate: face_cy = y1 + box_h / 2; Wherein, x1, y1 are the coordinates of the top left corner of the detection box, and box_w, box_h are the width and height of the detection box. The image offset of the face relative to the image center is calculated as follows: X-axis offset: offset_x = face_cx - center_x; Y-axis offset: offset_y = face_cy - center_y; Wherein, center_x = w / 2, center_y = h / 2; w and h are the image width and height, respectively.

5. The robot tracking and positioning method based on audio-visual fusion according to claim 1, characterized in that, In the S32 step, the sound source distribution image imga is obtained, and the mic.get_dir(imga) is called to calculate the sound source direction intensity array b, For each element b in array b i When b i ≥ 2 is involved in the sound source angle calculation: X coordinate component: AngleX = b i x sin(i x pi / 6); Y coordinate component: AngleY = b i x sin(i x pi / 6); Wherein, i is the array index of the audio input device, and the value range is 0-5; The sound source angle is calculated according to the X and Y coordinate components, specifically: When AngleY < 0, AngleAddPi = 180; When AngleX < 0 and AngleY > 0, AngleAddPi = 360; When AngleY = 0, Angle = 90 (AngleX > 0) or 270 (AngleX < 0); Otherwise, Angle = AngleAddPi + round(atan(AngleX / AngleY) × 180 / π, 4); Computing the sound source intensity: .

6. An audio-visual fusion based robot tracking and positioning system based on the audio-visual fusion based robot tracking and positioning method of any one of claims 1-5, characterized in that, The system comprises a sound source positioning module, a visual detection module, a steering engine control module and a communication interface module, and the sound source positioning module, the visual detection module and the steering engine control module are connected in communication through the communication interface module.

7. The audio-visual fusion based robot tracking and positioning system according to claim 6, wherein, The sound source positioning module comprises a controller and a microphone array, the visual detection module comprises a client and a camera, and the steering engine control module comprises a steering engine controller and a three-axis steering engine assembly.

8. The audio-visual fusion based robot tracking and positioning system of claim 6, wherein, The system comprises an exception handling measure, and the exception handling measure comprises hardware fault detection, data validity verification and algorithm robustness design.

9. The audio-visual fusion based robot tracking and positioning system of claim 8, wherein, The data validity verification comprises: Sound source angle data range check: 0° ≤ angle ≤ 360°, data beyond the range is discarded; Visual detection confidence filtering: detection results with confidence < 0.5 do not participate in control.

Images