A speaker tracking method based on audio-visual dual modality

Abstract

This invention discloses a speaker tracking method based on audio-visual dual-modal fusion. The method includes: acquiring speaker location information through a camera and a microphone array; fusing audio and image information to locate the speaker based on the number of people in the image and the presence of voice input; deriving different motion control instructions based on the different positioning results, planning a tracking path in real time, and driving the motion control system to track the speaker. In cases where the visual module fails to detect a person, a method is provided to actively search for the speaker using voice positioning information, making the system more intelligent. Ultimately, the invention enables real-time speaker tracking by a mobile robot, facilitating human-machine interaction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition and speech recognition, and in particular relates to a speaker tracking method based on visual and audio dual modalities. Background Art

[0002] With the rapid development of computer technology and mobile smart terminals, intelligent service robots are increasingly demanding more efficient human-machine interactions. Locating, tracking, and actively interacting with interaction objects are key to the development of service robots. Accurate and real-time tracking of people is a crucial challenge for the human-machine interaction systems of intelligent service robots. By accurately tracking people, intelligent service robots can provide a variety of services based on user needs. Furthermore, the interaction system can obtain more accurate information about user needs, enabling more precise and responsive service.

[0003] Existing tracking methods and systems mostly use voice positioning information. In more complex and noisy environments, voice positioning information is difficult to achieve accurate positioning, resulting in inaccurate tracking of intelligent robots, greatly limiting the fluency and usability of human-computer interaction technology. Summary of the Invention

[0004] In view of this, the present invention proposes a speaker tracking method based on audio-visual dual modality, comprising the following steps:

[0005] S1, obtain image and voice information through the camera and microphone array respectively;

[0006] S2: Fusing sound and image information, localizing the speaker based on the number of people in the image and whether there is voice input.

[0007] S3. Plan the tracking path in real time based on the positioning results and drive the motion control system to track.

[0008] Furthermore, step S2 is specifically as follows:

[0009] S21. Determine a person to be tracked based on the acquired image and voice information;

[0010] S22. Use the rectangular frame of the human body recognition model to lock the person, and return the pixel width and center coordinates of the rectangular frame through the model;

[0011] S23. Substitute the pixel width and center coordinates of the rectangular frame returned by the model and the known pixel width and center coordinates of the image into the following distance calculation model and azimuth calculation model to obtain the distance and azimuth between the robot's camera and the person:

[0012]

[0013] Where d is the distance between the human body and the camera, x is the number of pixels that the object occupies the width of the screen, and k is a constant value;

[0014]

[0015] Where, χ is the angle between the line connecting the optical center of the human body and the camera and the optical axis of the camera, β is the opening angle of the camera, It is the ratio of the distance from the center of the identification frame to the edges of the two sides of the picture in the pixel coordinate system.

[0016] Furthermore, step S21 is specifically as follows:

[0017] When the camera cannot detect a person, the microphone array captures sound information and outputs the azimuth angle, and the robot is controlled to move according to the azimuth angle;

[0018] When the camera detects multiple people and detects someone talking, it calculates the distance between the robot and each speaker and locks on the speaker closest to it;

[0019] When the camera detects multiple people and no one is talking, it locks on the person closest to it;

[0020] When the camera detects a single person, it locks on that person.

[0021] Furthermore, the robot uses a motion control system of a four-wheel mobile platform to achieve movement of the robot in different directions through the movement of the four-wheel mobile platform.

[0022] Furthermore, when the camera cannot detect a person, the microphone array captures sound information and outputs the azimuth angle. The robot movement is controlled according to the azimuth angle as follows:

[0023] If the microphone array outputs an azimuth angle of 0° to 90°, the four-wheeled mobile platform maintains state s1-, state s2-, at the set speed and acceleration. After maintaining this state for one second, the states s1 and s2 are reversed, maintained for one second, and reversed again, and this cycle repeats until the camera detects a speaker.

[0024] If the microphone array output azimuth angle is 90° to 180°, the four-wheeled mobile platform maintains state s1+ and state s2+ at the set speed and acceleration for one second, then reverses states s1 and s2 for one second, and reverses again, repeating this process until the camera detects a speaker.

[0025] If the microphone array output azimuth angle is between -180° and -90°, the four-wheeled mobile platform maintains state s1+ and state s2- at the set speed and acceleration for one second, then reverses states s1 and s2 for one second, and reverses again, repeating this process until the camera detects a speaker.

[0026] If the microphone array outputs an azimuth angle between -90° and 0°, the four-wheeled mobile platform maintains state s1- and state s2+ at the set speed and acceleration for one second, then reverses states s1 and s2 for one second, and reverses again, repeating this process until the camera detects a speaker.

[0027] Among them, the above coordinate system is established with the robot as the coordinate origin. The camera's forward direction is the robot's forward direction. Looking down at the robot, the camera's forward direction rotates counterclockwise for one circle, and the corresponding microphone array output azimuth angle changes from 0° to 180°, and then from -180° to 0°; s1+ means the four-wheel mobile platform moves forward, s1- means the four-wheel mobile platform moves backward, s2+ means the four-wheel mobile platform turns left at a set angle, and s2- means the four-wheel mobile platform turns right at a set angle.

[0028] Furthermore, when the camera locks on a person,

[0029] if The motion state is s1+, if Then the motion state is s1-, otherwise it stops;

[0030] if The motion state is s2+, if The motion state is s2-, otherwise it stops;

[0031] Among them, d m is the measurement value of the tracking distance, χ m is the tracking azimuth measurement value, d is the target value of the tracking distance, χ is the target value of the tracking azimuth, when the tracking distance measurement value falls within the interval The tracking distance measurement value reaches the ideal value. When the tracking azimuth angle measurement value falls within the interval The measurement value of the tracking azimuth angle is considered to have reached the ideal value.

[0032] The beneficial effects brought about by the technical solution provided by the present invention are:

[0033] This invention proposes a speaker tracking method based on audio-visual dual-modal fusion. A circular microphone array and camera capture both the speaker's voice and image information. The system then fuses these two positioning information, deriving different motion control instructions based on the different positioning results. Furthermore, if the visual module fails to detect a person, a method is provided to proactively locate the speaker using voice positioning information, making the system more intelligent. Ultimately, this invention enables real-time speaker tracking by a mobile robot, facilitating human-machine interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1This is a flow chart of a speaker tracking method based on audio-visual dual modality of the present invention;

[0035] Figure 2 This is an autonomous path planning strategy diagram of a speaker tracking method based on audio-visual dual modality of the present invention;

[0036] Figure 3 The distance calculation model between the object and the robot according to the embodiment of the present invention;

[0037] Figure 4 The object and robot azimuth angle calculation model of the embodiment of the present invention;

[0038] Figure 5 Schematic diagram of the robot tracking motion state according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0040] Traditional speaker tracking methods and systems mostly use only single-modal positioning information. Because single-modal information cannot fully reflect environmental information, the tracking system suffers from low sensitivity, poor stability and real-time performance, and is prone to tracking blind spots. Humans are complex targets, making single-modal tracking methods even less effective. Therefore, to achieve accurate and real-time target tracking, embodiments of the present invention propose a multimodal fusion speaker recognition and tracking method that leverages information redundancy to compensate for single-modal blind spots, improving accuracy and stability. A mobile robot is designed, using an intelligent vehicle as its motion system. A support platform is constructed on top of it, and a microphone array and camera are installed. The microphone array uses the time difference of arrival (TDOA) to obtain speaker azimuth information within a range of 0 to 360°. The camera uses captured images to obtain human azimuth and distance information within a smaller range (approximately ±27°). When the speaker is out of view, the microphone array's wide coverage is utilized to control the vehicle to search for the speaker. When the speaker is within view, the vehicle is controlled to track the speaker based on the visual positioning results. In addition, a multi-target screening strategy is implemented through speech word recognition, ultimately achieving accurate real-time tracking of the speaker.

[0041] The following terms appearing in this embodiment are explained:

[0042] Speaker voice: refers to the sound produced by the user or user in human-computer interaction, including speaking voice and other sounds containing information expressed by the user or user.

[0043] Speaker image: refers to the physical features of the user or user in human-computer interaction. In this system, the facial features of the user or user are the main focus.

[0044] Speaker localization refers to the real-time calculation of the speaker's position relative to a reference point during human-computer interaction, based on sensor information. In this context, the reference point is the physical center of the tracking system. Localization is expressed as the angle between the line connecting the speaker and the positioning system and the system's current positive direction, as well as the distance from the speaker to the system.

[0045] Speaker tracking: In the present invention, it refers to the system performing real-time path planning and tracking based on the speaker's location information and the set motion logic control system to keep the relative distance and angle between the system and the speaker at a certain value.

[0046] Microphone array: A system consisting of a certain number of acoustic sensors arranged according to certain rules and used to sample the spatial characteristics of the sound field.

[0047] Coordinate transformation: Coordinate transformation is the description of the position of a spatial entity. It is the process of transforming from one coordinate system to another. It is achieved by establishing a one-to-one correspondence between the two coordinate systems.

[0048] Information fusion: In this context, it refers to extracting different types of information from different sensors and normalizing them through a fusion algorithm to obtain the desired information. Information fusion can mitigate or avoid errors caused by a single information source, to some extent overcoming the system's detection blind spots.

[0049] This embodiment provides a speaker tracking method based on audio-visual dual modality. Figure 1 , specifically including the following steps:

[0050] S1. Obtain image and voice information through the camera and microphone array respectively.

[0051] S2. Fusion of speech and image information to locate the speaker based on the number of people in the image and whether there is speech input.

[0052] S21. Determine a person to be tracked based on the acquired sound and image information;

[0053] ① When the camera cannot detect a person, the microphone array captures sound information and outputs the azimuth, and controls the robot's movement based on the azimuth;

[0054] ② When the camera detects multiple people and detects someone talking, the robot calculates the distance between each speaker and locks on the closest speaker; when the camera detects multiple people and detects no one talking, it locks on the closest person;

[0055] ③When the camera detects a single person, it locks on that person;

[0056] S22. Use the rectangular frame of the human body recognition model to lock the person, and return the pixel width and center coordinates of the rectangular frame through the model;

[0057] S23. Substitute the pixel width and center coordinates of the rectangular frame returned by the model and the known pixel width and center coordinates of the image into the following distance calculation model and azimuth calculation model to derive the distance and azimuth between the robot's camera and the person (if sound information is also obtained, the distance and azimuth obtained from the image information are combined with the distance and azimuth obtained from the sound information):

[0058] refer to Figure 3 , L1 and L2 are the object length and the image length, respectively. O is the optical center of the camera. α is the plane passing through the optical center and perpendicular to the optical axis. d and f are the distance from the object to plane α and the focal length, respectively.

[0059] According to the properties of similar triangles, we have the following relationship:

[0060]

[0061] It can be seen that when f and L1 are fixed values, d and L2 are inversely proportional. f can be obtained based on camera parameters, and L1 in this embodiment is the width of the human body and can be regarded as a fixed value.

[0062] Let x be the number of pixels that the object occupies in the width of the screen. In this embodiment, it is the number of pixels that the human body detection mark frame occupies in the width of the screen. Replace L2 with xx, and the above formula can be transformed into:

[0063]

[0064] Where d is the distance from the person to the camera, x is the number of pixels of the object occupying the width of the screen, and k is a fixed value. The value of k can be obtained by measuring multiple different target points, recording the values ​​of d and x each time, and finally taking the average to obtain the value of k:

[0065]

[0066] Therefore, the distance from the human body to the plane α (ie, the distance from the human body to the camera) can be calculated according to the proportion of the human body in the picture.

[0067] refer to Figure 4 , Figure 4 This is a model for calculating the azimuth angles of objects and robots in an embodiment of the present invention.

[0068] Apply the Law of Sines to an isosceles triangle:

[0069]

[0070] The λ value is:

[0071]

[0072] The angle χ between the line connecting the human body and the optical center of the camera and the optical axis of the camera is:

[0073]

[0074] Finally, we can get:

[0075]

[0076] Among them, χ is the angle between the line connecting the human body and the optical center of the camera and the optical axis of the camera, β is the opening angle of the camera, It is the ratio of the distance from the center of the identification frame to the edges of the two sides of the picture in the pixel coordinate system.

[0077] S3, based on the positioning results, real-time planning of the tracking path and driving the motion control system to track. Figure 2 , the robot tracking motion state diagram of the embodiment of the present invention is as follows Figure 5 .

[0078] ① When the camera cannot detect a person, the microphone array captures sound information and outputs the azimuth, and controls the robot's movement based on the azimuth;

[0079] If the microphone array outputs an azimuth angle of 0° to 90°, the four-wheeled mobile platform maintains state s1-, state s2-, at the set speed and acceleration. After maintaining this state for one second, the states s1 and s2 are reversed, maintained for one second, and reversed again, and this cycle repeats until the camera detects a speaker.

[0080] If the microphone array output azimuth angle is 90° to 180°, the four-wheeled mobile platform maintains state s1+ and state s2+ at the set speed and acceleration for one second, then reverses states s1 and s2 for one second, and reverses again, repeating this process until the camera detects a speaker.

[0081] If the microphone array output azimuth angle is between -180° and -90°, the four-wheeled mobile platform maintains state s1+ and state s2- at the set speed and acceleration for one second, then reverses states s1 and s2 for one second, and reverses again, repeating this process until the camera detects a speaker.

[0082] If the microphone array outputs an azimuth angle between -90° and 0°, the four-wheeled mobile platform maintains state s1- and state s2+ at the set speed and acceleration for one second, then reverses states s1 and s2 for one second, and reverses again, repeating this process until the camera detects a speaker.

[0083] The coordinate system is established with the robot as the origin. The camera's forward direction is the robot's forward direction. Looking down at the robot, the camera's forward direction rotates counterclockwise for one circle. The corresponding microphone array output azimuth angle changes from 0° to 180°, and then from -180° to 0°. s1+ indicates the four-wheeled mobile platform is moving forward, s1- indicates the four-wheeled mobile platform is moving backward, s2+ indicates the four-wheeled mobile platform is turning left at a set angle, and s2- indicates the four-wheeled mobile platform is turning right at a set angle. Motion state reference Figure 5 .

[0084] ②When the camera locks on one person:

[0085] if The motion state is s1+, if Then the motion state is s1-, otherwise it stops;

[0086] if The motion state is s2+, if The motion state is s2-, otherwise it stops.

[0087] Among them, d m is the measurement value of the tracking distance, χ m is the tracking azimuth measurement value, d is the target value of the tracking distance, χ is the target value of the tracking azimuth, when the tracking distance measurement value falls within the interval The tracking distance measurement value reaches the ideal value. When the tracking azimuth angle measurement value falls within the interval Within , the measurement value of the tracking azimuth angle is considered to have reached the ideal value. The robot motion state is the superposition of the two motion states.

[0088] Speed ​​proportional control based on distance difference is introduced. When the actual relative position between the speaker and the robot deviates greatly from the target position, the robot speed is increased, allowing the robot to quickly approach the target position; when the actual relative position between the speaker and the robot deviates less than the target position, the robot speed is reduced, and the position is fine-tuned.

[0089] Current speaker tracking methods and systems are often based on single-modal positioning methods (sound or image). Tracking systems based on single-modal information are easily affected by environmental factors such as noise and light. They suffer from low sensitivity and poor stability, making it impossible to accurately locate speakers in real time. This significantly limits the reliability and accuracy of speaker tracking. Furthermore, current speaker tracking methods struggle to integrate positioning systems with motion control systems, making it difficult to actively search for people in the environment by driving the motion system. Consequently, their level of intelligence is low.

[0090] This invention proposes a speaker tracking method based on audio-visual dual-modal fusion. A circular microphone array and camera capture the speaker's voice and image information. The system then fuses these two types of positioning information, deriving different motion control instructions based on the different positioning results. This system also provides a solution for proactively searching for the speaker using voice positioning information in situations where the visual module fails to detect a person, making it more intelligent. This positioning method, based on dual-modal information fusion, effectively overcomes the shortcomings of single-modal tracking systems, which are susceptible to environmental factors such as noise and light. These systems suffer from low sensitivity and poor stability, making it impossible to accurately locate the speaker in real time, significantly limiting the reliability and accuracy of speaker tracking. Furthermore, this method overcomes the shortcomings of current speaker tracking methods, which struggle to organically integrate the positioning system with the motion control system, and are unable to proactively search for people in the environment by driving the motion system, resulting in a low level of intelligence. The system can simulate different human-computer interaction scenarios and select different tracking modes based on user needs. It can also implement speaker screening strategies for multiple targets, broadening its application scope.

[0091] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A speaker tracking method based on audio-visual dual-modal fusion, characterized in that: The following steps are involved: S1, obtain image and voice information through the camera and microphone array respectively; S2: Fusing sound and image information, localizing the speaker based on the number of people in the image and whether there is voice input. S3, plan the tracking path in real time based on the positioning results and drive the motion control system to track; Step S2 is specifically as follows: S21. Determine a person to be tracked based on the acquired image and voice information; S22. Use the rectangular frame of the human body recognition model to lock the person, and return the pixel width and center coordinates of the rectangular frame through the model; S23. Substitute the pixel width and center coordinates of the rectangular frame returned by the model and the known pixel width and center coordinates of the image into the following distance calculation model and azimuth calculation model to obtain the distance and azimuth between the robot's camera and the person: Where d is the distance between the human body and the camera, x is the number of pixels that the object occupies the width of the screen, and k is a constant value; Where, χ is the angle between the line connecting the optical center of the human body and the camera and the optical axis of the camera, β is the opening angle of the camera, It is the ratio of the distance between the center of the identification frame and the edges of the two sides of the picture in the pixel coordinate system; Step S21 is specifically as follows: When the camera cannot detect a person, the microphone array captures sound information and outputs the azimuth angle, and the robot is controlled to move according to the azimuth angle; When the camera detects multiple people and detects someone talking, it calculates the distance between the robot and each speaker and locks on the speaker closest to it; When the camera detects multiple people and no one is talking, it locks on the person closest to it; When the camera detects a single person, it locks on that person.

2. The speaker tracking method based on audio-visual dual-modal fusion according to claim 1, characterized in that: The robot uses a motion control system with a four-wheel mobile platform.

3. The speaker tracking method based on audio-visual dual-modal fusion according to claim 2, characterized in that: When the camera cannot detect a person, the microphone array captures sound information and outputs the azimuth angle. The robot's movement is controlled according to the azimuth angle as follows: If the microphone array output azimuth angle is 0°~90°, the four-wheel mobile platform maintains state s1-, state s2- at the set speed and acceleration, and then changes state to s 1, s 2. Reverse for one second, then reverse again, and repeat until the camera detects the speaker. If the microphone array output azimuth angle is 90°~180°, the four-wheel mobile platform maintains state s1+, state s2+ at the set speed and acceleration, and then changes state to s 1, s 2. Reverse for one second, then reverse again, and repeat until the camera detects the speaker. If the microphone array output azimuth angle is -180°~-90°, the four-wheel mobile platform maintains state s1+ and state s2- at the set speed and acceleration for one second and then changes state s 1, s 2. Reverse for one second, then reverse again, and repeat until the camera detects the speaker. If the microphone array output azimuth angle is -90°~0°, the four-wheel mobile platform maintains state s1-, state s2+ at the set speed and acceleration, and then changes state to s1- after one second. s 1, s 2. Reverse for one second, then reverse again, and repeat until the camera detects the speaker. Among them, the above coordinate system is established with the robot as the coordinate origin. The camera's forward direction is the robot's forward direction. Looking down at the robot, the camera's forward direction rotates counterclockwise for one circle, and the corresponding microphone array output azimuth angle changes from 0° to 180°, and then from -180° to 0°; s1+ means the four-wheel mobile platform moves forward, s1- means the four-wheel mobile platform moves backward, s2+ means the four-wheel mobile platform turns left at a set angle, and s2- means the four-wheel mobile platform turns right at a set angle.

4. The speaker tracking method based on audio-visual dual-modal fusion according to claim 3, characterized in that: When the camera locks on a person, if d m ≥ , then the motion state is s1+, if d m ≤ , then the motion state is s1-, otherwise it stops; if χ m > , then the motion state is s2+, if χ m < , then the motion state is s2-, otherwise it stops; in, d m is the measurement value of the tracking distance, χ m To track azimuth measurements, d is the target value of the tracking distance, χ To track the azimuth target value, when the measured value of the tracking distance falls within the interval The tracking distance measurement value reaches the ideal value. When the tracking azimuth angle measurement value falls within the interval The measurement value of the tracking azimuth angle is considered to have reached the ideal value.

Images