A robot attention scheduling method, medium and device
By identifying and prioritizing candidate users in multi-person speaking scenarios, and setting head-turning time control and position verification, the problem of frequent random head turning of the robot was solved, and stable multi-person interactive response was achieved.
Patent Information
- Application Number
- CN202611577905.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-13
- Publication Date
- 2026-11-10
AI Technical Summary
When multiple people speak simultaneously or alternately, the robot's head frequently turns erratically, making it unable to respond stably to each user. Existing solutions struggle to set reasonable head-turning intervals and strategies, resulting in an unstable interaction process.
By identifying candidate speaking users within the interaction area, establishing interactive tasks to be responded to and calculating task priorities, setting head-turning time control strategies and position confidence checks, controlling the robot's head to turn towards the current task user, and updating the task set in attention-locked state to avoid frequent head turns.
It improves the robot's head-turning accuracy and interaction continuity in multi-person speaking scenarios, ensuring that each user can receive a stable response and avoiding frequent head-turning and interruptions to the current interaction.
Smart Images

Figure CN123068049A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of human-computer interaction technology, and in particular to a robot attention scheduling method, medium, and device. Background Technology
[0002] In scenarios where multiple users consult, ask questions, or interact with a robot, the robot typically needs to turn its head toward the user who is speaking to form a gaze toward that user, and then use voice, facial expressions, or body language to provide interactive feedback.
[0003] Existing solutions mostly focus on "identifying the current speaker" or "locating the current sound source," that is, estimating the direction of arrival of a single sound source through a microphone array, or locating the face of the person speaking through a camera, and then the controller drives the robot's head to turn according to the real-time direction of the sound source. This type of solution can work normally when a single person is speaking continuously, but it lacks head-turning interval control and head-turning strategies for scenarios where multiple people are speaking simultaneously.
[0004] When two or more users speak at the same time, take turns speaking, or speak continuously in a short period of time, the direction of the real-time sound source will change rapidly between different users. The robot's head is prone to swinging back and forth following the instantaneous sound source direction, which manifests as disorientation and frequent random turning, making the interaction process unstable.
[0005] Furthermore, when multiple people speak simultaneously, the robot needs to determine who to respond to first, when to switch to the next person, whether to reconfirm the target location before switching, and how to prevent users from being ignored for extended periods. If the robot turns its head immediately after detecting a new sound source direction, it will interrupt the current user's unfinished interaction. If it doesn't switch for a long time, other users will not be able to receive responses in sequence. Existing solutions that simply turn their heads based on sound intensity, distance, fixed user priority, or the direction of the last sound source are all insufficient to create a stable, continuous response order that aligns with the multi-user interaction experience.
[0006] Therefore, when multiple people are speaking simultaneously or alternately, how to set the time interval and head-turning strategy for the robot's head to turn so that the robot can respond to each speaking user one by one, while avoiding the frequent random head turning between multiple sound sources, has become an urgent technical problem to be solved. Summary of the Invention
[0007] To address the aforementioned technical problems, the present invention provides a robot attention scheduling method, which includes the following steps:
[0008] S1 identifies candidate speaking users within the interactive area based on audio and image information within the interactive area.
[0009] S2, when at least two candidate speaking users are detected to have overlapping speech times, or when at least two speaking events that meet the preset valid conditions occur within a preset time window, the multi-person speaking scheduling state is entered, and corresponding interactive tasks to be responded to are established for each candidate speaking user and the task priority is calculated to obtain an interactive task set.
[0010] S3. Based on the preset head-turning time control strategy and task priority, determine a current task that has passed the position confidence check from the set of interactive tasks, and control the robot's head to turn towards the candidate speaking user corresponding to the current task, enter the attention lock state and execute the interactive response. In the attention lock state, if a new sound source is detected, only the set of interactive tasks is updated and the head turning is not triggered. If there is an interactive task waiting to be responded to in the set of interactive tasks that meets the preset preemption conditions, the attention lock state is released and S4 is executed.
[0011] S4. When the current task meets the preset response completion conditions, is marked as invalid, or is preempted, if it is in the attention lock state, the attention lock state is released and the next current task is determined. If the current task is preempted, the preempted task is determined as the next current task. If the current task meets the preset response completion conditions or is marked as invalid, the next current task is selected from the remaining interactive tasks to be responded to in the interactive task set according to the task priority.
[0012] S5, update the spatial location of the next candidate speaking user corresponding to the current task, and verify the location confidence.
[0013] S6. If the updated spatial orientation and position confidence meet the preset interaction conditions, the robot head is controlled to turn towards the candidate speaking user according to the updated spatial orientation, enters the attention lock state and executes the interaction response, and returns to execute S4. If the updated spatial orientation or position confidence does not meet the preset interaction conditions, the next current task is marked as invalid, and returns to execute S4 until there are no pending interaction tasks in the interaction task set that meet the preset interaction conditions, and exits the multi-person speaking scheduling state.
[0014] The present invention also provides a non-transitory computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the above-described robot attention scheduling method.
[0015] The present invention also provides an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.
[0016] This invention has at least the following beneficial effects: By entering the multi-person speaking scheduling state only when multiple valid speaking events occur within a preset time window due to overlapping speech times, and by establishing a separate interactive task to be responded to for each candidate speaking user, the problem of the robot only retaining one voice and losing the requests of other users when multiple people speak at the same time is solved; by setting a head-turning time control strategy, the robot's head turns at a controllable rhythm, maintaining attention lock while the current task is not completed, and only updating the task without triggering head turning when a new sound source is present, the problem of the head frequently swaying with instantaneous sound sources and the interruption of the current interaction is solved; by updating the spatial orientation of the candidate speaking user corresponding to the next current task and verifying the position confidence before each head turn, and marking the next current task as invalid and reselecting it when the preset interaction conditions are not met, the problem of the robot turning back to the old position due to user movement during the waiting period is solved, thus improving the head-turning hit rate and interaction continuity. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart of a robot attention scheduling method provided in Embodiment 1 of the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is understood that, where appropriate, the terms used to distinguish similar objects can be interchanged so that the invention can also be implemented in other embodiments besides the illustrated or described embodiments. Furthermore, the terms "including," "having," and any variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.
[0021] Example 1
[0022] This embodiment provides a robot attention scheduling method, such as Figure 1 As shown, the robot attention scheduling method includes the following steps:
[0023] S1 identifies candidate speaking users within the interactive area based on audio and image information within the interactive area.
[0024] The interaction area is a pre-defined spatial range around the robot where effective voice interaction can occur. It can be set according to the robot's pickup distance, the camera's field of view, and the standing habits of the users. For example, it can be a circular area with a radius of 3 to 5 meters centered on the robot's base, or a fan-shaped area with a radius of 4 meters at a 120-degree angle directly in front of the robot.
[0025] The audio information is acoustic data collected and processed by the sound acquisition device on the robot, and the image information is visual data collected by the image acquisition device on the robot.
[0026] Candidate speaking users are those identified as currently speaking or who have just spoken within the interaction area. Unlike single-objective schemes that retain only one audio stream, multiple candidate speaking users can exist simultaneously, each corresponding to a pending interaction task, thus ensuring that each speaking user is included in the scheduling queue.
[0027] In one specific embodiment, S1 includes the following steps:
[0028] S11, acquire audio information within the interactive area through a sound acquisition device, wherein the audio information includes the direction of the sound source, sound intensity, and duration of speech activity.
[0029] S12, image information within the interactive area is acquired through an image acquisition device, wherein the image information includes at least one of face information, human body position information, and mouth movement information.
[0030] S13, receive speaker identification information output by an external speech processing module, wherein the speaker identification information includes speaker identification, speech text, overlapping speech markers and speaker recognition confidence.
[0031] S14. Based on audio information, image information and speaker identification information, identify candidate speaking users in the interaction area, and determine the spatial location and tracking identifier corresponding to each candidate speaking user. The tracking identifier is used to associate speech events of the same candidate speaking user that are adjacent in time when the user identifier cannot be obtained.
[0032] The sound acquisition device includes a microphone array, a distributed microphone, or a directional pickup module. As those skilled in the art know, the sound acquisition device can be implemented using any pickup structure known in the art, such as a six-microphone ring array, a four-microphone linear array, or a dual-microphone array.
[0033] The direction of the sound source is estimated by the sound acquisition device based on the arrival time difference or phase difference between each microphone channel, and can be represented by the horizontal azimuth and pitch angles relative to the front of the robot. Sound intensity is a measure of the energy of the speech signal within a preset frequency band, and can be expressed as short-time energy, sound pressure level, or decibels. In one specific embodiment, the sound intensity can be taken as the average of the root mean square energy of several consecutive frames of speech signals to reduce the impact of single-frame fluctuations. Speech activity time is the time interval formed by the start and end times of a speech segment, i.e., the speech start and end time. Speech activity time can be obtained through speech activity detection, which is used to determine whether human speech exists in the current audio frame. In one specific embodiment, when the number of frames in a consecutive audio frame where speech appears is greater than or equal to a preset frame number threshold, it is determined as a speech activity, and its start and end times are recorded.
[0034] Image acquisition devices include RGB cameras, depth cameras, wide-angle cameras, or gimbal cameras.
[0035] Facial information refers to at least one of the following: the location of a face bounding box, facial landmarks, or face orientation detected from image information. Human body position information refers to the location of a human body bounding box or the location of human skeletal landmarks detected from image information. Mouth movement information is the opening and closing amplitude or frequency obtained based on changes in the mouth area across consecutive image frames; it is used to assist in determining whether a user is speaking when the user is not uttering sufficiently loud speech or when there is strong ambient noise.
[0036] An external speech processing module is a functional unit that is independent of or directly integrated into the robot for generating speaker identification information, such as an automatic speech recognition module, a voiceprint recognition module, or a multi-person speech structuring module. Those skilled in the art will understand that the aforementioned external speech processing module can be implemented using any voiceprint or speech processing model known in the art, such as a speaker embedding model based on d-vector or x-vector, or an end-to-end multi-person speech recognition model with speaker separation capabilities. The above model selection is merely illustrative; those skilled in the art can use other models with the same speaker discrimination capabilities as substitutes, as long as the function of outputting speaker identification is possible. These alternative solutions all fall within the protection scope of this invention.
[0037] Speaker identifiers are labels that distinguish different speakers. They can be stable identifiers bound to a registered user in a voiceprint registry, or temporary serial numbers assigned during a single interaction. Speech text is the text content obtained after speech recognition. Overlapping speech markers are flags indicating whether two or more users are speaking simultaneously in the current speech segment. Their values can be binary or the proportion of overlapping time in a speech segment. Speaker recognition confidence is a measure of the reliability of the external speech processing module's speaker attribution determination; its value can be normalized to between 0 and 1.
[0038] Spatial orientation describes the position of a candidate speaker relative to the robot. It can be represented in polar coordinates, such as a triplet consisting of horizontal azimuth, pitch, and distance, or converted to two-dimensional or three-dimensional coordinates in the robot's base coordinate system. In one specific implementation, spatial orientation can be obtained by fusing the sound source direction and distance estimation results, or directly from the center of the face bounding box in the image information combined with the distance measured by the depth camera. Tracking identifiers are intermediate numbers used to associate the same candidate speaker across time clusters when stable user identity identifiers are unavailable. For example, when registered users A and B are present simultaneously, stable identity identifiers obtained through registration can be used. When a user is not registered or voiceprint matching fails, a temporary number is assigned by the camera's human tracking, and this number remains unchanged in subsequent frames through positional continuity. The tracking identifier establishes a correspondence with task merging in step S2 and position relocation in step S5, forming the basis for subsequent task merging and position confidence calculation among multiple speeches by the same user.
[0039] The above-mentioned method solves the problem of difficulty in distinguishing different speakers when multiple people are speaking in overlapping speech by integrating audio information, image information and speaker identification information output by the external speech processing module in the interaction area, thereby improving the completeness and accuracy of candidate speaker identification.
[0040] S2, when at least two candidate speaking users are detected to have overlapping speech times, or when at least two speaking events that meet the preset valid conditions occur within a preset time window, the multi-person speaking scheduling state is entered, and corresponding interactive tasks to be responded to are established for each candidate speaking user and the task priority is calculated to obtain an interactive task set.
[0041] Voice time overlap refers to a situation where the voice activity times of two or more candidate speakers share a common interval on the timeline. Voice time overlap, commonly known as interruption or simultaneous speaking, is the first type of triggering condition for entering a multi-person speaking scheduling state.
[0042] The preset time window is a sliding time length used to capture alternating speech scenarios. During alternating speech, the voice segments of two users may not overlap, but they occur successively within a short time interval. In this case, simply detecting speech overlap will not trigger the multi-person speaking scheduling state; therefore, a second triggering scenario is needed: at least two valid speech events must exist simultaneously within the preset time window. The specific value of the preset time window can be set by the implementer according to the actual situation. In one specific implementation, the preset time window can be set to 2 to 5 seconds, for example, 3 seconds.
[0043] Multi-person speech scheduling state refers to a special control state that the robot enters. In this state, the robot no longer determines its head orientation based on the instantaneous direction of the sound source. Instead, it determines which candidate user to turn its head to based on the task priority of each interactive task to be responded to in the interactive task set, combined with a preset head-turning time control strategy. In contrast, when not in multi-person speech scheduling state, the robot can use the conventional single-target sound source following method.
[0044] Each pending interaction task is a record generated for each candidate speaking user, with at least one pending interaction task corresponding to each candidate speaking user. The interaction task set is a collection of all pending interaction tasks that have not yet been completed and have not been marked as invalid. Subsequent selections by the robot are made within this set. Task priority is a quantified score used to rank pending interaction tasks; a higher score indicates that the task should be responded to more frequently. Task priorities are recalculated before each scheduling, thus reflecting the dynamic changes in factors such as the waiting time and urgency of each candidate speaking user.
[0045] In one specific implementation, determining whether a speech event meets preset valid conditions includes the following steps:
[0046] S211, obtain the speech duration, sound intensity, speech recognition confidence, and speech text corresponding to the speech events generated by the candidate speaking users.
[0047] S212, if the speech duration is greater than or equal to a preset duration threshold and the sound intensity is greater than or equal to a preset intensity threshold, then the speech event is determined as a candidate speech event.
[0048] S213, if the speech recognition confidence level corresponding to the candidate speech event is greater than or equal to the preset confidence threshold, or if the speech text corresponding to the candidate speech event contains a preset wake-up word or preset interaction keyword, then the candidate speech event is determined as a speech event that meets the preset valid conditions.
[0049] A speech event is a series of consecutive statements by a candidate user, which can be defined by the start and end times of the voice activity. When the silence interval between two statements by the same user is less than a preset silence duration, the two statements can be merged into a single speech event.
[0050] Speech duration is the length of the speech segment corresponding to a speech event. A preset duration threshold is used to filter out invalid speech such as noise tapping, coughing, and short responses. In one specific implementation, the preset duration threshold can be set to 0.3 seconds to 0.8 seconds, for example, 0.5 seconds.
[0051] A preset intensity threshold is used to filter out speech that is too far away or too low in volume. In one specific embodiment, the preset intensity threshold can be set according to the signal-to-noise ratio relative to the ambient noise level, for example, a signal-to-noise ratio greater than or equal to 10 dB, or an absolute sound pressure level greater than or equal to 45 dB.
[0052] Speech recognition confidence score is a numerical value output by an external speech processing module that reflects the reliability of the speech recognition result. Its value range can be normalized to between 0 and 1. In one specific implementation, the preset confidence threshold can be set to 0.5 to 0.7, for example, 0.6.
[0053] Preset wake words are fixed phrases used to attract the robot's attention, such as the robot's name. Preset interaction keywords are a vocabulary related to the business scope; for example, in a service hall scenario, they could include words like "consultation," "processing," and "guidance." By setting preset wake words or preset interaction keywords, even if the speech recognition confidence is low, as long as a phrase clearly points to the robot, it should be considered a valid speech event to avoid missing responses from users who genuinely want to interact with the robot.
[0054] The speech energy collected by the microphone array contains a large number of sound sources that are not intended for interaction. If an interactive task is created for each sound source without differentiation, the set of interactive tasks will be filled with noisy tasks, causing users who really need to respond to be at the end of the queue. Therefore, this embodiment first filters out unintentional short sounds by using the duration and intensity at the physical layer, and then confirms the interaction intent by using the recognition confidence or keywords at the semantic layer. This two-level filtering can achieve a balance between the false negative rate and the false positive rate.
[0055] In one specific implementation, a corresponding interactive task to be responded to is established for each candidate speaking user, including the following steps:
[0056] S221, when a candidate speaking user is identified to produce a speech event that meets the preset valid conditions, a response interaction task is established or updated for the candidate speaking user. The response interaction task includes a task identifier, a user identifier or tracking identifier, a speaker identifier, a sound source direction, a spatial orientation, a sound intensity, a speech start and end time, a speech text, a waiting time, a task status, a priority score, whether it is in an overlapping speech interval, and a last update time. The task status includes pending confirmation, pending response, currently locked, interrupted, completed, and invalid.
[0057] S222, if the time interval between adjacent speech events of the same candidate speaking user is less than the preset merging time, the azimuth difference is less than the preset merging angle threshold, and the semantics belong to the same round of requests, then the new speech event is merged into the existing pending interaction task of the candidate speaking user.
[0058] S223 If different candidate users speak at the same time, then create or update the corresponding interactive task to be responded to for each candidate user.
[0059] S224. If the corresponding candidate speaking user is not detected within the preset failure time, the corresponding candidate speaking user leaves the interaction area, or the relocation fails, then the interactive task to be responded to is marked as failed and removed from the interactive task set.
[0060] The task identifier is a unique number for the interactive task awaiting a response, which can be an auto-incrementing sequence number or a globally unique identifier. The waiting time is the cumulative duration from the time the interactive task was created to the current time. The last update time is the time when this field of the task was most recently modified. The "whether it is in an overlapping speech interval" flag indicates whether the speech segment at the time the task was created overlaps with the speech segments of other users; this flag can serve as an auxiliary input for calculating the semantic urgency of the task later.
[0061] Task status describes the stage of an interactive task within its lifecycle. Specifically, "Pending Confirmation" indicates that the task has just been established and requires continuous confirmation within the target stabilization time; "Pending Response" indicates that the task has entered the interactive task set and participated in the sorting; "Currently Locked" indicates that the task has been identified as the current task and the robot is in an attention-locked state; "Interrupted" indicates that the task was paused during execution due to task preemption and is waiting to be re-queued; "Completed" indicates that the task has been successfully responded to; and "Invalidated" indicates that the task has been removed from the schedule due to user departure, inability to locate, or timeout.
[0062] The preset merging time is used to merge segmented statements from the same user, preventing a single request from being segmented into multiple segments by the voice endpoint detection and then repeatedly queued. In one specific implementation, the preset merging time can be set to 0.8 seconds to 2 seconds, for example, 1.2 seconds.
[0063] A preset merging angle threshold is used to determine whether two adjacent speech segments come from users in the same direction. In one specific implementation, the preset merging angle threshold can be set to 10 to 20 degrees, for example, 15 degrees.
[0064] The semantic similarity between two adjacent audio text segments can be used to determine if they belong to the same round of requests. For example, if two audio text segments are concatenated to form a complete question, or if the semantic similarity between the two audio text segments is greater than or equal to a preset similarity threshold, they are determined to be requests in the same round.
[0065] It is important to emphasize that for repeated messages from the same user within the preset merging time, the processing method is to merge rather than create new messages, in order to avoid the same user repeatedly speaking and continuously crowding the queue. For simultaneous messages from different users, the processing method is to create separate tasks for each user, to ensure that each user's request is independently preserved, and not just one audio stream is retained.
[0066] A preset expiration time is used to clean up tasks that have been stagnant for a long time, preventing zombie tasks in the interactive task set from continuously occupying sorting resources. In one specific implementation, the preset expiration time can be set to 3 to 10 seconds, for example, 6 seconds.
[0067] Relocation failure refers to a situation where, within a preset failure time, methods such as sound source localization, face tracking, human body tracking, or voiceprint matching are unable to obtain the user's effective spatial location.
[0068] In one specific implementation, step S223 involves creating or updating corresponding interactive tasks for each candidate speaking user, including the following steps:
[0069] S2231, when different candidate speaking users are detected speaking at the same time, the speech segments of simultaneous speaking are divided into speech segments corresponding to each candidate speaking user according to the overlapping speech marker and speaker identifier, so as to obtain the speech start and end time and speech text corresponding to each candidate speaking user.
[0070] S2232, for any candidate speaking user, based on the user identifier or tracking identifier, speaker identifier and sound source direction corresponding to the candidate speaking user, search for the existing pending interactive tasks of the candidate speaking user in the interactive task set.
[0071] S2233, if an existing interactive task awaiting response is found for the candidate speaker, the start and end times of the candidate speaker's speech, the speech text, and the sound intensity are added to the existing interactive task awaiting response. The status of whether the speech is in an overlapping speech interval is updated to yes, and the last update time is updated to obtain the updated interactive task awaiting response.
[0072] S2234 If no existing interactive task for the candidate speaker is found, a new interactive task is created for the candidate speaker. The source direction, spatial orientation, start and end time of the speech, speech text, sound intensity, and whether it is in an overlapping speech interval are written into the new interactive task. Each interactive task for the candidate speaker is independent of the others and saves its own speech text. The speech text of one candidate speaker is not overwritten by the speech text of another candidate speaker.
[0073] The process of splitting simultaneously speaking speech segments into speech segments corresponding to each candidate speaker refers to dividing a mixed speech segment containing the voices of two or more candidate speakers into several segments according to the speaker's affiliation, so that each segment corresponds to only one candidate speaker. In one specific implementation, when the overlapping speech marker output by the external speech processing module indicates that there is overlapping speech in the current speech segment, the frame-by-frame speaker affiliation label output by the external speech processing module can be obtained. Consecutive frames with the same speaker affiliation are merged into the speech segment corresponding to that speaker. The time of the earliest frame and the time of the latest frame in each speech segment are respectively used as the start and end times of the speech corresponding to the candidate speaker. The recognition result of each speech segment is used as the speech text corresponding to the candidate speaker. Those skilled in the art will understand that the aforementioned speaker attribution determination and speech segment segmentation can be implemented using any speaker separation or speaker log algorithm known in the art, such as a clustered speaker log system. The algorithm selection mentioned above is merely illustrative, and those skilled in the art can use other algorithms with the same speaker attribution discrimination capability as long as they can split the mixed speech into their respective corresponding speech segments according to the speaker. All these alternative solutions fall within the protection scope of this invention.
[0074] Since stable identity identifiers are unaffected by the speaking state, and tracking identifiers are only valid when time is adjacent, prioritizing matching based on tracking identifiers can easily lead to misassociating another candidate speaker with the current one. Therefore, when searching for existing pending interaction tasks for the candidate speaker, the following matching criteria are used in descending order of reliability: First, matching is performed using the user identifier, which is the stable identity identifier corresponding to the registered user, and has the highest matching reliability. When the user identifier cannot be obtained, matching is performed using the speaker identifier. When the speaker identifier also cannot be obtained, matching is performed using the tracking identifier combined with the fact that the azimuth difference between the sound source direction and the spatial orientation recorded by the existing task is less than a preset merging angle threshold.
[0075] In one specific implementation, when two or more existing interactive tasks that need to be responded to simultaneously meet the matching conditions, the existing interactive task with the smallest azimuth difference is selected as the found interactive task to be responded to.
[0076] Adding a new task is used when the same candidate speaker continues speaking based on an existing task. Their speech text is saved using a concatenated method, allowing the robot to respond based on the complete question text, rather than just the last short sentence. Creating a new task is used when the candidate speaker did not previously have a task in the interaction task set. Whether adding or creating a new task, each candidate speaker's corresponding pending interaction task is an independent entry in the interaction task set, participating independently in task priority calculation and sorting, without overlapping with each other.
[0077] The above-mentioned method solves the problem of only retaining one voice and discarding or overwriting the requests of other candidate users when multiple people are speaking at the same time. This is achieved by first splitting the mixed speech into corresponding speech segments according to the speaker's affiliation when different candidate users are detected speaking at the same time, and then searching for existing tasks according to the priority order of user identifier, speaker identifier, and tracking identifier.
[0078] In one specific implementation, calculating task priority includes the following steps:
[0079] S231, for any candidate speaking user, update the non-response compensation score corresponding to the candidate speaking user according to the number of times the candidate speaking user has continuously established pending interaction tasks and the cumulative non-service duration, using a monotonically increasing function.
[0080] S232, obtain the priority parameters corresponding to the candidate speaking user and normalize them respectively. The priority parameters include the speaking order score, waiting time score, semantic urgency score, user permission score, voice validity score, updated non-response compensation score, turning cost score, and low confidence penalty score. The turning cost score is determined based on the turning angle between the robot's current orientation and the candidate speaking user. The low confidence penalty score is determined based on the location confidence and speaker recognition confidence of the candidate speaking user.
[0081] S233, the normalized priority parameters are weighted and summed according to the preset weights to obtain the task priority of the interactive task to be responded to for the candidate speaking user. Among them, the weights corresponding to the turning cost score and the low confidence penalty score are negative weights.
[0082] S234. Sort the interactive tasks to be responded to in the interactive task set in descending order of task priority.
[0083] The speaking order score reflects the order in which candidate users begin speaking, with the user who speaks first receiving a higher score. In one specific implementation, the scores can be assigned according to the ascending order of the speaking events within the set of interactive tasks. For example, the earliest speaking user can be assigned a score of 1, the latest speaking user a score of 0, and intermediate users linearly interpolated according to their ranking.
[0084] The waiting time score reflects the time a candidate user has waited to speak; the longer the wait, the higher the score. In one specific implementation, the waiting time score can be the ratio of the waiting time to a preset maximum waiting time, and is limited to between 0 and 1.
[0085] The semantic urgency score is used to reflect whether the speech text contains content that requires immediate attention. In one specific implementation, the speech text can be classified or matched with keywords. When it contains urgent expressions such as "help," "fire," or "someone has fallen," it is assigned a value of 1. When it only contains general inquiry content, it is assigned a value of 0. In intermediate cases, it is assigned a value between 0 and 1 according to a preset mapping table.
[0086] User permission scores are used to differentiate the service levels of different users. For example, staff, managers, or users who have completed real-name authentication can be given higher scores.
[0087] The sound effectiveness score reflects the intensity of the sound or the intelligibility of the speech; the greater the sound intensity and the clearer the speech, the higher the score.
[0088] The unresponsiveness compensation score is used to prevent a task from remaining in a waiting state for an extended period due to the continuous insertion of new tasks. Specifically, the more times the same candidate speaking user establishes pending interaction tasks and the longer the cumulative unservice time, the higher the score. In one specific implementation, the unresponsiveness compensation score for the i-th candidate speaking user is... i =clip(α×n i +β×t wait i (,0,1). Where n i The number of times t is consecutively created as an interactive task to be responded to for the i-th candidate speaking user. wait i The cumulative unservice duration for the user is represented by α and β, which are preset coefficients. `clip()` indicates that the calculation result should be limited to preset upper and lower limits. `i` = 1, 2, ..., N, where N is the total number of candidate speaking users. In one specific implementation, α can be set to 0.15, and β can be set to 0.05 per waiting unit.
[0089] In another specific implementation, the non-response compensation score can be assigned in a tiered manner: when the waiting time exceeds a first preset waiting threshold, the score is increased by one tier; when the waiting time exceeds a second preset waiting threshold, the task is promoted to the priority response candidate set, ensuring it receives at least one priority response opportunity. The above two methods of assigning non-response compensation scores are merely illustrative examples. Those skilled in the art can use other monotonically increasing assignment functions instead, as long as they achieve the effect of ranking higher for longer wait times. These alternative solutions all fall within the protection scope of this invention.
[0090] The turning cost score is used to suppress situations where the robot needs to turn its head significantly in response to a user, thereby reducing the overall head travel. In one specific implementation, the turning cost score can be the ratio of the robot's current head angle relative to the candidate speaking user to a preset maximum head angle.
[0091] The low confidence penalty score is used to de-weight users whose location or speaker recognition results are unreliable. In one specific implementation, the low confidence penalty score can be the inverse measure of the combination of location confidence and speaker recognition confidence, for example, taking 1 minus the weighted sum of location confidence and speaker recognition confidence. In another specific implementation, linear normalization can be used, that is, subtracting the minimum value of each parameter and dividing by the difference between the maximum and minimum values. Alternatively, a segmented lookup table can be used to map each parameter to a range of 0 to 1.
[0092] The normalized priority parameters are summed using preset weights. The specific values of these preset weights can be set by the implementer based on actual conditions. The weights corresponding to the turning cost score and the low confidence penalty score are negative. In one specific implementation, the weights corresponding to the speaking order score, waiting time score, semantic urgency score, user permission score, voice validity score, updated non-response compensation score, turning cost score, and low confidence penalty score can be set to 0.15, 0.20, 0.20, 0.10, 0.10, 0.15, -0.07, and -0.08, respectively.
[0093] The above approach addresses the issues of some users' requests being dropped and some users waiting for a long time without a response when multiple people are speaking at the same time. By creating separate interactive tasks for each candidate user, merging repeated speeches by the same user, retaining concurrent speeches by different users, and calculating task priority by combining a non-response compensation score that increases with waiting time and number of non-response times, the fairness and rationality of the response order are improved.
[0094] S3. Based on the preset head-turning time control strategy and task priority, determine a current task that has passed the position confidence check from the set of interactive tasks, and control the robot's head to turn towards the candidate speaking user corresponding to the current task, enter the attention lock state and execute the interactive response. In the attention lock state, if a new sound source is detected, only the set of interactive tasks is updated and the head turning is not triggered. If there is an interactive task waiting to be responded to in the set of interactive tasks that meets the preset preemption conditions, the attention lock state is released and S4 is executed.
[0095] The preset head-turning time control strategy refers to a set of time parameters and their usage rules used to constrain the timing of the robot's head rotation. It is used to set a time threshold between two consecutive head rotations so that the robot will not immediately change its orientation due to the instantaneous change in the direction of the sound source.
[0096] The current task is the interactive task that was selected in this scheduling, passed the position confidence check, and is currently being faced and responded to by the robot. Attention lock state refers to the state in which the robot remains facing the candidate speaking user corresponding to the current task until the current task is completed or fails. In this state, even if the robot detects other users' speech or changes in the direction of the sound source, it maintains its current head orientation and records the newly generated speech event as a new interactive task to be responded to, thereby ensuring the completion and continuity of each round of interaction.
[0097] Interactive response refers to the feedback action that the robot makes to the candidate speaking user corresponding to the current task, which may include at least one of the following: voice reply, facial expression display, screen content switching, arm movement or nodding action.
[0098] A new sound source refers to a speech event that appears newly during the attention lock state and does not belong to the candidate speaker of the current task. For a new sound source, the robot establishes it as a new interactive task to be responded to and adds it to the interactive task set, without changing the current head orientation.
[0099] Pre-defined preemption conditions are exceptional conditions that allow interruption of the current task and early switching of the response object, ensuring that urgent or high-priority requests can be processed promptly. In one specific implementation, the pre-defined preemption conditions may include: the voice text of the interactive task to be responded to contains urgent semantics, or the candidate speaking user corresponding to the task has a preset permission level and issues a control command. Interactive tasks to be responded to that meet the pre-defined preemption conditions are preempted tasks.
[0100] If all interruptions are strictly prohibited, emergency requests for help will not receive a timely response while the robot is answering routine inquiries. If arbitrary interruptions are allowed, it degenerates into frequent sound source following. Therefore, a preemption condition is set to reserve a clear, strictly triggered exception channel in addition to the default lockout.
[0101] Specifically, the spatial orientation of the candidate speaking user corresponding to the current task is updated, and the location confidence of the candidate speaking user corresponding to the current task is calculated based on the deviation between the updated and unupdated spatial orientations and the matching confidence of the updated spatial orientation. If the location confidence of the updated spatial orientation is greater than or equal to a preset confidence threshold, the location confidence check is passed. Otherwise, the current task is marked as invalid, and a new current task is determined from the set of interactive tasks. S3 is repeated until there are no interactive tasks in the set of interactive tasks that meet the location confidence check.
[0102] In one specific implementation, the preset head-turning time control strategy includes the following steps:
[0103] S31, obtain the minimum head-turning interval time, the current user's shortest holding time, the current task's maximum holding time, and the target stabilization time.
[0104] S32: Based on the minimum head-turning interval, the current user's shortest holding time, the current task's maximum holding time, and the target stabilization time, control the robot's head-turning rhythm. Specifically, outside of interactive tasks awaiting response that meet preset preemption conditions, after completing a valid head turn, the robot will not turn its head again due to a new sound source within the minimum head-turning interval. Furthermore, once the robot enters an attention-locked state, it will maintain this state for the current user's shortest holding time. If the current task is not completed within its maximum holding time, reassess whether to switch to other interactive tasks awaiting response in the interactive task set. When a candidate speaking user corresponding to an interactive task awaiting response in the interactive task set continuously exists or is continuously confirmed within the target stabilization time, the interactive task corresponding to that candidate speaking user is allowed to be designated as the next current task.
[0105] S33 sets a directional backlash angle for the robot's head. For candidate speakers in the attention-locked state, when the detected sound source direction fluctuates within the directional backlash angle range, the robot's head orientation remains unchanged. For candidate speakers not in the attention-locked state, the stability condition corresponding to the target stabilization time is met only when the change in the sound source direction of the candidate speaker within the continuous confirmation window is less than the preset fluctuation threshold.
[0106] The minimum head-turning interval is the minimum time interval that must be satisfied between two consecutive effective head turns, used to suppress high-frequency reciprocating head movements. The minimum current user hold time is the shortest time the robot must maintain its attention lock state after entering it, ensuring the current user receives a full gaze duration. The maximum current task hold time is the maximum time a single current task is allowed to occupy the robot's attention; exceeding this time forces a switch evaluation, preventing a single user from occupying attention for extended periods. The target stabilization time is the shortest time a candidate speaking user must exist or be continuously confirmed before being allowed as a head-turning target, preventing turning to a fleeting target that may disappear immediately.
[0107] The specific values of the time parameters can be set by the implementer according to the actual situation. In one specific implementation, the minimum head-turning interval can be set to 0.6 seconds to 1.8 seconds, for example, 1.2 seconds. The minimum holding time for the current user can be set to 1.2 seconds to 4 seconds, for example, 2.5 seconds. The maximum holding time for the current task can be set to 6 seconds to 15 seconds, for example, 10 seconds. The target stabilization time can be set to 0.3 seconds to 1 second, for example, 0.6 seconds.
[0108] The minimum head-turning interval constrains the rhythm between two head turns, while the shortest current user hold time constrains the rhythm within a single gaze; together, they constitute the lower limit for anti-shake. The maximum current task hold time constitutes the upper limit to prevent monopolization. The target stabilization time affects the selection of the next head-turning target; only when a candidate speaking user is stably present is its corresponding pending interaction task allowed to be determined as the next current task. Through the combination of upper and lower limits and time thresholds, the robot's head will neither turn randomly too frequently nor remain on a single user for too long.
[0109] A valid head turn refers to a head turn in which the robot's head actually rotates and enters an attention-locked state. Specifically, this includes turning the robot's head towards the user corresponding to the current task after successful verification; and turning the robot's head towards the next user corresponding to the current task after successful verification. It does not include head turns that only require minor orientation corrections due to angle differences being less than a preset fine-tuning threshold, nor does it include fine-tuning that is not replanned because the change in target orientation does not exceed a preset update threshold.
[0110] In one specific implementation, the above time parameters can be dynamically adjusted according to the interaction scenario. The current user's minimum hold time T hold =clip(T hold 0 -a1×N wait +a2×S reply -a3×C next H min H max ); Target settling time T stable =clip(T stable 0 +b1×J dir -b2×C next ,S min ,S max ), where T hold 0 T is the initial reference value for the shortest retention time for the current user. stable 0 N is the initial reference value for the target settling time. wait S represents the number of tasks in the interactive task set that are in a waiting state. reply C represents the coefficient for the current response stage. next J dir H represents the fluctuation of the candidate direction within the continuous confirmation window, where a1, a2, a3, b1, and b2 are preset adjustment coefficients. min and H max These are the lower and upper limits of the current user's minimum retention time, respectively, S min and S maxThese are the lower and upper limits of the target settling time, respectively. clip() means that the calculation result is restricted to the preset upper and lower limits.
[0111] In one specific implementation, T hold 0 It can be set to 2.5 seconds, H min Set to 1.2 seconds, H max Set to 4 seconds. stable 0 It can be set to 0.6 seconds, S min Set to 0.3 seconds, S max Set to 1 second. a1, a2, and a3 can be set to 0.15, 0.8, and 0.5 respectively. b1 and b2 can be set to 0.4 and 0.3 respectively.
[0112] Correspondingly, when the number of waiting tasks N wait When an increase occurs, the minimum retention time of the current user is shortened, thus accelerating the rotation. When the robot is outputting a key response fragment, the current response stage coefficient S... reply Take a larger value to extend the retention time until the segment ends. When the location confidence C of the priority candidate user... next When the threshold is low, the minimum retention time for the current user is extended, and the target stabilization time is extended, thereby avoiding hasty switching when the next target is still uncertain.
[0113] The specific values of the above coefficients can be set by the implementer according to the actual situation. The current response stage coefficient is used to characterize the importance of the current stage of the robot's voice output. In a specific implementation, it can be valued as follows: 0 when the robot has no voice output, 0.5 when it is in the middle of a normal response, and 1 when it is at the beginning of a key response segment or when it is broadcasting an important prompt.
[0114] The directional backlash angle is used to create a non-sensitive zone near the locked target. For candidate speakers in a state of attention lock, even if the detected sound source direction fluctuates slightly, the robot will not adjust its orientation, thus avoiding continuous fine-tuning. For candidate speakers not yet locked, the change in their orientation within the continuous confirmation window must still be less than a preset fluctuation threshold for the candidate speaker to be considered stable. In one specific implementation, the directional backlash angle can be set to 5 to 15 degrees, for example, 10 degrees. The preset fluctuation threshold can be set to 5 to 12 degrees, for example, 8 degrees. The duration of the continuous confirmation window can be the same as the target stabilization time, or it can be 2 to 3 times that time.
[0115] The above-mentioned method constrains the head turning rhythm by combining four time parameters: minimum head turning interval, shortest current user holding time, maximum current task holding time, and target stabilization time. It also forms an insensitive zone by combining the direction backlash angle. This solves the problems of frequent head swaying following the instantaneous sound source direction when multiple people are speaking at the same time, as well as the interruption of the current interaction by a new sound source. This improves the stability of head movement and the completion rate of single-person interaction.
[0116] S4. When the current task meets the preset response completion conditions, is marked as invalid, or is preempted, if it is in the attention lock state, the attention lock state is released and the next current task is determined. If the current task is preempted, the preempted task is determined as the next current task. If the current task meets the preset response completion conditions or is marked as invalid, the next current task is selected from the remaining interactive tasks to be responded to in the interactive task set according to the task priority.
[0117] Specifically, releasing attention lock means the robot stops looking at the candidate speaking user corresponding to the current task, allowing their head orientation to be reassigned. Preset response completion conditions are a set of conditions used to determine when the current task has been successfully completed, and are used to determine when attention can be released. A task being marked as failed means that the current task no longer has value for continued response due to user departure, inability to locate, or cancellation; in this case, attention also needs to be released.
[0118] The task priorities are updated based on the current audio and image information. Then, the highest-priority task from the remaining pending interactive tasks in the task set is selected as the next current task. Tasks that are already completed or have expired are no longer considered in this sorting process.
[0119] In one specific embodiment, S4 includes the following steps:
[0120] S41. Before each scheduling, update the task priority of each interactive task to be responded to in the interactive task set, and sort the interactive tasks to be responded to that meet the validity conditions from high to low according to the task priority.
[0121] S42, if the voice text corresponding to the interactive task to be responded to in the interactive task set contains urgent semantics, or if the candidate speaking user corresponding to the interactive task to be responded to is a user with a preset permission level and issues a control command, then the interactive task to be responded to is determined to meet the preset preemption condition, the interactive task to be responded to is determined to be a preemption task, and the preemption task is determined to be the next current task.
[0122] S43. If the time since the last valid turn is less than the minimum turn interval, then a normal handover is not performed. Normal handover refers to the handover between pending interactive tasks other than preemptive tasks.
[0123] S44: Save the task state of the interactive tasks that were interrupted due to preemption and are pending response, other than the preemption task, and then re-add them to the interactive task set.
[0124] S45: When the current user finishes speaking, the robot completes its response, and the current task reaches its maximum holding time, it is determined that the current task meets the preset response completion conditions.
[0125] S46. When the candidate speaking user corresponding to the current task leaves the interaction area, the location of the candidate speaking user corresponding to the current task cannot be confirmed, the current task is canceled by the user, or the location confidence verification of the current task fails, the current task is marked as invalid.
[0126] The validity condition refers to the prerequisites that an interactive task awaiting a response must meet to participate in the ranking process. These conditions may include: the task is not marked as invalid, the candidate speaking user corresponding to the task is still within the interaction area, and the spatial orientation corresponding to the task is valid. The time since the last valid head turn is the duration from the moment the most recent actual head turn was completed to the current moment. Its value is compared with the minimum head turn interval to determine whether the current time is within the lockout period of the head turn rhythm. Normal switching refers to switching that occurs due to the completion of a normal response or normal rotation, as opposed to preemptive switching due to a preemptive task. The two are subject to different time thresholds: normal switching is constrained by the minimum head turn interval and the current user's shortest hold time, while preemptive switching is not subject to these constraints. Normal interactive tasks awaiting a response are those other than preemptive tasks.
[0127] Emergency semantics refers to semantic categories that indicate the need for immediate intervention. In one specific implementation, this can be determined through keyword matching or a text classification model. Keywords may include "alarm," "fire," "someone has fallen," "machine malfunction," etc. The text classification model can be any binary or multi-class classification model known in the art, such as a finely tuned short text classification model. The above implementation is merely illustrative, and those skilled in the art can use other models with the same emergency semantic recognition capabilities, as long as they can output a determination of whether an action is urgent or not. These alternative solutions all fall within the protection scope of this invention. As a specific example, when a candidate user's voice text is recognized as "someone has fallen over there," it can be determined that the task meets the preset preemption conditions.
[0128] Users with preset permission levels refer to users with permission levels higher than ordinary users. In one specific implementation, this can be determined based on user registration information, employee ID card recognition results, facial whitelist, or voiceprint whitelist. Permission levels can be divided into four levels: tourists, registered users, staff, and managers. Staff and above are considered to have preset permission levels.
[0129] Control commands are statements with explicit execution requirements, such as "Stop broadcasting", "Follow me to the service desk", and "Retrieve the queue information for window number three".
[0130] The above-described permission determination method is merely an illustrative example. Those skilled in the art can use other methods with the same permission differentiation capabilities to replace it, and these alternative solutions all fall within the protection scope of this invention.
[0131] The robot changes the status of the currently executing ordinary task to "interrupted," while saving the completed processing status, the progress already broadcast, and the original audio text. It then adds the task back to the interactive task set for the next round of sorting. Because preemption of ordinary tasks occurs infrequently, interrupted ordinary tasks are usually selected again quickly in the next round of sorting, allowing the remaining responses to be completed, thus ensuring that preempted ordinary tasks are not discarded.
[0132] The end of a user's speech can be determined based on the voice activity detection results. That is, if no new voice is heard after a preset silence confirmation period following the end of the candidate user's voice segment corresponding to the current task, the speech can be considered to have ended. In one specific implementation, the preset silence confirmation period can be set to 0.5 seconds to 1.5 seconds, for example, 0.8 seconds.
[0133] "Robot response complete" means that the robot has finished broadcasting its voice response for the current task and executed the corresponding actions. Reaching the maximum duration for the current task constitutes a fallback condition; even if the user's speech has not finished and the robot's response has not been completed, the process is forcibly initiated into a release and reselection phase to prevent a single user from occupying the task for an extended period.
[0134] The departure of a candidate speaking user corresponding to the current task from the interaction area can be determined based on lost face tracking, the user's bounding box moving out of the interaction area, or the user not being detected again within a preset failure time. "Location cannot be confirmed" means that relocation of the current user failed. The cancellation of the current task by the user can be determined based on a clear cancellation voice or the user actively leaving the area.
[0135] The above approach addresses the lack of clear switching opportunities and rules between multiple interactive tasks awaiting response by updating and sorting task priorities before each scheduling, constraining normal switching with time thresholds, reserving preemption channels with strict trigger conditions, and saving the state of interrupted normal tasks before re-enqueuing them. This improves the orderliness of the multi-person rotation response process and ensures that urgent requests can be responded to in a timely manner.
[0136] S5, update the spatial location of the next candidate speaking user corresponding to the current task, and verify the location confidence.
[0137] There is a waiting period between the creation of the task and the selection of the next candidate speaker for the current task. During this waiting period, the user may have moved. If the robot turns its head directly according to the old orientation recorded when the task was created, the robot's head will face the wrong direction, requiring correction or even repeated corrections. Therefore, the user's current spatial orientation must be re-acquired before the actual head turn is performed.
[0138] Location confidence measures the reliability of the updated spatial orientation, and its value can be normalized to between 0 and 1. A higher location confidence indicates a more reliable positioning result, making it more suitable for a turn. A low location confidence indicates that the user's location is uncertain, and the turn should be abandoned and the task marked as invalid.
[0139] In one specific embodiment, S5 includes the following steps:
[0140] S51 updates the spatial location of the next candidate speaking user corresponding to the current task by at least one of sound source localization, face tracking, human body tracking or voiceprint matching.
[0141] S52, based on the deviation between the updated spatial orientation and the original spatial orientation, and the matching confidence corresponding to the updated spatial orientation, calculate the location confidence of the candidate speaking user corresponding to the next current task.
[0142] S53, if the candidate speaking user corresponding to the next current task has left the interaction area or cannot be located, then mark the next current task as invalid.
[0143] Among them, sound source localization refers to the method of re-estimating the direction of the incoming wave from the current speaker using a sound acquisition device, which is applicable when the user is speaking. Face tracking and human body tracking refer to the method of maintaining the same target number and outputting its position between consecutive image frames using an image acquisition device, which is applicable when the user is not speaking but is still within the field of vision. Voiceprint matching refers to the method of comparing voiceprint features with temporally adjacent speech events to confirm that two speech events come from the same user, which is applicable to distinguishing the sound source when multiple people speak at the same time. Those skilled in the art will know that the above specific tracking and matching algorithms can be implemented using any target tracking algorithm or speaker confirmation algorithm known in the art, such as detection-based tracking paradigms or correlation filtering tracking algorithms. The above algorithm selection is only for illustrative purposes, and those skilled in the art can use other algorithms with the same trajectory maintenance capability as alternatives, as long as they can output the spatial orientation of the candidate speaker. These alternative solutions all fall within the protection scope of this invention.
[0144] Match confidence is a reliability value given by the localization or matching result itself, such as the tracking confidence given by face tracking, the similarity score given by voiceprint matching, or the spatial spectrum peak significance given by sound source localization.
[0145] In one specific implementation, the location confidence level can be calculated using the following formula: Conf i =w p ×S match +w d ×(1-min(D i / D max ,1)), where S match To match the confidence level, D i D represents the distance or angle deviation between the updated and unupdated spatial orientations. max To preset the maximum permissible deviation, min() is the function that takes the minimum value, w p w d Assuming a preset weighting coefficient and the sum of the two is 1, in this embodiment, w p It can be set to 0.6, w d Set it to 0.4.
[0146] The above-described method for calculating location confidence is merely an illustrative example. Those skilled in the art can use other monotonic functions that can simultaneously reflect matching confidence and the degree of location jump, and these alternatives all fall within the protection scope of this invention.
[0147] If the spatial orientation deviation before and after the update is too large, it may indicate that the user has indeed moved, or it may indicate that the tracking target has been misassociated, i.e., another user has been mistaken for the target user. Therefore, this embodiment introduces the deviation before and after the location update as a negative factor to penalize misassociations. Introducing the matching confidence score as a positive factor can reward high-quality positioning results. The location confidence score obtained by combining the two factors reflects both the accuracy of the positioning itself and whether the positioning result is consistent with the old location.
[0148] The above-mentioned method solves the problem of the robot turning to the old position due to user movement during the waiting period by re-acquiring the spatial orientation of the next candidate speaking user corresponding to the current task before each turn and calculating the position confidence by combining the deviation before and after the update with the matching confidence. This improves the hit rate of a single turn and reduces repeated correction actions caused by target deviation.
[0149] S6, if the updated spatial orientation and position confidence meet the preset interaction conditions, then control the robot's head to turn towards the candidate speaker based on the updated spatial orientation, enter the attention lock state and execute the interaction response, then return to execute S4. If the updated spatial orientation or position confidence does not meet the preset interaction conditions, mark the next current task as invalid, return to execute S4, until there are no pending interaction tasks in the interaction task set that meet the preset interaction conditions, then exit the multi-person speaking scheduling state.
[0150] The preset interaction conditions determine whether a turn is actually executed; a turn can only occur if all conditions are met simultaneously. If the conditions are not met, instead of retrying, the next current task is marked as invalid, and the process returns to S4 to select the next current task. This forms a loop structure of selection, verification, turn if successful, and removal and reselection if unsuccessful, until the set of interactive tasks is cleared or there are no more tasks that meet the preset interaction conditions. After exiting the multi-person speech scheduling state, the robot can return to the normal single-target sound source following mode or standby orientation.
[0151] In one specific embodiment, S6 includes the following steps:
[0152] S61, calculate the horizontal angle difference and pitch angle difference based on the robot's current head angle and the target angle corresponding to the updated spatial orientation.
[0153] S62 generates a smooth head-turning trajectory based on the horizontal angle difference and pitch angle difference, under the constraints of the robot's maximum angular velocity and maximum angular acceleration.
[0154] S63, if the absolute values of the horizontal angle difference and the pitch angle difference are both less than the preset fine-tuning threshold, then keep the robot head still or make a small correction to the orientation of the robot head.
[0155] S64, during the head-turning process, if the target orientation corresponding to the target angle changes and the change exceeds the preset update threshold, then update the target angle and return to execute S61.
[0156] S65, if the updated spatial orientation is within the interaction area, the position confidence is greater than or equal to the preset confidence threshold, and the next current task is not marked as invalid, then the preset interaction conditions are met, and a head-turning control command is generated based on the updated spatial orientation and maximum angular velocity to control the robot's head to turn towards the candidate speaking user corresponding to the next current task.
[0157] The robot's current head angle is the current orientation obtained from feedback by the head joint encoder or motor, and can be represented by two components: the horizontal joint angle and the pitch joint angle. The target angle is the desired orientation after the updated spatial orientation is mapped onto the robot's head joint space, and also includes two components: the target horizontal angle and the target pitch angle. The angle difference refers to the difference between the current head angle and the target angle, corresponding to the horizontal angle difference and the pitch angle difference, respectively.
[0158] Maximum angular velocity refers to the maximum angular velocity allowed for head rotation, and maximum angular acceleration refers to the maximum angular acceleration allowed for head rotation. These two parameters constitute the kinematic envelope of the rotation process, used to avoid mechanical shock and make the rotation appear natural. The specific values of the kinematic parameters can be set by the implementer according to the actual situation, such as based on the robot head's moment of inertia and motor capabilities. In one specific implementation, the maximum angular velocity can be set from 60 degrees per second to 120 degrees per second, for example, 90 degrees per second. The maximum angular acceleration can be set from 120 degrees per square second to 300 degrees per square second, for example, 200 degrees per square second.
[0159] A smooth turning trajectory is an angle sequence that gradually transitions from the current angle to the target angle while satisfying the aforementioned kinematic constraints. In one specific embodiment, trapezoidal velocity planning can be used, that is, generating the trajectory by combining acceleration, constant speed, and deceleration segments. S-shaped velocity planning can also be used to further reduce start-stop shock. The above trajectory planning methods are merely illustrative examples, and those skilled in the art can use other trajectory generation methods with the same smoothing effect as alternatives, as long as they can output an angle sequence that satisfies the kinematic constraints. These alternative solutions all fall within the protection scope of this invention.
[0160] When the difference between the target and the current orientation is small, performing a full head turn is neither necessary nor will it introduce perceptible jitter; therefore, remaining still or making only minor adjustments is sufficient. In one specific implementation, the preset fine-tuning threshold can be set to 3 to 8 degrees, for example, 5 degrees.
[0161] The preset update threshold is used to determine whether the trajectory needs to be recalculated during execution. Correspondingly, if the target user moves slightly during the turn, frequent recalculation will cause the trajectory to jitter. If the target user moves significantly, continuing to turn towards the original endpoint will result in deviation. Therefore, the process only returns to S61 for recalculation when the change in the target's orientation exceeds the preset update threshold.
[0162] The specific values of each threshold can be set by the implementer according to the actual situation. In one specific implementation, the preset update threshold can be set to 5 degrees to 15 degrees, for example, 10 degrees. The preset confidence threshold is used to determine whether the location confidence level has reached a level that allows turning the head, and can be set to 0.5 to 0.8, for example, 0.65.
[0163] The preset interaction conditions include: the spatial orientation is still within the interaction area, the location confidence is greater than or equal to the preset confidence threshold, and the next current task has not been marked as invalid. All three conditions must be met simultaneously for the preset interaction conditions to be considered satisfied.
[0164] The above-mentioned methods solve the problems of stiff head turning and repeated replanning due to slight target movement by generating smooth head turning trajectories under kinematic envelope constraints and setting fine-tuning dead zones and target replanning thresholds, thereby improving the naturalness and stability of robot head movements. By performing gate judgment on spatial orientation, position confidence and task status before head turning, the problem of erroneous head turning on invalid or untrusted targets is solved, thereby improving the effectiveness of each head turn.
[0165] In one specific embodiment, the above method can also be organized in the form of a state machine, with the robot corresponding to the following states: idle state, current user locked state, queue update state, preemption judgment state, next task selection state, target relocation state, and head-turning execution state. Specifically, in the idle state, a task is created and the current user is selected after a valid speaking event is detected. In the current user locked state, the robot remains facing the current user. In the queue update state, when a new sound source is detected, only the waiting tasks are updated without immediately turning the robot. In the preemption judgment state, it is determined whether there is a preemption task that meets the preset preemption conditions. In the next task selection state, after unlocking, the next candidate speaking user is selected according to task priority. In the target relocation state, the spatial orientation and position confidence of the next candidate speaking user are reconfirmed. In the head-turning execution state, the head is turned according to kinematic constraints. The above state division is only an illustrative example; those skilled in the art can use other process organization forms with the same control logic as alternatives, and these alternatives all fall within the protection scope of this invention.
[0166] In one specific embodiment, the robot executing the above method may further include a multi-person voice and orientation acquisition module, a speaking user identification module, a candidate user maintenance module, a head-turning interval control module, a response order scheduling module, an attention locking module, a head-turning execution module, and a task status update module. The multi-person voice and orientation acquisition module is used to acquire the voiceprint features of multiple people, the direction of the sound source, the sound intensity, and the start and end times of the speech. The speaking user identification module is used to distinguish between different candidate speaking users currently participating in the speech. The candidate user maintenance module is used to maintain the set of interactive tasks and perform task establishment, merging, and invalidation. The head-turning interval control module is used to limit the minimum time interval between two consecutive valid head turns and the maximum holding time of the current task. The response order scheduling module is used to determine the sequential response order of each candidate speaking user. The attention locking module is used to keep the robot facing the user until the current user has completed their response. The head-turning execution module is used to control the robot's head to turn towards the target user according to the scheduling results. The task status update module is used to update the task status and the last update time at each stage. The above module division is only an illustrative example; those skilled in the art can use other module division methods with the same functions as alternatives, and these alternatives all fall within the protection scope of this invention.
[0167] This embodiment enters the multi-person speaking scheduling state only when multiple valid speaking events occur within a preset time window or when there is overlap in speech time. It also establishes a separate interactive task for each candidate speaker, solving the problem of the robot only retaining one voice and losing requests from other users when multiple people speak simultaneously. By setting a head-turning time control strategy, the robot's head rotates at a controllable rhythm, maintaining attention lock while the current task is not completed, and updating the task without triggering a head turn when a new sound source appears. This solves the problems of frequent head swaying following instantaneous sound sources and interruptions to the current interaction. Furthermore, by updating the spatial orientation of the next candidate speaker for the current task and verifying the position confidence before each head turn, and marking the next current task as invalid and reselecting it if the preset interaction conditions are not met, the embodiment solves the problem of the robot turning back to its old position due to user movement during the waiting period, improving the head-turning hit rate and interaction continuity.
[0168] Example 2
[0169] Embodiment 2 of the present invention provides a non-transitory computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the robot attention scheduling method provided in the above embodiment.
[0170] Example 3
[0171] Embodiment 3 of the present invention provides an electronic device, which includes a processor and the non-transitory computer-readable storage medium of Embodiment 2 of the present invention.
[0172] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A robot attention scheduling method, characterized in that, The method includes the following steps: S1, Based on the audio and image information within the interactive area, identify the candidate speaking users within the interactive area; S2, when at least two candidate speaking users are detected to have overlapping speech times, or when at least two speaking events that meet the preset valid conditions occur within the preset time window, the multi-person speaking scheduling state is entered, and corresponding interactive tasks to be responded to are established for each candidate speaking user and the task priority is calculated to obtain an interactive task set; S3, according to the preset head-turning time control strategy and the task priority, determine a current task that has passed the position confidence verification from the interactive task set, and control the robot head to turn towards the candidate speaking user corresponding to the current task, enter the attention lock state and execute the interactive response. In the attention lock state, if a new sound source is detected, only the interactive task set is updated and the head turning is not triggered. If there is an interactive task waiting to be responded to in the interactive task set that meets the preset preemption conditions, the attention lock state is released and S4 is executed. S4, when the current task meets the preset response completion condition, is marked as invalid or preempted, if it is in the attention lock state, the attention lock state is released and the next current task is determined. If the current task is preempted, the preempted task is determined as the next current task. If the current task meets the preset response completion condition or is marked as invalid, the next current task is selected from the remaining interactive tasks to be responded to in the interactive task set according to the task priority. S5, update the spatial location of the next candidate speaking user corresponding to the current task, and verify the location confidence; S6, if the updated spatial orientation and the position confidence score meet the preset interaction conditions, then control the robot head to turn towards the candidate speaking user according to the updated spatial orientation, enter the attention lock state and execute the interaction response, and return to execute S4; if the updated spatial orientation or the position confidence score does not meet the preset interaction conditions, mark the next current task as invalid, and return to execute S4; until there are no pending interaction tasks in the interaction task set that meet the preset interaction conditions, exit the multi-person speaking scheduling state.
2. The robot attention scheduling method according to claim 1, characterized in that, S1 includes the following steps: S11, Acquire audio information within the interactive area through a sound acquisition device, wherein the audio information includes the direction of the sound source, sound intensity, and duration of the speech activity; S12, image information within the interactive area is acquired through an image acquisition device, wherein the image information includes at least one of face information, human body position information, and mouth movement information; S13, receive speaker identification information output by an external speech processing module, wherein the speaker identification information includes speaker identification, speech text, overlapping speech markers and speaker recognition confidence; S14, based on the audio information, the image information and the speaker identification information, identify the candidate speaking users in the interaction area, and determine the spatial location and tracking identifier corresponding to each candidate speaking user, wherein the tracking identifier is used to associate the speaking events of the same candidate speaking user that are adjacent in time when the user identifier cannot be obtained.
3. The robot attention scheduling method according to claim 1, characterized in that, Determining whether the speech event meets the preset valid conditions includes the following steps: S211, obtain the speech duration, sound intensity, speech recognition confidence, and speech text corresponding to the speech events generated by the candidate speaking users; S212, if the duration of the speech is greater than or equal to a preset duration threshold and the sound intensity is greater than or equal to a preset intensity threshold, then the speech event is determined as a candidate speech event; S213, if the speech recognition confidence level corresponding to the candidate speech event is greater than or equal to a preset confidence threshold, or if the speech text corresponding to the candidate speech event contains a preset wake-up word or a preset interaction keyword, then the candidate speech event is determined as a speech event that meets the preset valid conditions.
4. The robot attention scheduling method according to claim 2, characterized in that, Create corresponding interactive tasks to be responded to for each candidate speaking user, including the following steps: S221, when a candidate speaking user is identified to produce a speaking event that meets the preset valid conditions, a pending interaction task is established or updated for the candidate speaking user. The pending interaction task includes a task identifier, a user identifier or the tracking identifier, a speaker identifier, a sound source direction, a spatial orientation, a sound intensity, a speech start and end time, a speech text, a waiting time, a task status, a priority score, whether it is in an overlapping speech interval, and a last update time. The task status includes pending confirmation, pending response, currently locked, interrupted, completed, and invalid. S222, if the time interval between adjacent speaking events of the same candidate speaking user is less than the preset merging time, the azimuth difference is less than the preset merging angle threshold, and the semantics belong to the same round of requests, then the new speaking event is merged into the existing pending interaction task of the candidate speaking user. S223, If different candidate users speak at the same time, then create or update the corresponding interactive task to be responded to for each candidate user. S224, if the interactive task to be responded to does not detect the corresponding candidate speaking user within a preset failure time, or if the corresponding candidate speaking user leaves the interaction area or fails to relocate, then the interactive task to be responded to is marked as failed and exits the interactive task set.
5. The robot attention scheduling method according to claim 2, characterized in that, Calculating task priorities involves the following steps: S231, For any candidate speaking user, based on the number of times the candidate speaking user has continuously established pending interaction tasks and the cumulative non-service duration, update the non-response compensation score corresponding to the candidate speaking user according to a monotonically increasing function. S232, obtain the priority parameters corresponding to the candidate speaking user and perform normalization processing on them respectively. The priority parameters include the speaking order score, waiting time score, semantic urgency score, user permission score, voice validity score, updated non-response compensation score, turning cost score, and low confidence penalty score. The turning cost score is determined based on the turning angle between the robot's current orientation and the candidate speaking user. The low confidence penalty score is determined based on the position confidence and speaker recognition confidence of the candidate speaking user. S233, the normalized priority parameters are weighted and summed according to the preset weights to obtain the task priority of the interactive task to be responded to for the candidate speaking user. The weights corresponding to the turning cost score and the low confidence penalty score are negative weights. S234. Sort the interactive tasks to be responded to in the interactive task set in descending order of task priority.
6. The robot attention scheduling method according to claim 1, characterized in that, The preset head-turning time control strategy includes: S31, obtain the minimum head-turning interval time, the current user's shortest holding time, the current task's maximum holding time, and the target stabilization time; S32, based on the minimum head-turning interval, the current user's shortest holding time, the current task's maximum holding time, and the target stabilization time, control the head-turning rhythm of the robot. Specifically, outside of interactive tasks awaiting response that meet the preset preemption conditions, after the robot completes one effective head turn, it does not turn its head again due to a new sound source within the minimum head-turning interval. Furthermore, after entering the attention-locked state, the robot maintains the attention-locked state within the current user's shortest holding time. If the current task is not completed even after exceeding the current task's maximum holding time, reassess whether to switch to other interactive tasks awaiting response in the interactive task set. When a candidate speaking user corresponding to an interactive task awaiting response in the interactive task set continuously exists or is continuously confirmed within the target stabilization time, the interactive task awaiting response corresponding to that candidate speaking user is allowed to be designated as the next current task. S33, set a direction backlash angle for the robot head. For candidate speaking users in the attention-locked state, when the detected sound source direction fluctuates within the direction backlash angle range, keep the robot head facing the same direction. For candidate speaking users not in the attention-locked state, the stability condition corresponding to the target stabilization time is met only when the change in the sound source direction of the candidate speaking user within the continuous confirmation window is less than the preset fluctuation threshold.
7. The robot attention scheduling method according to claim 6, characterized in that, S4 includes the following steps: S41. Before each scheduling, update the task priority of each interactive task to be responded to in the interactive task set, and sort the interactive tasks to be responded to that meet the validity conditions from high to low according to the task priority. S42, if the voice text corresponding to the interactive task to be responded to in the interactive task set contains urgent semantics, or if the candidate speaking user corresponding to the interactive task to be responded to is a user with a preset permission level and issues a control command, then it is determined that the interactive task to be responded to meets the preset preemption condition, and the interactive task to be responded to is determined as a preemption task. S43, if the time since the last valid turn is less than the minimum turn interval, then a normal handover is not performed, wherein the normal handover is the handover between pending interactive tasks other than preemptive tasks; S44, save the task status of the interactive tasks that were interrupted due to the preemption task and are not related to the preemption task, and then add them back to the interactive task set. S45, when the current user finishes speaking, the robot completes its response, and the current task reaches its maximum holding time, it is determined that the current task meets the preset response completion condition; S46. When the candidate speaking user corresponding to the current task leaves the interaction area, the location of the candidate speaking user corresponding to the current task cannot be confirmed, the current task is canceled by the user, or the location confidence verification of the current task fails, the current task is marked as invalid.
8. The robot attention scheduling method according to claim 1, characterized in that, S6 includes the following steps: S61, calculate the horizontal angle difference and pitch angle difference based on the robot's current head angle and the target angle corresponding to the updated spatial orientation; S62, under the constraints of the robot's maximum angular velocity and maximum angular acceleration, generate a smooth head-turning trajectory based on the horizontal angle difference and the pitch angle difference; S63, if the absolute values of the horizontal angle difference and the pitch angle difference are both less than the preset fine-tuning threshold, then keep the robot head still or make a small correction to the orientation of the robot head. S64, during the head-turning process, if the target orientation corresponding to the target angle changes and the change exceeds the preset update threshold, then update the target angle and return to execute S61; S65, if the updated spatial orientation is within the interaction area, the position confidence is greater than or equal to the preset confidence threshold, and the next current task is not marked as invalid, then it is determined that the preset interaction conditions are met, and a head-turning control command is generated according to the updated spatial orientation and the maximum angular velocity to control the robot head to turn towards the candidate speaking user corresponding to the next current task.
9. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the robot attention scheduling method as described in any one of claims 1-8.
10. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 9.
