A robot intelligent voice broadcasting method and system for a complex visual scene

Abstract

The application provides a robot intelligent voice broadcasting method and system for a complex visual scene, and the method comprises the following steps: collecting environment images and recognizing targets through a target detection model to obtain target features of each target; performing consistency verification on each target through space-time joint analysis, and if the target meets the consistency verification condition, the target is added to a stable target set; calculating a threat score of each target in the stable target set and determining a threat level; determining an azimuth angle and a distance classification of each target in the stable target set, and determining a position of the target through the azimuth angle; judging whether a broadcasting trigger mechanism is met or not, and if the broadcasting trigger mechanism is met, generating voice according to the threat level and broadcasting according to a dynamic broadcasting interval control mechanism, and if the broadcasting trigger mechanism is not met, not broadcasting. The application solves the contradiction between the richness of a visual scene and the limitation of voice expression, and optimizes the information broadcasting efficiency and accuracy of a robot in a dynamic environment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot voice interaction technology, specifically a robot intelligent voice broadcasting method and system for complex visual scenarios. Background Technology

[0002] With the rapid development of robotics technology, its application in complex visual scenarios such as security patrols, guide assistance for the visually impaired, intelligent services, and industrial inspections is becoming increasingly widespread. These scenarios typically feature rich visual information, rapid dynamic changes, and diverse target types, requiring robots to accurately and in real-time transmit visual perception results to users via voice broadcast. However, existing robot voice broadcasting systems generally suffer from the following technical problems when handling complex visual scenarios:

[0003] 1. The contradiction between the richness of visual scenes and the limitations of voice expression. Robot visual perception systems can capture a large amount of environmental data and identify various targets and their spatial positions, motion states, etc. However, voice broadcasting is limited by its small information capacity and simple expression form, making it difficult to completely and accurately convey the complexity of visual scenes, resulting in users having difficulty quickly understanding the scene situation.

[0004] 2. High false alarm rate, impacting user experience. Traditional broadcasting systems often rely on single-frame images for target detection and broadcasting, making them prone to false alarms due to factors such as lighting changes, occlusion, and motion blur. This severely interferes with user judgment. Although some systems attempt to reduce false alarms through simple filtering or threshold settings, they lack effective utilization of continuous frame information, resulting in limited false alarm suppression.

[0005] 3. Lack of target prioritization and spatial location cues. In complex scenarios, the threat levels of different targets vary significantly. For example, a rapidly approaching pedestrian is more threatening than a stationary object and should be prioritized for alerting. However, existing systems typically use a "first-come, first-served" or fixed-order approach, failing to consider the dynamic threat level of the target. Furthermore, the lack of spatial location cues in voice broadcasts makes it difficult for users to quickly locate targets using voice information, reducing the practical value of voice broadcasts.

[0006] 4. Improper control of broadcast frequency, leading to information overload or omissions. Some systems employ high-frequency broadcast strategies to avoid missing information, resulting in voice information overload and making it difficult for users to distinguish important information; while other systems set excessively long broadcast intervals to avoid interference, potentially missing key events. Existing technology lacks a mechanism for dynamically adjusting the broadcast frequency, making it difficult to achieve a balance between information integrity and user experience.

[0007] 5. Lack of intelligent perception of changes in target quantity. In dynamic scenarios, the quantity of targets may change frequently. Existing systems often broadcast the changes for each one, resulting in excessive redundant information.

[0008] To address the aforementioned issues, and given the current lack of a method to balance accurate information filtering and efficient voice delivery in complex visual scenarios, this invention proposes a robot intelligent voice broadcasting system and method based on multi-strategy optimization for complex visual scenarios. Summary of the Invention

[0009] The purpose of this invention is to propose a robot intelligent voice broadcasting method and system for complex visual scenes, which resolves the contradiction between the richness of visual scenes and the limitations of voice expression, and optimizes the efficiency and accuracy of robot information broadcasting in dynamic environments.

[0010] The technical solution to achieve the purpose of this invention is as follows:

[0011] A method for intelligent voice broadcasting of robots in complex visual scenes includes the following steps:

[0012] The system acquires environmental images and identifies targets using a target detection model, obtaining the target features of each target.

[0013] Through spatiotemporal joint analysis, consistency verification is performed on each target. If a target meets the consistency verification conditions, it is added to the stable target set.

[0014] Calculate the threat score for each target in the stable target set and determine the threat level;

[0015] Determine the azimuth and range classification of each target in the stable target set, and determine the target's location using the azimuth.

[0016] Determine if the current condition meets the broadcast trigger mechanism. If it does, generate a voice message based on the threat level and broadcast it according to the dynamic broadcast interval control mechanism. If the condition does not meet the requirements, do not broadcast.

[0017] Furthermore, the target detection model includes a Yolov8 model, a multi-target tracking module, and an output module, wherein the Yolov8 model outputs a target detection bounding box. ,category and confidence level The target detection box The corresponding RGB image input multi-target tracking module first uses a lightweight ReID network to extract the target appearance feature vector, then calculates the matching index, and then uses the Hungarian algorithm to perform target matching. If the matching is successful, the motion state and the number of consecutive occurrences of the target are updated, and the features of the target are output through the output module. If the matching is unsuccessful, the target is temporarily stored or deleted.

[0018] The matching metric is:

[0019] ;

[0020] ;

[0021] in, To match the metrics, For feature cosine similarity, , for the target of extraction In time The target feature vector, The IoU distance between the detection box and the prediction box. The coefficient is set.

[0022] Furthermore, the output module outputs the following characteristics of the target:

[0023] ;

[0024] in, For the goal A unique tracking identifier, For the goal Category tags, For the goal In time 2D detection box coordinates , Target 3D spatial coordinates For the goal Velocity components in 3D space For the goal The number of consecutive frames, For the goal The detection confidence level, Let t be the target quantity.

[0025] Furthermore, the consistency verification is performed on each target. In the time window Cross-frame consistency analysis is performed, including temporal correlation checks, trajectory continuity checks, and visual and motion feature consistency checks. Targets satisfying all three checks are designated as stable targets and added to the stable target set.

[0026] The temporal correlation check is as follows:

[0027] ;

[0028] ;

[0029] in, The average confidence threshold of the joint matching cost within the time window. The threshold for allowing fluctuations in the joint matching cost within the time window. , Target In the time window Set of internal matching indicator sequences;

[0030] The trajectory continuity check is as follows: if consecutive frames appear... If so, the check is satisfied. The set frame rate threshold;

[0031] The consistency check between visual and motion features is as follows:

[0032] ;

[0033] in, The displacement of the center point of the detection box between adjacent frames of the target. For the detection boxes between adjacent frames , For indicator functions, Displacement threshold Cumulative frame threshold Target Category Adaptive threshold adjustment, when If so, the visual and motion feature consistency check is satisfied.

[0034] Furthermore, the target's threat score is:

[0035] ;

[0036] in, , , , The weighting coefficients are for category, speed, distance, and stability. For predefined target categories The corresponding value, Let i be the velocity of target i. This represents the maximum speed. The distance between the target and the robot body. For the specific minimum value set, This represents the maximum value within the time window.

[0037] right Perform Min-Max normalization and fall into The range; the threat level is determined as follows:

[0038] like If the value is ≥ 0.7, the threat level is high-risk target, and the corresponding voice field is emergency.

[0039] If 0.4 ≤ If the value is less than 0.7, the threat level is medium-risk and the corresponding voice field is a warning.

[0040] like If the value is less than 0.4, the threat level is low-risk, and the corresponding voice field is a prompt.

[0041] Furthermore, based on the azimuth angle, the target's location is determined as front, right front, right side, right rear, rear, left rear, left side, or left front, and the distance classification is as follows:

[0042] ;

[0043] in, The distance between the target and the robot body.

[0044] Furthermore, the broadcast triggering mechanism is as follows:

[0045] If the condition is met, a broadcast will be triggered; otherwise, no broadcast will be triggered. Let be the target quantity at time t. This is the sensitivity coefficient.

[0046] Furthermore, the generation of voice based on threat level specifically includes:

[0047] For high-risk targets, the generated voice includes urgency, location, number of targets, target type, and distance classification; for medium- and low-risk targets, the generated voice includes location, target type, and distance classification.

[0048] Furthermore, the dynamic broadcast interval control mechanism is as follows: the timing and interruption of voice output are controlled through a dynamic priority queue based on threat levels and a preemptive broadcast mechanism; different broadcast intervals are applied to targets with different threat scores. The broadcast interval is specified. for

[0049] ;

[0050] in, The set basic interval.

[0051] A robot intelligent voice broadcasting system for complex visual scenes includes:

[0052] The scene understanding module is used to acquire environmental images and identify targets through a target detection model, obtaining the target features of each target;

[0053] The temporal consistency verification module performs consistency verification on each target through spatiotemporal joint analysis. If a target meets the consistency verification conditions, it is added to the stable target set.

[0054] Dynamic Threat Classification Decision Module: Used to calculate the threat score for each target in the stable target set and determine the threat level;

[0055] The spatial semantic mapping module determines the azimuth and distance classification of each target in the stable target set. The azimuth determines the location of the target, and the output is a target set with azimuth-distance classification and threat score semantics.

[0056] The report trigger module sets the broadcast trigger mechanism to determine whether to broadcast;

[0057] The dynamic semantic aggregation module generates the voice message to be broadcast based on the threat level.

[0058] The dynamic broadcast module is used to design a dynamic broadcast interval control mechanism and broadcast according to the dynamic broadcast interval control mechanism.

[0059] Compared with the prior art, the present invention has the following significant advantages:

[0060] 1. This invention associates cross-frame detection results with target IDs and combines geometric and motion features to determine target stability, significantly reducing the false alarm rate and improving the accuracy of voice broadcasting;

[0061] 2. This invention dynamically calculates threat scores and broadcasts them in a tiered manner by integrating target category, movement speed, relative distance, and temporal stability characteristics. Based on the threat score, the broadcast priority is dynamically adjusted to ensure that high-risk targets (such as those rapidly approaching vehicles) are reached first, resulting in a significant improvement in response speed and adaptive threat broadcasting. Through preemptive queue management, high-threat commands immediately interrupt low-priority broadcasts, significantly reducing the delay in reaching critical events.

[0062] 3. This invention dynamically generates semantic location-distance descriptions by fusing threat level and relative azimuth, helping users quickly locate targets and adapting to application scenarios and tasks.

[0063] 4. This invention generates incremental natural language commands by jointly analyzing target categories, threat levels, and status changes within a directional partition, avoiding redundant information and improving user experience.

[0064] 5. This invention improves the comfort of human-computer interaction by controlling the timing of voice command transmission through a threat level-adaptive cooldown timer and priority preemption strategy. Attached Figure Description

[0065] Figure 1 This is a block diagram of the overall system architecture of the present invention.

[0066] Figure 2 This is a schematic diagram illustrating the threat assessment calculation of the present invention.

[0067] Figure 3 This is a schematic diagram of the broadcast priority queue of the present invention.

[0068] Figure 4 This is a schematic diagram of an urban sidewalk collected for an embodiment of the present invention. Detailed Implementation

[0069] Combination Figure 1 This invention provides a robot intelligent voice broadcasting system for complex visual scenes, comprising:

[0070] (1) Scene Understanding Module

[0071] It acquires environmental images in real time and identifies targets, extracting features such as target type, location, and motion state. Depending on the device's sensor type, RGB images are typically available. Depth map LiDAR point cloud Depending on the specific scenario requirements, an end-to-end multimodal detection scheme or a cascaded fusion scheme can be adopted. This invention uses a cascaded scheme that prioritizes real-time performance as an example.

[0072] The Yolov8 model is used to detect targets, and the target detection bounding boxes are output. and categories (e.g., people, vehicles, stairs, tables, chairs, doors, windows, obstacles, etc.), and output the confidence level. To correlate targets across frames and eliminate detection jitter, a multi-target tracking module is introduced, with the specific process as follows:

[0073] 1. Detection box input: The detection box output by YOLOv8 And the corresponding RGB image region input tracker.

[0074] 2. Appearance Feature Extraction: A lightweight ReID network (such as MobileNetV3) is used to extract the target appearance feature vector. .

[0075] 3. Data Association:

[0076] Motion matching: Predict the target's position in the next frame based on Kalman filtering, and calculate the IoU distance between the detection box and the predicted box. .

[0077] Appearance matching: Calculate feature cosine similarity .

[0078] Combined matching: Integrates motion and appearance information, matching tracking ID:

[0079]

[0080] For the set coefficients, the system follows The size is matched and allocated using the Hungarian algorithm to obtain the optimal cross-frame association result.

[0081] Status update: For those based on joint matching metrics Once a target is determined to be a successful match, its motion state (velocity, acceleration) and the number of consecutive frames it appears in are updated. For unmatched targets, the process proceeds to temporary storage or deletion according to the tracker's policy.

[0082] By combining depth maps or LiDAR point clouds, 2D detection boxes are mapped to 3D space, outputting a complete feature set for each target for use by subsequent modules.

[0083]

[0084] As shown in the formula above For the goal A unique tracking identifier, For the goal Category labels (such as people, vehicles, obstacles, etc.). For the goal In time 2D detection box coordinates , Target 3D spatial coordinates (calculated based on depth map or LiDAR). For the goal The velocity components in 3D space (which can be estimated using Kalman filtering). For the goal The number of consecutive frames, For the goal The detection confidence level.

[0085] This is the target category label, which will be used as the Name field in the subsequent voice broadcast module for filling in the voice template.

[0086] (2) Timing consistency verification module

[0087] Output-based band The target sequence, through spatiotemporal joint analysis, for each target In the time window Cross-frame consistency analysis is performed to verify the authenticity and stability of the target and suppress transient false detections.

[0088] Temporal correlation check:

[0089] A unique target identifier assigned by the multi-target tracking module. The detection results of the current frame and historical frames are verified at the target level to ensure the consistency of target identity during cross-frame analysis and avoid the problem of mis-association or identity drift of targets due to target occlusion, similar appearance or detection jitter.

[0090] For the target During the time window Within, record its historical joint matching cost sequence:

[0091] Calculate the mean and volatility of the cost:

[0092]

[0093] The target is deemed to have passed the temporal correlation check when the following conditions are met:

[0094]

[0095] in The "average trust threshold" for the joint matching cost within a time window is used to determine whether a target ID is stably and reliably associated overall. The "fluctuation allowable threshold" for the joint matching cost within the time window is used to determine whether the association process of the ID is smooth and whether identity drift has occurred. If the above relationship is not satisfied, the target is considered to have identity drift risk and will not be included in the subsequent stable target set;

[0096] Trajectory continuity check: Utilizing the number of consecutively occurring frames maintained by the multi-target tracking module. Filter out targets that appear only briefly (such as If the trajectory continuity check is met, the subsequent check will proceed; otherwise, the target will not be included in the subsequent stable target set.

[0097] Spatial consistency calculation: Calculate the detection boxes between adjacent frames. and center point displacement :

[0098]

[0099]

[0100] Dynamic threshold adjustment: based on target category Adaptive adjustment of thresholds (e.g., pedestrians) ,vehicle ).

[0101] Based on a combination of visual and motion features, the consistency verification criteria are as follows:

[0102]

[0103] in This is an indicator function (it takes the value 1 if the condition is true, and 0 otherwise). Displacement threshold This is the cumulative frame threshold. When... When this happens, it is classified into the set of stable targets.

[0104] (3) Dynamic Threat Classification Decision Module

[0105] Stable target set based on output Dynamically calculate each target by integrating multi-dimensional features Based on the threat score, the broadcast priority is dynamically adjusted, combined with Figure 2 The threat score is calculated using the following formula:

[0106]

[0107] in, , , , The weighting coefficients for category, speed, distance, and stability are configured according to scenario requirements. The final component is the stability compensation term, indicating that long-term stable targets (such as pedestrians continuously approaching) require an increased threat level even at low speeds, and the broadcast queue is adjusted accordingly. Sort in descending order.

[0108] Specifically, the category score, speed item, distance item, and stability item have all been normalized, and their value ranges are limited to [value range missing]. At the same time, the weights that make up the threat score satisfy... ,therefore The final result strictly falls into The interval. Furthermore, to adapt to dynamic scenarios, this invention can optionally adjust the original threat score. Perform Min-Max adaptive normalization to make The distribution is more stable, thus maintaining the consistency of the broadcast strategy in multi-person, multi-target scenarios.

[0109] (4) Spatial semantic mapping module

[0110] Based on the output's hierarchical threat target set, the target's 3D location is determined. The location is converted into a location description that includes threatening semantics and is understandable to humans. The specific implementation is as follows:

[0111] Azimuth calculation:

[0112]

[0113] The announcements are broadcast according to eight directional intervals: directly in front, directly in front right, directly in front right, directly in front right, directly in front right, directly in front right, directly in front left, directly in front left, and directly in front left. This is based on the azimuth angle. The corresponding discrete direction interval can generate a target location description string, denoted as Position, which is used for speech template filling.

[0114] Distance levels are:

[0115]

[0116] Distance Level The corresponding Distance field is represented in the voice broadcast as readable distance descriptions such as "extremely close / near / medium distance / far distance". After processing by this module, the output will be a set of targets with location-threat semantics. The target set. For example, output example: Pedestrian very close directly in front (0.8m).

[0117] (5) Broadcast trigger module

[0118] In complex scenarios, the number of targets can fluctuate due to detection errors, occlusion, or slight movement. If we directly use a target set with location-threat semantics... Broadcasting announcements can lead to the system generating too many repetitive or meaningless instructions when the number of targets changes frequently. Therefore, this module introduces a dynamic threshold for changes in the number of targets to determine whether the scene change is significant enough to decide whether to trigger a broadcast.

[0119]

[0120] in, Let be the target quantity at time t. This is the sensitivity coefficient (default 1.5). This dynamic quantity change threshold is also related to the current quantity, better reflecting the magnitude of quantity changes and used to suppress invalid broadcasts caused by minor fluctuations. It can count changes based on directional zones, or merge zones based on the number of targets in the overall scene. The statistical results of the target quantity form a quantity description field CountInfo (e.g., "3"), used for voice template filling.

[0121] For targets within the scene, multi-dimensional semantic aggregation should be performed to simplify the broadcast, and a threat level hierarchy should be adopted, with each location group further subdivided according to threat level:

[0122]

[0123] Based on the range of threat scores The segmented mapping shown in Table 1 is adopted.

[0124] Table 1 Segmented Mapping

[0125]

[0126] The threat level labels mentioned above (such as "[Emergency]", "[Warning]", and "[Alert]") are collectively denoted by the ThreatLevel field and used for subsequent voice prefix generation. High-risk targets are prioritized for separate broadcasts, while medium- and low-risk targets can be broadcast together. For targets in the same location and with the same threat level, the number is combined according to category. For example, output example: 3 pedestrians to the left front.

[0127] (6) Cooling management and dynamic broadcasting scheduling module

[0128] To avoid information overload, this invention introduces dynamic broadcast interval control:

[0129]

[0130] in, Use a base interval (e.g., 10 seconds) and shorten the interval for high-threat targets.

[0131] For the broadcast queue generated above, high-risk targets are broadcast immediately, while medium-risk targets are broadcast with a delay to avoid information overload. If a new target's threat score exceeds 120% of the currently broadcast target's, the current broadcast is interrupted, and the newly emerging threat level target is broadcast first.

[0132] (7) Voice broadcast queue management and generation module

[0133] Based on the output of the above modules, dynamically populate the broadcast template:

[0134]

[0135] Field descriptions and generated speech are shown in Tables 2 and 3.

[0136] Table 2 Field Descriptions

[0137] Fields Source module Example value ThreatLevel Broadcast trigger module "[urgent]" Position Spatial semantic mapping "front left" CountInfo Semantic aggregation "3" Name Target detection "pedestrian" Distance Distance Classification Mid-range

[0138] Table 3 Broadcast Examples

[0139]

[0140] By using a dynamic priority queue and a preemptive broadcast mechanism, the timing and interruption logic of voice output are controlled to ensure that high-threat information reaches users in real time.

[0141] Combination Figure 3 Priority insertion logic, for high-threat instructions ( ): Immediately insert at the head of the queue, interrupting the current broadcast (if present). For example: If "vehicle on the right" is being broadcast and "emergency pedestrian ahead" is detected, immediately terminate the current broadcast and prioritize broadcasting the pedestrian warning. Regular commands are appended to the tail of the queue in chronological order. After broadcasting, the target ID and broadcast timestamp are sent back to the cooldown management and dynamic broadcast scheduling module to update the cooldown timer. When the queue length > 5, the command with the lowest threat level is discarded. Automatic retry (maximum 2 times); if still unsuccessful, the target ID is marked as "requires manual inspection".

[0142] Based on the aforementioned system, a robot intelligent voice broadcasting method for complex visual scenes includes the following steps:

[0143] The system acquires environmental images and identifies targets using a target detection model, obtaining the target features of each target.

[0144] Through spatiotemporal joint analysis, consistency verification is performed on each target. If a target meets the consistency verification conditions, it is added to the stable target set.

[0145] Calculate the threat score for each target in the stable target set and determine the threat level;

[0146] Determine the azimuth and range classification of each target in the stable target set, and determine the target's location using the azimuth.

[0147] Determine if the current condition meets the broadcast trigger mechanism. If it does, generate a voice message based on the threat level and broadcast it according to the dynamic broadcast interval control mechanism. If the condition does not meet the requirements, do not broadcast.

[0148] This invention addresses the pain points of existing technologies:

[0149] 1. The conflict between visual richness and the limitations of speech: Traditional broadcasting systems simply translate visual information, resulting in lengthy speech or missing key information. This invention uses strategies such as temporal consistency verification, threat classification, and semantic aggregation to filter key content (such as high-risk targets and spatial locations) from massive amounts of visual information and outputs it as a structured speech template. This solves the problems of "information overload" or "missing key information" in traditional systems and achieves efficient visual-speech mapping.

[0150] 2. Dynamic Priority and Real-Time Response: Based on threat scores according to target category, speed, and distance, the system dynamically adjusts broadcast priorities to ensure that high-risk targets (such as those rapidly approaching vehicles) are reached first, significantly improving response speed and enabling adaptive threat broadcasting. Through preemptive queue management, high-threat commands immediately interrupt low-priority broadcasts, significantly reducing the delay in reaching critical events.

[0151] 3. Multi-scenario adaptability and scalability: weighting coefficients ( The parameters and thresholds (such as IoU_thresh) can be automatically adjusted according to the scene (indoor / outdoor) and task (guided tour / security / industrial), allowing for dynamic configuration and rapid adaptation to application scenarios and tasks.

[0152] 4. Enhanced User Experience: Spatial location cues allow users to locate targets without visual assistance, making it suitable for security, guide systems, and other scenarios. The dynamic cooling mechanism of this invention adaptively adjusts the broadcast interval based on the threat level, improving the comfort of human-computer interaction.

[0153] Example

[0154] This embodiment uses the scenario of "intelligent voice broadcasting of guide robots in complex scenarios" as an example to demonstrate the present invention. Figure 4 The urban sidewalk shown includes pedestrians, bicycles, and obstacles (steps, streetlights). Blind users need to perceive high-risk targets in their surroundings in real time through voice prompts (as shown below). Figure 4 The location of a rapidly approaching bicycle and static obstacles (such as the steps on the right). A robot intelligent voice broadcasting method for complex visual scenes includes:

[0155] Step 1: The navigation robot detects pedestrians, bicycles, streetlights, and steps using RGB images, and provides 3D positions using LiDAR or depth cameras, resulting in the following output:

[0156] Bicycle: ID=101, Category=Bicycle, Speed=(-1.5m / s, 0, 0), Position=(3m, 1m, 0).

[0157] Pedestrian: ID=102, Category=Person, Speed=(0.5m / s, 0, 0), Location=(5m, 0.5m, 0).

[0158] Step: ID=103, Category=Obstacle, Speed=(0, 0, 0), Position=(4m, -1m, 0).

[0159] Step 2: Determine the authenticity and stability of the detection target through the time sequence consistency verification module.

[0160] Bicycles and people appear stably for 5 consecutive frames (count=5), with IoU fluctuation <0.1, and are judged as real targets.

[0161] The obstacle passed the verification directly due to its static characteristics.

[0162] A stable set of targets will be obtained. .

[0163] Step 3: The dynamic threat classification and decision-making module performs threat calculations.

[0164] Bicycle: Threat = 0.9 (high speed + medium distance).

[0165] Pedestrian: Threat=0.4 (low speed + long distance).

[0166] Step: Threat=0.3 (static + mid-distance).

[0167] The priority order is: bicycles > pedestrians > steps.

[0168] Step 4, Spatial Semantic Mapping Module

[0169] Direction and distance:

[0170] Bicycle: Direction: ,distance: Direction = Directly in front, DistanceLevel = Center distance.

[0171] Pedestrian: Direction: ,distance: Direction = Directly in front, DistanceLevel = Center distance.

[0172] Therefore, we get:

[0173] [High Risk] A bicycle is 3 meters away in front of you at mid-range;

[0174] [Note] Keep a distance of 5 meters from pedestrians directly in front.

[0175] Step 5: Dynamic Semantic Aggregation and Broadcast Trigger Module

[0176] Quantity change detection: Bicycles are newly appearing targets (ΔN_t=1 > 0), triggering a broadcast.

[0177] Step 6: Dynamic broadcast scheduling and cooling management module

[0178] Cooldown time:

[0179] Bicycle: Δt_c = 10s / 0.9 ≈ 11s (high-frequency reminder).

[0180] Pedestrian: Δt_c = 10s / 0.4 ≈ 25s (low-frequency reminder).

[0181] Dispatch result: Immediately broadcast bicycle warnings and prompt pedestrians to temporarily queue.

[0182] Step 7: The voice broadcast queue management and generation module outputs the following:

[0183] [Urgent] There is a bicycle directly in front of me, at medium distance!

[0184] [Note] There is one pedestrian directly in front of you, at a moderate distance.

[0185] The guide robot reminds users to avoid bicycles and can detect the location of other pedestrians in advance.

[0186] Using the technical solution of the present invention, the effects of this embodiment are as follows:

[0187] 1. Accuracy: The false alarm rate for bicycles has been significantly reduced.

[0188] 2. Real-time: High-risk targets are detected and promptly added to the broadcast queue.

[0189] 3. Spatial Awareness: Describes location from the user's perspective, facilitating real-time understanding of the weekly report environment.

[0190] 4. User experience: The broadcast frequency is moderate, and there is no information overload.

[0191] Therefore, this invention achieves end-to-end optimization of "multimodal perception → temporal verification → threat decision → semantic generation," filling the technological gap in robot voice broadcasting in complex scenarios and demonstrating significant innovation and application value. This technology combines computer vision, multimodal perception fusion, and adaptive broadcasting strategies to resolve the contradiction between the richness of visual scenes and the limitations of voice expression, optimizing the efficiency and accuracy of robot information broadcasting in dynamic environments.

[0192] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent voice broadcasting of robots in complex visual scenes, characterized in that, Including the following steps: The system acquires environmental images and identifies targets using a target detection model, obtaining the target features of each target. Through spatiotemporal joint analysis, consistency verification is performed on each target. If a target meets the consistency verification conditions, it is added to the stable target set. Calculate the threat score for each target in the stable target set and determine the threat level; Determine the azimuth and range classification of each target in the stable target set, and determine the target's location using the azimuth. Determine if the current condition meets the broadcast trigger mechanism. If it does, generate a voice message based on the threat level and broadcast it according to the dynamic broadcast interval control mechanism. If the condition does not meet the requirements, do not broadcast.

2. The intelligent voice broadcasting method for robots in complex visual scenes according to claim 1, characterized in that, The target detection model includes a YOLOv8 model, a multi-target tracking module, and an output module. The YOLOv8 model outputs target detection boxes. ,category and confidence level The target detection box The corresponding RGB image input multi-target tracking module first uses a lightweight ReID network to extract the target appearance feature vector, then calculates the matching index, and then uses the Hungarian algorithm to perform target matching. If the matching is successful, the motion state and the number of consecutive occurrences of the target are updated, and the features of the target are output through the output module. If the matching is unsuccessful, the target is temporarily stored or deleted. The matching metric is: ； ； in, To match the metrics, For feature cosine similarity, , for the target of extraction In time The target feature vector, The IoU distance between the detection box and the prediction box. The coefficient is set.

3. The intelligent voice broadcasting method for robots in complex visual scenes according to claim 2, characterized in that, The output module outputs the following characteristics of the target: ； in, For the goal A unique tracking identifier, For the goal Category tags, For the goal In time 2D detection box coordinates , Target 3D spatial coordinates For the goal Velocity components in 3D space For the goal The number of consecutive frames, For the goal The detection confidence level, Let t be the target quantity.

4. The intelligent voice broadcasting method for robots in complex visual scenes according to claim 2, characterized in that, The consistency verification is performed on each target. In the time window Cross-frame consistency analysis is performed, including temporal correlation checks, trajectory continuity checks, and visual and motion feature consistency checks. Targets satisfying all three checks are designated as stable targets and added to the stable target set. The temporal correlation check is as follows: ； ； in, The average confidence threshold of the joint matching cost within the time window. The threshold for allowing fluctuations in the joint matching cost within the time window. , Target In the time window Set of internal matching indicator sequences; The trajectory continuity check is as follows: if consecutive frames appear... If so, the check is satisfied. The set frame rate threshold; The consistency check between visual and motion features is as follows: ； in, The displacement of the center point of the detection box between adjacent frames of the target. For the detection boxes between adjacent frames , For indicator functions, Displacement threshold Cumulative frame threshold Target Category Adaptive threshold adjustment, when If so, the visual and motion feature consistency check is satisfied.

5. The intelligent voice broadcasting method for robots in complex visual scenes according to claim 3, characterized in that, The target's threat score is: ； in, , , , The weighting coefficients are for category, speed, distance, and stability. For predefined target categories The corresponding value, Let i be the velocity of target i. This represents the maximum speed. The distance between the target and the robot body. For the specific minimum value set, This represents the maximum value within the time window. right Perform Min-Max normalization and fall into The range; the threat level is determined as follows: like If the value is ≥ 0.7, the threat level is high-risk target, and the corresponding voice field is emergency. If 0.4 ≤ If the value is less than 0.7, the threat level is medium-risk and the corresponding voice field is a warning. like If the value is less than 0.4, the threat level is low-risk, and the corresponding voice field is a prompt.

6. The intelligent voice broadcasting method for robots in complex visual scenes according to claim 1, characterized in that, The target's location is determined based on its azimuth angle as front, right front, directly right, right rear, directly rear, left rear, directly left, or left front. The distance classification is as follows: ； in, The distance between the target and the robot body.

7. The intelligent voice broadcasting method for robots in complex visual scenes according to claim 1, characterized in that, The broadcast triggering mechanism is as follows: If the condition is met, a broadcast will be triggered; otherwise, no broadcast will be triggered. Let be the target quantity at time t. This is the sensitivity coefficient.

8. The intelligent voice broadcasting method for robots in complex visual scenes according to claim 1, characterized in that, The process of generating speech based on threat level specifically includes: For high-risk targets, the generated voice includes urgency, location, number of targets, target type, and distance classification; for medium- and low-risk targets, the generated voice includes location, target type, and distance classification.

9. A robot intelligent voice broadcasting method for complex visual scenes according to claim 1, characterized in that, The dynamic broadcast interval control mechanism is as follows: It controls the timing and interruption of voice output through a dynamic priority queue based on threat level and a preemptive broadcast mechanism; different broadcast intervals are applied to targets with different threat scores. The broadcast interval is specified. for ； in, The set basic interval.

10. A robot intelligent voice broadcasting system for complex visual scenes that implements the method of any one of claims 1-9, characterized in that, include: The scene understanding module is used to acquire environmental images and identify targets through a target detection model, obtaining the target features of each target; The temporal consistency verification module performs consistency verification on each target through spatiotemporal joint analysis. If a target meets the consistency verification conditions, it is added to the stable target set. Dynamic Threat Classification Decision Module: Used to calculate the threat score for each target in the stable target set and determine the threat level; The spatial semantic mapping module determines the azimuth and distance classification of each target in the stable target set. The azimuth determines the location of the target, and the output is a target set with azimuth-distance classification and threat score semantics. The report trigger module sets the broadcast trigger mechanism to determine whether to broadcast; The dynamic semantic aggregation module generates the voice message to be broadcast based on the threat level. The dynamic broadcast module is used to design a dynamic broadcast interval control mechanism and broadcast according to the dynamic broadcast interval control mechanism.

Images