A robot long-distance target stable recognition and perception method and system
Patent Information
- Application Number
- CN202611517380.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-29
- Publication Date
- 2026-10-30
AI Technical Summary
目标被遮挡时缺少可量化的遮挡状态和可见性评价，无法判断继续跟踪、重捕获或换视角的时机
[0050]1、本发明将跨模态目标发现、可见区域提取、遮挡状态评估、身份关联匹配、置信度生成、主动视角调整和记忆更新串联为统一的闭环处理流程，克服了现有技术中各模块相互分离、无法形成有效闭环的缺陷，保证了远距离目标识别过程中数据流的连贯性和反馈的及时性；
Smart Images
Figure CN123012631A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot environmental perception and computer vision technology, and more specifically, to a stable recognition and perception method and system for distant targets in robots. Background Technology
[0002] When robots perform tasks such as inspection, search, following, pre-grabbing localization, and security monitoring at long distances, they often need to identify targets such as traffic cones, valves, instruments, personnel, vehicles, defective parts, tooling fixtures, or designated materials at distances of tens of meters or even longer. Long-distance visual perception differs significantly from near-distance target recognition: targets occupy fewer pixels in the image, resulting in weaker local texture and edge information; robot movement or gimbal rotation introduces jitter and rolling shutter distortion; railings, branches, shelves, the robotic arm itself, and other personnel or mobile devices around the target can cause intermittent occlusion; simultaneously, strong light, backlight, rain, fog, dust, and low-light conditions at night degrade image quality. The combination of these factors makes conventional target detectors prone to missed detections, false detections, and frequent changes in the identity of the same target.
[0003] In existing technologies, one type of approach mainly relies on closed-category object detection networks or real-time detection networks to output object categories and bounding boxes in a single frame of an image. However, this requires a predefined set of categories and has limited generalization ability for new categories and language-specific objects. When the object is occluded, causing the visible area to be smaller than the typical appearance in the training samples, the detection confidence drops rapidly. In recent years, open-vocabulary detection and visual-language models have been able to discover objects based on category names or natural language descriptions, but their outputs still tend to favor single-frame semantic matching. Under conditions of long-distance occlusion and low resolution, they are prone to mistaking similar backgrounds or occluders for objects.
[0004] Another approach employs multi-target tracking, video target segmentation, or temporal feature aggregation methods to mitigate single-frame missed detections through cross-frame correlation. However, these approaches generally assume relatively stable detection results or rely solely on appearance similarity, motion prediction, and cross-union ratio (CUI). When a target reappears after being occluded, its scale, pose, and visible portion have changed. Furthermore, during robot observation, background parallax and target projection positions also undergo nonlinear changes. Simply relying on two-dimensional trajectory extrapolation can easily lead to identity switching.
[0005] Some active vision solutions improve the observation perspective by adjusting the gimbal, chassis, or robotic arm posture. However, they usually separate the active perspective planning from the target recognition network, lacking a joint evaluation of occlusion degree, recognition uncertainty, target 3D observability, and robot motion cost. When occlusion persists or multiple similar candidates exist near the target, the robot may repeatedly adjust but fail to obtain more reliable observations.
[0006] In summary, existing long-range robot target perception technologies have shortcomings in unified modeling of open semantic discovery, temporal segmentation, depth geometry, and active observation. When a target is occluded, there is a lack of quantifiable occlusion states and visibility evaluations, making it impossible to determine when to continue tracking, recapture, or change the perspective. After a target disappears briefly, there is a lack of an identity preservation mechanism that combines semantics, appearance, 3D position, and motion consistency, and there is also a lack of online memory update strategies for small targets at long distances and camera motion.
[0007] Therefore, there is an urgent need to provide a solution to the above problems. Summary of the Invention
[0008] To overcome the shortcomings of existing technologies in long-distance target perception, such as the lack of unified modeling for open semantic discovery, temporal segmentation, depth geometry and active observation, the lack of quantitative evaluation of occlusion states, the lack of a stable identity preservation mechanism after the target disappears for a short time, and the lack of online memory update strategies for long-distance small targets and camera motion, this invention provides a robot long-distance target stable recognition and perception method and system.
[0009] The technical solution of this invention is as follows:
[0010] A stable method for recognizing and perceiving distant targets in a robot includes the following steps:
[0011] The system acquires the current frame image, task text information, and robot state information; enhances the current frame image to generate image representations at different scales; and performs cross-modal matching based on the task text information and the image representations at different scales to obtain candidate target information.
[0012] Based on the candidate target information or historical information in the memory data, the visible region of the target is extracted from the current frame image, and the memory data is updated based on the visible region of the target. The occlusion degree of the target is evaluated based on the memory data and the robot state information to obtain occlusion state information.
[0013] Based on the candidate target information, the historical trajectory in the memory data and the occlusion state information, the target identity is determined by association and matching, and a target recognition confidence score is generated according to the candidate target information, the features of the target's visible area and the occlusion state information.
[0014] When the target recognition confidence is lower than the first threshold or the occlusion status information is higher than the second threshold, the action of improving the observation view is performed, and the step of performing the enhancement processing is returned based on the newly acquired image frame;
[0015] When the occlusion state information is lower than the third threshold and the target recognition confidence is higher than the fourth threshold, the current frame information is updated to long-term memory; otherwise, only the short-term prediction state is updated.
[0016] Preferably, the process of generating image representations at different scales includes:
[0017] Based on the camera focal length, target candidate depth, and robot state information, the expected scale of the target in the image is estimated. The current frame image is then normalized and scaled at multiple levels to generate a multi-scale image representation that includes the original image, a magnified image at a distance, and a locally high-resolution cropped image.
[0018] Preferably, the process of assessing the occlusion degree of the target based on the memory data and the robot state information to obtain occlusion state information includes:
[0019] Based on the historical 3D dimensions, current depth, and camera parameters in the memory data, the complete projection area of the target is predicted. The area ratio of the visible area of the target to the complete projection area of the target is calculated as the visibility rate. The visibility rate, boundary incompleteness rate, depth mutation rate, appearance drift degree, and trajectory prediction deviation are weighted and summed, and then the occlusion score is obtained by S-shaped function mapping.
[0020] The expression for the occlusion score is:
[0021] ;
[0022] in, For the first The goal at the current moment Obstruction rating For visibility, For boundary incompleteness rate, For deep mutation rate, For appearance drift, For trajectory prediction bias, , , , , and All of these are learnable or calibrable parameters.
[0023] Preferably, the process of performing association matching to determine the target identity includes:
[0024] A spatiotemporal correlation graph is constructed. The nodes of the spatiotemporal correlation graph include the current candidate target, historical trajectory fragments, and potential occluders. The edges of the spatiotemporal correlation graph include semantic consistency edges, appearance consistency edges, depth-geometric consistency edges, motion consistency edges, and robot pose consistency edges. For the current candidate target and historical trajectory, a joint matching score is calculated, and correlation matching is performed based on the joint matching score.
[0025] The expression for the joint matching score is:
[0026] ;
[0027] in, For the first The first goal and the first The degree of matching between the historical trajectories For semantic consistency, For the sake of appearance consistency, To ensure consistency in three-dimensional position and depth range, For consistency of trajectory motion, To ensure consistency of robot pose constraints, To cover up the penalty item, As weight.
[0028] Preferably, the process of generating target recognition confidence includes:
[0029] The semantic features of the candidate target information, the appearance features of the visible area of the target, the mask shape features, the depth geometric features, the motion features, and the occlusion state information are encoded into a unified target representation, and a stable recognition confidence is calculated based on the unified target representation.
[0030] The expression for the stable identification confidence level is:
[0031] ;
[0032] in, For the first The goal at the current moment The confidence score, To test the confidence level, For time series segmentation confidence, For the stability of historical identity, For three-dimensional positioning information, To obscure the rating, The trace of the uncertain covariance. As weight.
[0033] Preferably, the process of performing association matching to determine the target identity further includes:
[0034] When the occlusion state information indicates that the target is severely occluded, it enters the occlusion holding state. Based on the previous stable 3D position, robot motion, target motion model and scene geometry, the target hidden state is predicted. The target hidden state includes the 3D position mean, uncertainty covariance and identity memory features.
[0035] When the target reappears or similar candidates appear, the target's hidden state and long-term memory are used to recapture the candidates. If the joint matching score of the candidates exceeds the adaptive threshold, the original target identity is restored; otherwise, active observation is triggered.
[0036] Preferably, the process of performing the action to improve the observation perspective includes:
[0037] Candidate observation actions are enumerated in a preset action set, and the observation actions are scored based on an evaluation formula. The observation action with the highest score is selected as the candidate action to be executed.
[0038] The evaluation formula is as follows:
[0039] ;
[0040] in, Candidate actions The rating, To predict the returns from the perspective, To reduce profits by blocking out the view, To increase profits through scale For the price of sports, For safety constraints and penalties, As weight.
[0041] Preferably, the actions performed to improve the observation perspective include one or more combinations of gimbal pitch adjustment, gimbal yaw adjustment, chassis lateral movement, chassis forward movement, robotic arm end effector pose adjustment, and camera focal length adjustment.
[0042] Preferably, the memory data adopts a two-level structure of short-term memory and long-term memory. The short-term memory is used to store the target visible area, appearance features, depth distribution, two-dimensional center, three-dimensional position and robot state information of the most recent preset number of frames. The long-term memory is used to store key frame information under high confidence and low occlusion conditions.
[0043] A second aspect of the present invention provides a robot long-distance target stable recognition and perception system, comprising a candidate target generation module, an occlusion state evaluation module, an identity confidence output module, an active perception closed-loop module, and a memory strategy update module, wherein:
[0044] The candidate target generation module is used to acquire the current frame image, task text information and robot state information, perform enhancement processing on the current frame image to generate image representations at different scales, and perform cross-modal matching based on the task text information and the image representations at different scales to obtain candidate target information.
[0045] The occlusion state assessment module is used to extract the visible region of the target from the current frame image based on the candidate target information or historical information in the memory data, update the memory data based on the visible region of the target, assess the degree of occlusion of the target based on the memory data and the robot state information, and obtain occlusion state information.
[0046] An identity confidence output module is used to determine the target identity by performing association matching based on the candidate target information, the historical trajectory in the memory data and the occlusion state information, and to generate a target recognition confidence score based on the candidate target information, the features of the target's visible area and the occlusion state information.
[0047] An active perception closed-loop module is used to perform actions to improve the observation view when the target recognition confidence is lower than a first threshold or the occlusion state information is higher than a second threshold, and to return to the step of performing the enhancement processing based on the newly acquired image frame;
[0048] The memory strategy update module is used to update the current frame information to long-term memory when the occlusion state information is lower than the third threshold and the target recognition confidence is higher than the fourth threshold; otherwise, it only updates the short-term prediction state.
[0049] According to the above-described solution, the beneficial effects of this invention are as follows:
[0050] 1. This invention integrates cross-modal target discovery, visible area extraction, occlusion state assessment, identity association matching, confidence generation, active viewpoint adjustment, and memory update into a unified closed-loop processing flow, overcoming the shortcomings of existing technologies where modules are separated and cannot form an effective closed loop, and ensuring the continuity of data flow and the timeliness of feedback during the long-distance target recognition process.
[0051] 2. This invention calculates visibility rate, boundary incompleteness rate, depth mutation rate, appearance drift degree, and trajectory prediction deviation and weights them to map them into an occlusion score, thereby realizing a quantitative evaluation of the target occlusion degree. This enables the system to accurately distinguish between three states: the target has truly disappeared, is temporarily occluded, and is confused with similar candidates, providing a reliable quantitative basis for identity preservation and viewpoint adjustment.
[0052] 3. When the target is severely occluded, the present invention maintains the identity information by predicting the hidden state of the target, and performs joint rematching by combining semantic consistency, appearance consistency, depth geometric consistency, motion consistency and robot pose consistency when the target reappears, thus avoiding the problem of frequent identity switching caused by simply relying on two-dimensional trajectory extrapolation in the prior art.
[0053] 4. This invention uses a two-level structure of short-term memory and long-term memory to store target information, and decides to update the long-term memory or only update the short-term prediction based on the occlusion state and confidence level, which avoids the contamination of the target memory by occlusion objects or erroneous candidates and ensures the reliability of the target template during long-term tracking. Attached Figure Description
[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0055] Figure 1 A schematic diagram of the robot's stable long-distance target recognition and perception method provided by the present invention. Detailed Implementation
[0056] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0057] It should be noted that when a component is referred to as "fixed," "set," or "connected" to another component, it may be located directly or indirectly on that other component. The terms "upper," "lower," "left," "right," "front," "rear," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or position based on the accompanying drawings, and are for ease of description only, and should not be construed as limiting the technical solution. The terms "first," "second," etc., are used for ease of description only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features. "Many" means two or more, unless otherwise explicitly specified. "Several" means one or more, unless otherwise explicitly specified.
[0058] This invention provides a stable long-range target recognition and perception method for robots. See [link to relevant documentation]. Figure 1 This includes the following steps:
[0059] S1. Obtain the current frame image, task text information and robot state information; perform enhancement processing on the current frame image to generate image representations at different scales; perform cross-modal matching based on the task text information and the image representations at different scales to obtain candidate target information.
[0060] S2. Based on the candidate target information or historical information in the memory data, extract the visible area of the target from the current frame image, update the memory data based on the visible area of the target, evaluate the occlusion degree of the target based on the memory data and the robot state information, and obtain occlusion state information.
[0061] S3. Based on the candidate target information, the historical trajectory in the memory data and the occlusion state information, perform association matching to determine the target identity, and generate target recognition confidence based on the candidate target information, the features of the target's visible area and the occlusion state information;
[0062] S4. When the target recognition confidence is lower than the first threshold or the occlusion status information is higher than the second threshold, perform the action of improving the observation viewpoint, and return to the step of performing the enhancement processing based on the newly acquired image frame;
[0063] S5. When the occlusion state information is lower than the third threshold and the target recognition confidence is higher than the fourth threshold, the current frame information is updated to long-term memory; otherwise, only the short-term prediction state is updated.
[0064] In some embodiments, the process of performing step S1 includes:
[0065] S1.1 Image enhancement and generation of multi-scale image representations;
[0066] S1.2 Cross-modal matching and acquisition of candidate target information.
[0067] Specifically, during step S1.1, the current frame image, task text information, and robot status information are acquired. The current frame image originates from a single frame or multiple images in a long-range image sequence, acquired by a gimbal telephoto or wide-angle camera deployed on the robot. The task text information is a target description command issued by the user, such as "identify and continuously track the red valve at the end of pipe gallery number three" or "find the turnover box with a yellow label in the shelf aisle." The robot status information includes the robot chassis pose, gimbal attitude, and camera internal and external parameters.
[0068] Furthermore, the current frame image undergoes shading, low-light enhancement, local contrast enhancement, and multi-scale cropping. Based on the camera focal length, target candidate depth, and robot state information, the expected scale of the target in the image is estimated. The current frame image is then normalized and scaled at multiple levels to generate a multi-scale image representation that includes the original image, a magnified image, and a locally high-resolution cropped image. This effectively increases the effective pixel ratio of small targets and reduces image drift caused by robot motion.
[0069] Specifically, during step S1.2, the task text information is encoded into text features, and the multi-scale image representation is encoded into visual features. The cross-modal matching is implemented by an open vocabulary target discovery module, which can be implemented by a language-guided detector, a visual-language region matching network, or a cue-enabled detection model.
[0070] Furthermore, the cross-modal similarity between the text features and the visual features is calculated, and a candidate target set is obtained based on the cross-modal similarity. The candidate target information includes at least candidate bounding boxes, semantic scores, and appearance features. This generates semantic candidates under conditions of unknown or few-sample targets, avoiding the system's inability to recognize only preset categories.
[0071] In some embodiments, the process of performing step S2 includes:
[0072] S2.1 Extraction of the visible area of the target;
[0073] S2.2, Updating the memory data;
[0074] S2.3 Assessment of occlusion status.
[0075] Specifically, during step S2.1, the target visible mask of the current frame image is segmented using candidate bounding boxes, point cues, or the target mask of the previous frame as cues, to obtain the target visible mask of the current frame. This segmentation process can employ a video temporal memory segmentation method, that is, using candidate bounding boxes, point cues, or the target mask of the previous frame as cues to generate the target visible mask of the current frame.
[0076] Furthermore, during step S2.2, the visible mask of the target in the current frame, along with its corresponding appearance features, depth distribution, and current robot state information, are stored in the memory data. The memory data employs a two-level structure of short-term memory and long-term memory. Short-term memory stores the target's visible area, appearance features, depth distribution, 2D center, 3D position, and robot state information for the most recent preset number of frames. Long-term memory stores keyframe information under high-confidence and low-occlusion conditions.
[0077] Specifically, during step S2.3, the complete projection area of the target is predicted based on the historical 3D dimensions, current depth, and camera parameters in the memory data. The area ratio of the visible area of the target to the complete projection area of the target is calculated as the visibility rate. The boundary incompleteness rate, depth abrupt change rate, appearance drift, and trajectory prediction deviation of the visible area of the target are calculated. The visibility rate, boundary incompleteness rate, depth abrupt change rate, appearance drift, and trajectory prediction deviation are weighted and summed, and then mapped using a sigmoid function to obtain the occlusion score.
[0078] The expression for the occlusion score is:
[0079] ;
[0080] in, For the first The goal at the current moment Obstruction rating For visibility, For boundary incompleteness rate, For deep mutation rate, For appearance drift, For trajectory prediction bias, , , , , and All of these are learnable or calibrable parameters.
[0081] In one implementation, when a target is partially obscured by pipelines, guardrails, or pedestrians, the system calculates the visibility rate based on the target's complete projection area and the currently visible mask, and combines this with depth abrupt changes, boundary incompleteness, and appearance drift to obtain an occlusion score. A higher occlusion score indicates less visible portion of the target or a greater risk of occlusion. This enables the system to accurately determine the target's current occlusion state, providing a quantitative basis for subsequent identity preservation and viewpoint adjustment.
[0082] In some embodiments, the process of performing step S3 includes:
[0083] S3.1 Construction and joint matching of spatiotemporal correlation graphs;
[0084] S3.2, Generation of stable identification confidence.
[0085] Specifically, during step S3.1, a spatiotemporal correlation graph is constructed. The nodes of the spatiotemporal correlation graph include the current candidate target, historical trajectory segments, and potential occlusions. The edges of the spatiotemporal correlation graph include semantic consistency edges, appearance consistency edges, depth-geometric consistency edges, motion consistency edges, and robot pose consistency edges. For the current candidate target and historical trajectory, a joint matching score is calculated, and correlation matching is performed based on the joint matching score.
[0086] The expression for the joint matching score is:
[0087] ;
[0088] in, For the first The first goal and the first The degree of matching between the historical trajectories For semantic consistency, For the sake of appearance consistency, To ensure consistency in three-dimensional position and depth range, For consistency of trajectory motion, To ensure consistency of robot pose constraints, To cover up the penalty item, As weight.
[0089] In practice, the system performs Hungarian matching or graph neural network association based on the joint matching score to obtain a stable target identity. During matching, it compares not only appearance similarity but also task text semantics, target depth range, predicted projection after robot pose changes, and target motion continuity, resulting in greater stability in robot motion observation and complex backgrounds.
[0090] Furthermore, during the execution of step S3.2, the semantic features of the candidate target information, the appearance features of the visible area of the target, the mask shape features, the depth geometric features, the motion features, and the occlusion state information are encoded into a unified target representation, and a stable recognition confidence score is calculated based on the unified target representation.
[0091] ;
[0092] in, For the first The goal at the current moment The confidence score, To test the confidence level, For time series segmentation confidence, For the stability of historical identity, For three-dimensional positioning information, To obscure the rating, The trace of the uncertain covariance. The weights are used to differentiate between reliable identification and transient prediction, facilitating subsequent decision-making by the robot.
[0093] In some embodiments, during the execution of step S3.1, the following further steps are included:
[0094] When the occlusion state information indicates that the target is severely occluded, it enters the occlusion holding state. Based on the previous stable 3D position, robot motion, target motion model and scene geometry, the target hidden state is predicted. The target hidden state includes the 3D position mean, uncertainty covariance and identity memory features.
[0095] When the target reappears or similar candidates appear, the target's hidden state and long-term memory are used to recapture the candidates. If the joint matching score of the candidates exceeds the adaptive threshold, the original target identity is restored; otherwise, active observation is triggered.
[0096] In one specific implementation scenario, when a target is briefly occluded by a pedestrian during inspection, the occlusion score increases, but the trajectory prediction remains reliable. The system enters an occlusion-holding state, maintaining the target's identity number and outputting a 3D position with uncertainty. After the pedestrian leaves, the system performs joint matching between newly appearing candidates and historical trajectories. During matching, it compares not only appearance similarity but also task text semantics, target depth range, predicted projection after robot pose changes, target motion continuity, and keyframe memory features. When the joint matching score exceeds a threshold, the system restores the original target identity; when the score is insufficient, the system maintains a low-confidence output and triggers active observation.
[0097] In some embodiments, the process of performing step S4 includes:
[0098] S4.1 Enumeration and evaluation of candidate observation actions;
[0099] S4.2, Action execution and perception closed loop.
[0100] Specifically, during step S4.1, candidate observation actions are enumerated from a preset action set. These candidate observation actions include one or more combinations of gimbal pitch adjustment, gimbal yaw adjustment, chassis lateral movement, chassis forward movement, robotic arm end effector pose adjustment, and camera focal length adjustment. The observation actions are scored based on an evaluation formula, and the observation action with the highest score is selected as the candidate action to be executed.
[0101] The evaluation formula is as follows:
[0102] ;
[0103] in, Candidate actions The rating, To predict the returns from the perspective, To reduce profits by blocking out the view, To increase profits through scale For the price of sports, For safety constraints and penalties, As weight.
[0104] In a specific implementation scenario, when the target's right side is obscured by a pillar, the system evaluates the visual benefits, motion costs, and safety risks of actions such as the robot moving laterally to the left, the gimbal deflecting to the right, the robot approaching, and maintaining its original position. If moving laterally to the left significantly increases the target's visibility without violating the safe distance, the robot performs this action.
[0105] Furthermore, during step S4.2, after the robot performs the action, it re-acquires image frames and returns to the enhancement processing step in step S1 to form a perception-motion closed loop.
[0106] In a specific implementation scenario, let's take the identification of the red valve at the end of the No. 3 pipe gallery by an inspection robot as an example. After the system starts, the robot first generates a target semantic prompt based on the task text and performs coarse localization in a wide-angle image. When the target is far away or the pixel size is small, the system controls the gimbal telephoto camera to locally magnify and acquire the candidate area, while simultaneously performing image stabilization and multi-scale cropping. During continuous inspection, if the target is partially obscured by pipelines or pedestrians, the system determines whether to trigger active observation based on the occlusion score. When the stable recognition confidence is lower than the first threshold or the occlusion score is higher than the second threshold, the system evaluates the viewpoint gain and motion cost of each candidate action, selects the action with the highest overall gain for execution, and then returns to perform enhancement processing steps based on the newly acquired image frame.
[0107] In some embodiments, during the execution of step S5, when determining the memory data update strategy based on the occlusion state information and the target recognition confidence, if the occlusion state information is lower than a third threshold and the target recognition confidence is higher than a fourth threshold, the visible area of the target in the current frame, appearance features, depth distribution, and robot state information are written into long-term memory; if the occlusion state information is higher than the third threshold or the target recognition confidence is lower than the fourth threshold, only the short-term prediction state is updated, and the long-term memory is not updated, to prevent occlusions or erroneous candidates from contaminating the target memory.
[0108] In other embodiments, the present invention can be used by a robotic arm to locate a grasping target partially obscured by tooling or shelving under long-distance visual guidance. The robotic arm's end-effector camera acts as an active sensing sensor, acquiring mask and depth information from different sides of the target by slightly changing the viewing pose. Once the target's identity stability reaches a threshold, the system sends the target's three-dimensional position to the subsequent grasping planning module.
[0109] In a second aspect of the invention, a robot long-distance target stable recognition and perception system is also provided, comprising a candidate target generation module, an occlusion state evaluation module, an identity confidence output module, an active perception closed-loop module, and a memory strategy update module, wherein:
[0110] The candidate target generation module is used to acquire the current frame image, task text information and robot state information, perform enhancement processing on the current frame image to generate image representations at different scales, and perform cross-modal matching based on the task text information and the image representations at different scales to obtain candidate target information.
[0111] The occlusion state assessment module is used to extract the visible region of the target from the current frame image based on the candidate target information or historical information in the memory data, update the memory data based on the visible region of the target, assess the degree of occlusion of the target based on the memory data and the robot state information, and obtain occlusion state information.
[0112] An identity confidence output module is used to determine the target identity by performing association matching based on the candidate target information, the historical trajectory in the memory data and the occlusion state information, and to generate a target recognition confidence score based on the candidate target information, the features of the target's visible area and the occlusion state information.
[0113] An active perception closed-loop module is used to perform actions to improve the observation view when the target recognition confidence is lower than a first threshold or the occlusion state information is higher than a second threshold, and to return to the step of performing the enhancement processing based on the newly acquired image frame;
[0114] The memory strategy update module is used to update the current frame information to long-term memory when the occlusion state information is lower than the third threshold and the target recognition confidence is higher than the fourth threshold; otherwise, it only updates the short-term prediction state.
[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A stable recognition and perception method for long-distance targets in a robot, characterized in that, Includes the following steps: The system acquires the current frame image, task text information, and robot state information; enhances the current frame image to generate image representations at different scales; and performs cross-modal matching based on the task text information and the image representations at different scales to obtain candidate target information. Based on the candidate target information or historical information in the memory data, the visible region of the target is extracted from the current frame image, and the memory data is updated based on the visible region of the target. The occlusion degree of the target is evaluated based on the memory data and the robot state information to obtain occlusion state information. Based on the candidate target information, the historical trajectory in the memory data and the occlusion state information, the target identity is determined by association and matching, and a target recognition confidence score is generated according to the candidate target information, the features of the target's visible area and the occlusion state information. When the target recognition confidence is lower than the first threshold or the occlusion status information is higher than the second threshold, the action of improving the observation view is performed, and the step of performing the enhancement processing is returned based on the newly acquired image frame; When the occlusion state information is lower than the third threshold and the target recognition confidence is higher than the fourth threshold, the current frame information is updated to long-term memory; otherwise, only the short-term prediction state is updated.
2. The robot's stable long-distance target recognition and perception method according to claim 1, characterized in that, The process of generating image representations at different scales includes: Based on the camera focal length, target candidate depth, and robot state information, the expected scale of the target in the image is estimated. The current frame image is then normalized and scaled at multiple levels to generate a multi-scale image representation that includes the original image, a magnified image at a distance, and a locally high-resolution cropped image.
3. The robot's stable long-distance target recognition and perception method according to claim 1, characterized in that, The process of evaluating the occlusion degree of the target based on the memory data and the robot state information to obtain occlusion state information includes: Based on the historical 3D dimensions, current depth, and camera parameters in the memory data, the complete projection area of the target is predicted. The area ratio of the visible area of the target to the complete projection area of the target is calculated as the visibility rate. The visibility rate, boundary incompleteness rate, depth mutation rate, appearance drift degree, and trajectory prediction deviation are weighted and summed, and then the occlusion score is obtained by S-shaped function mapping. The expression for the occlusion score is: ； in, For the first The goal at the current moment Obstruction rating For visibility, For boundary incompleteness rate, For deep mutation rate, For appearance drift, For trajectory prediction bias, , , , , and All of these are learnable or calibrable parameters.
4. The robot's stable long-distance target recognition and perception method according to claim 1, characterized in that, The process of performing association matching to determine the target identity includes: A spatiotemporal correlation graph is constructed. The nodes of the spatiotemporal correlation graph include the current candidate target, historical trajectory fragments, and potential occluders. The edges of the spatiotemporal correlation graph include semantic consistency edges, appearance consistency edges, depth-geometric consistency edges, motion consistency edges, and robot pose consistency edges. For the current candidate target and historical trajectory, a joint matching score is calculated, and correlation matching is performed based on the joint matching score. The expression for the joint matching score is: ； in, For the first The first goal and the first The degree of matching between the historical trajectories For semantic consistency, For the sake of appearance consistency, To ensure consistency in three-dimensional position and depth range, For consistency of trajectory motion, To ensure consistency of robot pose constraints, To cover up the penalty item, For weights.
5. The robot's stable recognition and perception method for distant targets according to claim 1, characterized in that, The process of generating target recognition confidence includes: The semantic features of the candidate target information, the appearance features of the visible area of the target, the mask shape features, the depth geometric features, the motion features, and the occlusion state information are encoded into a unified target representation, and a stable recognition confidence is calculated based on the unified target representation. The expression for the stable identification confidence level is: ； in, For the first The goal at the current moment The confidence score, To test the confidence level, For time series segmentation confidence, For the stability of historical identity, For three-dimensional positioning information, To obscure the rating, The trace of the uncertain covariance. For weights.
6. The robot's stable long-distance target recognition and perception method according to claim 1, characterized in that, The process of performing association matching to determine the target identity also includes: When the occlusion state information indicates that the target is severely occluded, it enters the occlusion holding state. Based on the previous stable 3D position, robot motion, target motion model and scene geometry, the target hidden state is predicted. The target hidden state includes the 3D position mean, uncertainty covariance and identity memory features. When the target reappears or similar candidates appear, the target's hidden state and long-term memory are used to recapture the candidates. If the joint matching score of the candidates exceeds the adaptive threshold, the original target identity is restored; otherwise, active observation is triggered.
7. The robot's stable recognition and perception method for distant targets according to claim 1, characterized in that, The process of performing the action to improve the observation perspective includes: Candidate observation actions are enumerated in a preset action set, and the observation actions are scored based on an evaluation formula. The observation action with the highest score is selected as the candidate action to be executed. The evaluation formula is as follows: ； in, Candidate actions The rating, To predict the returns from the perspective, To reduce profits by blocking out the view, To increase profits through scale For the price of sports, For safety constraints and penalties, For weights.
8. The robot's stable recognition and perception method for distant targets according to claim 1, characterized in that, The actions performed to improve the observation perspective include one or more combinations of gimbal pitch adjustment, gimbal yaw adjustment, chassis lateral movement, chassis forward movement, robotic arm end effector pose adjustment, and camera focal length adjustment.
9. The robot's stable recognition and perception method for distant targets according to claim 1, characterized in that, The memory data adopts a two-level structure of short-term memory and long-term memory. The short-term memory is used to store the target's visible area, appearance features, depth distribution, two-dimensional center, three-dimensional position and robot state information of the most recent preset number of frames. The long-term memory is used to store key frame information under high confidence and low occlusion conditions.
10. A robot long-distance target stable recognition and perception system, characterized in that, It includes a candidate target generation module, an occlusion state evaluation module, an identity confidence output module, an active perception closed-loop module, and a memory strategy update module, among which: The candidate target generation module is used to acquire the current frame image, task text information and robot state information, perform enhancement processing on the current frame image to generate image representations at different scales, and perform cross-modal matching based on the task text information and the image representations at different scales to obtain candidate target information. The occlusion state assessment module is used to extract the visible region of the target from the current frame image based on the candidate target information or historical information in the memory data, update the memory data based on the visible region of the target, assess the degree of occlusion of the target based on the memory data and the robot state information, and obtain occlusion state information. An identity confidence output module is used to determine the target identity by performing association matching based on the candidate target information, the historical trajectory in the memory data and the occlusion state information, and to generate a target recognition confidence score based on the candidate target information, the features of the target's visible area and the occlusion state information. An active perception closed-loop module is used to perform actions to improve the observation view when the target recognition confidence is lower than a first threshold or the occlusion state information is higher than a second threshold, and to return to the step of performing the enhancement processing based on the newly acquired image frame; The memory strategy update module is used to update the current frame information to long-term memory when the occlusion state information is lower than the third threshold and the target recognition confidence is higher than the fourth threshold; otherwise, it only updates the short-term prediction state.