Robot vision control self-adaptive pre-judgment system and method

Abstract

The invention discloses a robot visual control adaptive pre-judgment system and method combining an event camera and an RGB-D camera, and relates to the technical field of robot visual control. The objective of the invention is to solve the problem of visual control lag in a high-dynamic scene in the prior art. The event camera and the RGB-D camera realize time axis accurate alignment (error + / -1ms) through a hardware synchronization module to form a collaborative acquisition mechanism: the event camera captures a dynamic sudden target and outputs a millisecond-level response brightness change event stream to guarantee dynamic sensing real-time performance; the RGB-D camera obtains an environment static three-dimensional structure, outputs a depth map and color textures, and provides spatial precision support. The method comprises the following steps of: performing lightweight processing on bimodal data, performing deep fusion, jointly inputting a motion vector extracted from an event stream and static depth information of RGB-D into a pre-judgment model based on Transform, staged attention to a Transform network, adapting to information requirements of a full life cycle before, during and after shielding, performing time sequence context sensing shielding compensation, dynamically calculating a compensation amount to correct pre-judgment deviation, and performing prediction on the prediction deviation. According to the technical scheme, multi-dimensional dynamic weight distribution is achieved, the bimodal fusion weight is adjusted in real time based on the scene type, the shielding degree and the target speed, the system can accurately pre-judge the position of a shielded target 50-100 milliseconds ahead of time, the total reasoning time delay is smaller than or equal to 40 ms, the autonomous control ability of a robot in industrial grabbing, service obstacle avoidance, outdoor inspection and other scenes is remarkably improved, and the real-time performance of the robot is improved. And the hardware cost is controllable and easy to integrate.

Description

Technical Field

[0001] This invention relates to the field of robot vision control and multimodal perception technology, specifically to a system and method for predicting occluded targets by combining an event camera and an RGB-D camera. It is applicable to dynamic and complex scenarios such as high-speed material handling in industrial production lines (e.g., metal boxes occluded in electronic component sorting), dynamic obstacle avoidance of service robots (e.g., pedestrians suddenly occluding objects in shopping malls), and path planning of outdoor inspection robots (e.g., trees occluding road obstacles). It enables robots to predict and control occluded targets with high accuracy and real-time performance. Background Technology

[0002] As robotics technology penetrates into dynamic and complex scenarios, accurate prediction of occluded targets has become a core bottleneck restricting robot performance. Existing technologies suffer from three major shortcomings: First, bimodal fusion is often a simple stitching process, lacking deep interaction between dynamic cues from event cameras and static spatial information from RGB-D models, failing to effectively fill in missing features in occluded areas; second, the prediction model allocates attention indiscriminately throughout the entire occlusion lifecycle (before, during, and after occlusion), leading to fragmented predictions or insufficient accuracy; third, occlusion compensation uses fixed values, failing to adapt to different occlusion types, scenarios, and target motion states, and its simplistic weight allocation results in poor cross-scenario adaptability. Therefore, there is an urgent need for an occluded target prediction solution with deep fusion capabilities, stage adaptability, and a dynamic compensation mechanism to address the pain points of existing technologies. Summary of the Invention

[0003] To overcome the aforementioned shortcomings of the current technology, this invention will provide a robot vision control adaptive prediction system and method to solve the prominent problems in the background technology. The technologies that need to be overcome are as follows:

[0004] Overcoming the limitations of single-modal sensors, it achieves complementary perception with high temporal resolution and high spatial accuracy in dynamic scenes.

[0005] By combining the collaborative acquisition of event cameras and RGB-D cameras, a more accurate prediction algorithm for targets can be achieved.

[0006] Achieve dynamic target trajectory prediction within 50-100ms, improving the foresight and accuracy of robot control.

[0007] Establish a hardware-level time synchronization mechanism to reduce time alignment errors in multimodal data.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0009] This invention achieves deep fusion of event cameras and RGB-D cameras and precise robot control through a six-level architecture of "hardware synchronization, feature-driven fusion, phased prediction, dynamic compensation, weight adaptation, and control closed loop." It also relies on four innovative algorithms to achieve accurate prediction of occluded targets. The specific technical solution is as follows:

[0010] Dual-modal hardware synchronization module: The synchronous triggering circuit controlled by FPGA controls the sampling timestamp error between the event camera and the RGB-D camera within ±1ms. The event camera is configured with 240×320 resolution and 1ms response time, while the RGB-D camera supports 480×640 color / depth dual output and has a depth measurement range of 0.3-10m.

[0011] Data synchronization strategy. Leveraging the advantages of event cameras, such as millisecond-level response speed and no inter-frame latency, dynamic targets (including the motion state of occluded targets before occlusion) are captured in real time. This compensates for the ≥100ms positioning latency caused by the fixed frame rate (30-60fps) of traditional RGB-D cameras, providing a high temporal resolution dynamic data foundation for subsequent extraction of motion vectors and prediction of target trajectories, ensuring that the target motion state is not lost in highly dynamic scenes.

[0012] Simultaneously, the event camera can capture the brightness changes of the target edge before and after occlusion, continuously output the event stream and extract motion vectors, and complete the target's motion state information during the occlusion period. This provides the prediction model with "motion trend before occlusion + dynamic clues during the occlusion process," avoiding prediction interruptions or deviations caused by occlusion. Event stream data within 10ms before and after the RGB-D frame timestamp is extracted, and sub-millisecond alignment is achieved through a timestamp interpolation algorithm. After synchronization, the event stream retains the polarity of brightness changes (-1: darkening, 1: brightening) and coordinate information. The RGB-D data is synchronously output as a color map, depth map, and intrinsic parameter matrix.

[0013] A dual-modal occluded object feature collaborative extraction and fusion module. Occlusion dynamic cues are completed using an event camera. The event stream is processed using temporal slice rasterization (1ms / slice), converting event data before and after occlusion into a temporal feature matrix; the module focuses on capturing sudden brightness changes at the moment of occlusion (such as the occluder entering / leaving) and the motion trajectory of the unoccluded portion of the target.

[0014] After Gaussian filtering and 1D smoothing for noise reduction, the centroid coordinates and motion vector (dx,dy) of each event are calculated, and the features of "motion trend before occlusion + event residue during occlusion" are extracted to form a [20,2]-dimensional temporal feature sequence (covering a 20ms time window) to compensate for the loss of target dynamic information during occlusion.

[0015] The spatial relationships of occlusion are clarified using an RGB-D camera. Occlusion regions are segmented on the RGB-D depth map, and the depth contours, dimensions, and spatial coordinates of occluders are extracted. At the same time, the depth statistical features (mean and variance) of the unoccluded areas of the target are preserved. The color image is used to identify the type of occlusion (such as metal occlusion in industrial scenes and pedestrian occlusion in service scenes). The spatial features are mapped to the event camera resolution through bilinear interpolation to form a [20,2] depth feature sequence that matches the event feature dimension, clarifying the spatial hierarchy between the target and the occluders, and avoiding mistaking occluders for targets during trajectory prediction.

[0016] A dual-modal dynamic-static feature fusion algorithm is used. The motion vector (dx, dy) extracted from the event stream is smoothed temporally to obtain a preliminary trajectory, which is then input into the dynamic guidance sublayer. Combined with the depth contour of the occluded object output by RGB-D (such as the coordinates of the edge of industrial metal occlusion and the torso size of pedestrians in the service scene), an occlusion region mask is generated to accurately mark the pixel range of the target being occluded.

[0017] The static correction sublayer uses this mask to filter out false event points caused by reflections from occluders in the event stream. At the same time, it uses the depth value of the unoccluded RGB-D region to constrain the amplitude of the motion vector. When the target approaches the occluder, the motion speed needs to match the rate of change of depth to avoid the trajectory drifting due to occlusion interference.

[0018] An "occlusion attention weight" is introduced. For unoccluded target areas (target integrity ≥ 80%), the RGB-D static feature weight is increased to 0.6-0.7 to ensure spatial positioning accuracy. For occluded areas (target integrity ≤ 80%), the event flow dynamic feature weight is increased to 0.6-0.7. The motion trend in the 10ms before occlusion is associated with the spatial position of the edge of the occluded object through an attention mechanism to generate a dynamic-static fusion feature map of [20,4] dimensions, which takes into account both real-time performance and information integrity.

[0019] A Transformer prediction network employs a phased attention mechanism. Based on the "brightness change density" of the event stream and the "target region integrity" of RGB-D, it automatically divides the process into three phases: before occlusion (target integrity ≥ 80%, low event stream brightness change density), during occlusion (target integrity 10%-80%, event stream brightness change density increases sharply), and after occlusion (target integrity ≤ 10%, event stream brightness change density decreases). At each layer of the Transformer encoder, a dedicated attention mask is output for each phase.

[0020] Before occlusion, attention weights are tilted towards the RGB-D target center region (0.6%), prioritizing static feature localization. During occlusion, weights are tilted towards the event stream "last 3ms motion vector before occlusion" and the RGB-D "occluder edge" (total 0.7%), focusing on motion continuity. After occlusion, weights are tilted towards the event stream "temporal acceleration features" (Δdx / Δt, Δdy / Δt) and the RGB-D "occluder depth gradient," using gradients to infer the target motion path. An occlusion feature compression module is added to the encoder output layer to reduce the dimensionality of redundant occluder background features, keeping the number of network parameters below 600,000 and the inference latency ≤18ms, meeting the real-time control requirements of the robot's edge.

[0021] Lightweight Adaptive Occlusion Prediction Network Design. A simplified Transformer model with 2-layer encoder and 4-head attention (parameter count ≤ 800,000) is used as input, which is the fused temporal features of "occlusion dynamic cues + spatial features" [20,4]. The dimension is increased to 128 dimensions through the feature encoding layer (Linear + ReLU + Dropout), and the attention mechanism is optimized by adding "occlusion region attention weights" to prioritize the dynamic relationship between the unoccluded target and the edge of the occluded object, thereby reducing occlusion interference.

[0022] The occluded object trajectory prediction mechanism takes the features from the last time step of the Transformer output and calculates the predicted position by predicting the target's position offset in the x and y directions from the output of the prediction head, combined with the following occlusion compensation logic: For partial occlusion: the target motion vector is corrected based on the depth of the unoccluded area obtained from RGB-D (to avoid trajectory offset caused by obstruction); for complete occlusion: the trajectory fitting time is extended based on the event flow motion trend 10-20ms before occlusion to ensure prediction continuity. Final prediction formula:

[0023] pred_x = current_x + offset_x × times (T / 20) + occlusion compensation amount

[0024] pred_y = current_y + offset_y × times (T / 20) + occlusion compensation amount

[0025] (T is the prediction time, 50-100ms; 20 is the time window corresponding to the feature sequence)

[0026] A temporal context-aware occlusion compensation algorithm. From the 20ms temporal window of the event stream, additional features of "motion acceleration" (ax = Δdx / Δt, ay = Δdy / Δt) and "directional stability" (motion direction deviation over 3ms) are extracted. Combined with the "occluder motion state" (stationary / moving) obtained from RGB-D, a temporal context feature vector [ax, ay, directional stability, occluder velocity] is constructed.

[0027] The compensation amount is dynamically output based on the occlusion stage to correct prediction deviations: Before occlusion → During occlusion: Compensation amount = 0.3 × ax × T + 0.2 × distance from the edge of the occluded object (obtained by RGB-D), correcting trajectory offset caused by occlusion entry; During occlusion → After occlusion: Compensation amount = 0.4 × directional stability × T + 0.1 × velocity of the occluded object (effective when the occluded object moves), avoiding prediction lag caused by the movement of the occluded object; where T is the prediction time (50-100ms), the compensation amount is in pixels, and the coefficients are optimized through training on the scene dataset. Every 20ms, the actual position of the target after occlusion is removed is compared with the predicted position, and the compensation function coefficients are iteratively updated to improve long-term prediction accuracy.

[0028] Multi-dimensional dynamic weight allocation strategy. Three-dimensional decision factor construction: including scene type (industrial / service / outdoor, identified through RGB-D static features), occlusion degree (complete occlusion / partial occlusion, calculated through RGB-D target integrity), and target speed (high speed / low speed, calculated through event flow motion vector amplitude). Non-linear weight allocation function: W_e = α × scene coefficient + β × occlusion degree coefficient + γ × target speed coefficient (α = 0.2, β = 0.5, γ = 0.3), where the scene coefficient corresponds to 0.3 for industrial, 0.7 for service, and 0.5 for outdoor; the occlusion degree coefficient corresponds to 0.6 for complete occlusion and 0.4 for partial occlusion; and the target speed coefficient corresponds to 0.7 for high speed and 0.3 for low speed, W_r = 1 - W_e. Real-time update mechanism: every 10ms, the three-dimensional decision factors and weight values ​​are updated based on newly acquired bimodal data to adapt to scene switching (e.g., industrial workshop → outdoor inspection) or changes in occlusion status (partial occlusion → complete occlusion). Purpose of the invention

[0029] The purpose of this invention is to provide a robot vision control adaptive prediction system and method that combines an event camera and an RGB-D camera to solve the problem of visual control lag in high dynamic scenes in the prior art. Technical solution

[0030] This invention provides a robot vision control adaptive prediction system combining an event camera and an RGB-D camera, comprising a hardware acquisition module, an algorithm processing module, and a robot control module. In the hardware acquisition module, the event camera captures dynamic, sudden targets in real time, the RGB-D camera acquires the static 3D structure of the environment, and the hardware synchronization module aligns the timeline. The algorithm processing module processes and predicts the dual-modal data, and the adaptive adjustment unit achieves cross-scene parameter self-tuning. The robot control module adjusts its actions based on the processing results. A method for utilizing this system is also provided. Beneficial effects

[0031] Enhanced perception performance: Dual-modal fusion reduces dynamic target detection latency to less than 15ms, with a positioning error of ≤2 pixels in high-speed motion scenarios of 10m / s, and improved robustness by 40% in low-light (1 lux) and strong light (100,000 lux) environments, outperforming single RGB-D solutions (latency ≥80ms, error ≥5 pixels).

[0032] Balancing real-time performance with lightweight design: The lightweight Transformer model has an inference latency of ≤20ms, and the total time for hardware synchronization and feature processing is ≤40ms, meeting the real-time control requirements of the edge. The number of parameters is reduced by 84% compared to the standard Transformer.

[0033] Improved prediction accuracy: Position error within the 50-100ms prediction window is ≤3 pixels, increasing the success rate of industrial grasping from 68% in traditional solutions to 92%, and reducing the obstacle avoidance stuttering rate of service robots from 45% to 12%.

[0034] Enhanced scenario adaptability: The adaptive weighting mechanism ensures that the overall performance fluctuation of the system is ≤5% in industrial, service, and outdoor scenarios, enabling cross-scenario deployment without manual parameter adjustment.

[0035] Controllable hardware costs: Using conventional event cameras and RGB-D cameras (total hardware cost ≤ 5,000 yuan), the cost is reduced by 75% compared to the LiDAR fusion solution (cost ≥ 20,000 yuan), and the structure is compact and easy to integrate with robots.

[0036] This invention, through the collaborative work of an event camera and an RGB-D camera, combined with a dynamic prediction model and an adaptive mechanism, can predict the position of dynamic targets in advance, solving the control lag problem in highly dynamic scenarios and improving the performance of robots in industrial grasping, service obstacle avoidance, outdoor inspection and other scenarios, and has broad application prospects. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the system of the present invention;

[0038] Figure 2 This is a flowchart illustrating the method of the present invention.

Claims

1. A robot vision control adaptive prediction system combining event cameras and RGB-D cameras, comprising: a dual-mode data acquisition and FPGA synchronization module, a dual-mode dynamic-static feature mutual driving fusion module, an occlusion stage division module, a phased attention mechanism Transformer prediction network, a timing context awareness occlusion compensation module, a multi-dimensional dynamic weight distribution module, a closed-loop parameter fine-tuning module, and a robot control module. 1.1 The dual-mode data acquisition and FPGA synchronization module configures an event camera with a resolution of 240x320 and a response speed of 1ms, and an RGB-D camera supporting 480x640 color / depth dual output (depth measurement range 0.3-10m), and controls the timestamp error of the dual-camera sampling time within ±1ms through the synchronization trigger circuit controlled by the FPGA host, intercepts the event stream within 10ms before and after the timestamp of the RGB-D frame, and realizes sub-millisecond alignment through timestamp interpolation, and synchronously outputs the brightness change polarity and coordinate information of the event stream, the color map, the depth map and the intrinsic matrix of the RGB-D; 1.2 The dual-mode dynamic-static feature mutual driving fusion module includes a dynamic guiding sublayer, a static correction sublayer, and a cross-modal attention fusion sub-module; the dynamic guiding sublayer generates preliminary trajectories based on the motion vectors extracted from the event stream, and generates an occlusion area mask in combination with the depth profile of the occluder output by the RGB-D; the static correction sublayer filters abnormal event points in the event stream based on the mask, and restricts the amplitude of the motion vector through the depth value of the unoccluded area of the RGB-D; the cross-modal attention fusion sub-module assigns feature weights according to the target occlusion state, and the RGB-D static feature weight of the unoccluded area is 0.6-0.7, and the event stream dynamic feature weight of the occluded area is 0.6-0.7; 1.3 The occlusion stage division module divides the occlusion process into three stages: before occlusion (target completeness ≥80%), during occlusion (target completeness 10%-80%), and after occlusion (target completeness ≤10%) based on the brightness change density of the event stream and the integrity of the target area of the RGB-D; 1.4 The phase-based attention mechanism Transformer prediction network outputs exclusive attention masks in each occlusion stage: the weight in the pre-occlusion stage is inclined to the RGB-D target center area (accounting for 0.6), the weight in the middle-occlusion stage is inclined to the motion vector 3ms before occlusion and the occluder edge (total accounting for 0.7), and the weight in the post-occlusion stage is inclined to the event stream time sequence acceleration feature and the RGB-D occluder depth gradient; 1.5 The time sequence context-aware occlusion compensation module extracts motion acceleration and direction stability features from a 20ms time window of the event stream, constructs a context feature vector in combination with the motion state of the occluder obtained by the RGB-D, and calculates the compensation amount through a segmented compensation function: before occlusion→during occlusion compensation amount = 0.3×motion acceleration×prediction time+0.2×occluder edge distance, during occlusion→after occlusion compensation amount = 0.4×direction stability×prediction time+0.1×occluder speed, and the prediction time ranges from 50ms to 100ms; 1.6 The multi-dimensional dynamic weight distribution module constructs a three-dimensional decision factor of "scene type-occlusion degree-target speed", calculates the event camera weight W_e through a nonlinear function W_e = α×scene coefficient + β×occlusion degree coefficient + γ×target speed coefficient (α = 0.2, β = 0.5, γ = 0.3), and the RGB-D weight W_r = 1-W_e, and the weight values are updated in real time every 10ms; 1.7 The closed-loop parameter fine-tuning module collects the blocked target processing success rate and obstacle avoidance stall rate of the robot motion sensor feedback every 20 ms, dynamically fine-tunes the attention weight, compensation function coefficient, and bimodal fusion weight; 1.8 The robot control module converts the pre-judgment position into joint space coordinates and generates "obstacle avoidance-grabbing" composite control instructions with linear speed ±0.5 m / s and angular speed ±1 rad / s.

2. The system of claim 1, wherein, The parameter quantity of the staged attention mechanism Transformer pre-judgment network is controlled within 600,000, and the inference delay is ≤18 ms.

3. The system of claim 1, wherein, In the three-dimensional decision factor, the scene coefficient corresponds to 0.3 for industrial scenes, 0.7 for service scenes, and 0.5 for outdoor scenes; the obstruction degree coefficient corresponds to 0.6 for complete obstruction and 0.4 for partial obstruction; the target speed coefficient corresponds to 0.7 for high speed (≥0.5 pixels / ms) and 0.3 for low speed (<0.5 pixels / ms). The abnormal event point is a false brightness change event caused by the reflection of the obstruction.

4. The system of claim 1, wherein, The timing acceleration feature is calculated by the change rate of the motion vector 5 ms before the event stream.

5. The system of claim 1, wherein, The steps include:

6. A method for predicting an occluded object target using the system of any one of claims 1-5. 6.1 Dual-modal data acquisition and synchronization step: synchronize dual-camera data through FPGA to obtain aligned event stream and RGB-D data; 6.2 Feature mutual driving fusion step: generate an obstruction area mask, filter abnormal event points, and constrain motion vectors, assign feature weights according to the obstruction state to generate a fusion feature map; 6.3 Obstruction phase division step: divide the obstruction into three phases before, during, and after based on target completeness; 6.4 Staged attention pre-judgment step: assign attention weights according to the obstruction phase and output the preliminary pre-judgment position through the Transformer network; 6.5 Timing context compensation step: extract context feature vectors and calculate dynamic compensation to correct the final pre-judgment position; 6.6 Multi-dimensional weight allocation step: real-time update bimodal fusion weights based on three-dimensional decision factors; 6.7 Closed-loop fine-tuning step: collect sensor feedback data and iteratively optimize algorithm parameters; 6.8 Control instruction generation step: convert the pre-judgment position into control instructions to drive the robot to perform actions. In step 6.7, the closed-loop fine-tuning parameters include staged attention weights, obstruction compensation function coefficients, and bimodal fusion weights.

7. The method of claim 6, wherein, In step 6.2, the feature weight of cross-modal attention fusion dynamically switches with the target obstruction state.

8. The method of claim 6, wherein, ​

Images