Intelligent obstacle avoidance and gesture control method and system fusing machine vision and ultrasonic wave

Abstract

The application relates to the fields of obstacle avoidance and gesture control, and specifically provides an intelligent obstacle avoidance and gesture control method and system fusing machine vision and ultrasonic waves, the method comprising the following steps: step 1, data acquisition; step 2, pretreatment and space-time synchronization; step 3, multi-modal fusion; and step 4, decision output. The application realizes collaborative obstacle avoidance of the fusion of a machine vision sensor and an ultrasonic sensor, expands a detection range and improves environmental adaptability; the multi-modal sensor fusion improves gesture recognition accuracy and response speed. Through the fusion effect of the machine vision and the ultrasonic sensor, the application finally forms a new generation of a sensing and interaction system which is high in reliability, high in accuracy, low in delay and intelligent and self-adaptive.

Description

Technical Field

[0001] This invention relates to the fields of obstacle avoidance and gesture control technology, and in particular to an intelligent obstacle avoidance and gesture control method and system that integrates machine vision and ultrasound. Background Technology

[0002] Gesture control technology, as an important method of human-computer interaction, has been widely used in consumer electronics, smart homes, and robotics in recent years. Currently, mainstream gesture recognition solutions include:

[0003] ① Vision-based recognition: Gesture images are captured by a camera, and then image processing and machine learning algorithms are used for recognition.

[0004] ② Gesture recognition technology based on depth sensors: Common technologies include structured light technology and time-of-flight (ToF) technology.

[0005] ③ Gesture recognition technology based on inertial sensors: Inertial sensors such as accelerometers and gyroscopes are generally used to detect the movement state and direction of the hand, thereby recognizing the gesture.

[0006] ④ Gesture recognition technology based on millimeter-wave radar: The millimeter-wave radar technology developed by Google's Project Soli team can accurately recognize subtle gestures, such as pinching and swiping. This technology is used in Google Pixel series phones.

[0007] ⑤ Multimodal fusion gesture recognition technology: Combining the advantages of multiple sensors, such as visual sensors with inertial sensors and millimeter-wave radar, and integrating data through algorithms, it improves the accuracy and robustness of gesture recognition and is currently widely used in scenarios such as smart cockpits.

[0008] Intelligent obstacle avoidance technologies mainly fall into the following categories: visual obstacle avoidance, lidar obstacle avoidance, ultrasonic obstacle avoidance, and infrared obstacle avoidance.

[0009] Disadvantages of existing technology:

[0010] Gesture recognition: Vision-based recognition is greatly affected by the environment and occlusion, and has privacy and computational cost issues; depth sensor-based technology is expensive, has limited recognition distance, is susceptible to environmental interference, and is bulky; inertial sensor-based technology has cumulative errors, depends on the wearing position, has a narrow recognition range, and high power consumption; millimeter-wave radar-based technology is expensive, recognition accuracy is affected by distance and angle, is susceptible to electromagnetic interference, and has limited ability to recognize complex gestures; multimodal fusion technology is complex, expensive, difficult to fuse data, and also has power consumption and size issues.

[0011] Obstacle avoidance functionality: Single-sensor obstacle avoidance solutions lack reliability in complex environments. Visual obstacle avoidance is greatly affected by lighting, transparent and reflective objects, and complex backgrounds, relying on efficient algorithms and computing power; LiDAR-based obstacle avoidance is costly, its performance degrades in adverse weather conditions, and it has weak reflection from non-metallic objects; ultrasonic obstacle avoidance has a short detection range, is susceptible to noise interference, has angular blind spots, and low resolution; infrared obstacle avoidance has a short detection range and narrow range, is susceptible to ambient light interference, and has weak recognition capabilities for dark and transparent objects.

[0012] Limited control methods: Relies on remote control or mobile app, lacking natural interaction (such as gesture control).

[0013] Poor real-time performance: The sensor data fusion and decision-making algorithms are not optimized, resulting in significant control delays (e.g., gesture → execution takes more than 1 second).

[0014] Current technological gaps: Currently, gesture control and intelligent obstacle avoidance are mostly independent systems (such as gesture control relying on vision / millimeter wave, and obstacle avoidance relying on lidar / infrared). There is no low-cost fusion solution of "machine vision + ultrasound" to simultaneously achieve "full-domain obstacle avoidance" and "highly robust gesture interaction". This results in low system integration, high cost or weak environmental adaptability, making it difficult to meet the integrated needs of "autonomous obstacle avoidance + convenient control" in multiple scenarios. Summary of the Invention

[0015] This invention provides an intelligent obstacle avoidance and gesture control method and system that integrates machine vision and ultrasound, and aims to solve the following problems:

[0016] (1) How to achieve collaborative obstacle avoidance by fusing machine vision sensors and ultrasonic sensors, thereby expanding the detection range and improving environmental adaptability;

[0017] (2) How to improve the accuracy and response speed of gesture recognition through multimodal sensor fusion.

[0018] To achieve the above objectives, the present invention adopts the following technical solution:

[0019] Intelligent obstacle avoidance and gesture control methods integrating machine vision and ultrasound include:

[0020] Step 1. Data Acquisition: Acquire raw machine vision data, raw ultrasonic echo data, and ambient temperature data using machine vision sensors and ultrasonic sensors;

[0021] Step 2. Preprocessing and Spatiotemporal Synchronization:

[0022] Data preprocessing: The raw machine vision data is filtered, illumination compensated, and distortion corrected to obtain preprocessed vision data; the raw ultrasonic echo data is digitally filtered and the ranging value is calibrated in conjunction with ambient temperature data to obtain preprocessed ultrasonic data.

[0023] Time synchronization: Align the timestamps of the preprocessed visual data and the preprocessed ultrasonic data to obtain time-synchronized dual-modal data;

[0024] Spatial registration: Taking the time-synchronized bimodal data as the object, the spatial coordinate transformation parameters are optimized through a calibration algorithm to obtain the time-space synchronized bimodal data;

[0025] Step 3. Multimodal fusion:

[0026] Obstacle avoidance fusion: Based on the spatiotemporally synchronized dual-modal data, visual confidence and ultrasonic confidence are calculated. According to the relative difference between the two types of confidence, the three-dimensional perception results of obstacles are output through dynamic weighted fusion weights.

[0027] Gesture fusion: Fine-align the spatiotemporally synchronized bimodal data, extract visual semantic features and ultrasonic motion features and fuse them to output gesture recognition results;

[0028] Step 4. Decision Output: Generate obstacle avoidance action commands based on the 3D obstacle perception results; generate equipment control commands based on the gesture recognition results.

[0029] In this specification, the intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasound also includes step 5. Command execution and environmental feedback: execute obstacle avoidance action commands and equipment control commands, collect environmental feedback data after command execution, and send it back to step 1 as a supplement to the new round of machine vision raw data and ultrasonic raw echo data.

[0030] In this specification, the intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasound also includes step 6. System stability assurance: periodically calibrate the clock to maintain the time accuracy of the dual-modal data after spatiotemporal synchronization; monitor the sensor status to ensure the effective acquisition of raw machine vision data and raw ultrasonic echo data; when the ambient temperature data changes beyond the threshold, re-perform spatial registration to ensure the spatial accuracy of the dual-modal data after spatiotemporal synchronization.

[0031] In this specification, in step 1, the raw machine vision data includes RGB images and depth images, or ordinary RGB images; the machine vision sensor is a depth camera or a photosensitive module equipped with a monocular depth estimation algorithm.

[0032] In this specification, in step 1, the raw ultrasonic echo data includes distance data, echo intensity data, and echo signal-to-noise ratio data; the ultrasonic sensor supports external triggering, echo signal output, and temperature compensation functions.

[0033] In this specification, in step 2, the distortion correction of the data preprocessing is performed based on the intrinsic parameter matrix and distortion coefficients of the machine vision sensor, which are obtained through camera calibration experiments.

[0034] In this specification, in step 2, time synchronization is performed using either hardware-level synchronization or software-level time synchronization: hardware-level synchronization uses a microcontroller to generate a synchronization pulse to trigger the sensor and record a reference timestamp; software-level time synchronization uses a precision clock protocol to generate timestamp matching data.

[0035] In this specification, in step 3, the visual confidence level of obstacle avoidance fusion is determined based on the image sharpness score and the number and distribution uniformity of feature points; the ultrasonic confidence level is determined based on the echo signal-to-noise ratio and the peak voltage of the echo signal.

[0036] In this specification, in step 3, the visual semantic features of gesture fusion are extracted through a pre-trained convolutional neural network, and the ultrasonic motion features include instantaneous distance, distance change rate and acceleration. The fusion adopts an attention model or feature splicing method.

[0037] An intelligent obstacle avoidance and gesture control system integrating machine vision and ultrasound, applying any one of the above-mentioned intelligent obstacle avoidance and gesture control methods integrating machine vision and ultrasound, wherein the intelligent obstacle avoidance and gesture control system integrating machine vision and ultrasound includes:

[0038] The data acquisition module is used to collect raw machine vision data, raw ultrasonic echo data, and ambient temperature data.

[0039] The preprocessing and spatiotemporal synchronization module is used for:

[0040] Data preprocessing: The raw machine vision data is filtered, illumination compensated, and distortion corrected to obtain preprocessed vision data; the raw ultrasonic echo data is digitally filtered and the ranging value is calibrated in conjunction with ambient temperature data to obtain preprocessed ultrasonic data.

[0041] Time synchronization: Align the timestamps of the preprocessed visual data and the preprocessed ultrasonic data to obtain time-synchronized dual-modal data;

[0042] Spatial registration: Taking the time-synchronized bimodal data as the object, the spatial coordinate transformation parameters are optimized through a calibration algorithm to obtain the time-space synchronized bimodal data;

[0043] The multimodal fusion module is used for:

[0044] Obstacle avoidance fusion: Based on the spatiotemporally synchronized dual-modal data, the confidence scores of vision and ultrasound are calculated, and the three-dimensional perception results of obstacles are output through weighted fusion.

[0045] Gesture fusion: Fine-align the spatiotemporally synchronized bimodal data, extract visual semantic features and ultrasonic motion features and fuse them to output gesture recognition results;

[0046] The decision output module is used to: generate obstacle avoidance action commands based on the 3D perception results of obstacles; and generate device control commands based on the gesture recognition results.

[0047] The instruction execution module is used to execute obstacle avoidance commands and equipment control commands;

[0048] The environmental feedback module is used to collect environmental feedback data after the command is executed, and send it back as a supplement to the new round of machine vision raw data and ultrasonic raw echo data.

[0049] In summary, the present invention has at least the following beneficial effects:

[0050] (1) Significantly improved perception robustness and reliability in complex environments.

[0051] ① Problems solved: Completely solved the problem of perception failure of a single vision sensor in low light, textureless, transparent objects (such as glass doors) and low obstacles (such as ground cables), as well as the problem of a single ultrasonic sensor being unable to identify the type of obstacle.

[0052] ② Technical means and causes of effects:

[0053] a. Benefiting from the "complementary priority" fusion mechanism: This invention does not simply stitch together data from two sensors, but fundamentally allows each to perform its own function and complement each other's blind spots. The ultrasonic sensor directly provides precise distance information that is difficult to perceive visually, effectively penetrating visual blind spots; while machine vision provides semantic category information that ultrasonic waves cannot provide. This natural complementarity in physical characteristics fundamentally expands the system's effective detection range and environmental adaptability.

[0054] b. Benefiting from the "dynamic weighted" fusion mechanism: This invention further introduces a dynamic weight adjustment algorithm based on environmental characteristics (light, occlusion). For example, in scenes with strong light or obstructed vision, the system automatically increases the confidence weight of the ultrasonic sensor, prioritizing its distance data and avoiding the inherent defect of the visual sensor's performance deteriorating sharply in such scenarios; conversely, in open and bright environments, visual data is prioritized. This dynamic adaptive capability makes the reliability of the system's perception no longer dependent on fixed environmental conditions, thus achieving stable perception in all environments. The improved robustness in testing stems from this mechanism.

[0055] (2) It achieves highly accurate and low-latency perception and interaction, improving the system's practicality.

[0056] ① Problems solved: The traditional "visual + ultrasonic simple splicing" solution has solved the problems of perception misalignment and high misjudgment rate caused by data asynchrony and coordinate misalignment, as well as the large algorithm latency that cannot meet the real-time interaction requirements.

[0057] ② Technical means and causes of effects:

[0058] a. Benefiting from the "spatiotemporal synchronization and data registration" technology: through hardware-level synchronous triggering (unified timestamp, error ≤5ms) and precise coordinate mapping (hand-eye calibration), this invention ensures that the "pixels" seen visually and the "distance" measured by ultrasound come from the same time and spatial location. This solves the core pain point of "data source asynchrony" that leads to fusion failure, laying a solid foundation for subsequent high-precision fusion, making the output "category + location + precise distance" 3D perception results realistic and reliable, and significantly reducing the misjudgment rate.

[0059] b. Benefiting from the "layered fusion architecture" and "embedded optimization": At the algorithm layer, the "data-feature-decision" three-layer fusion architecture avoids the limitations of single-level fusion. In particular, the introduction of a cross-modal attention mechanism at the feature layer can intelligently extract and fuse the most effective features from two modalities, rather than simply splicing them together, thereby improving recognition accuracy at the source. At the hardware layer, heterogeneous computing (CPU+NPU) and algorithm lightweighting (model pruning, quantization) techniques are used to allocate computationally intensive tasks to dedicated hardware acceleration, ensuring that the end-to-end latency from perception to decision is optimized to within 100ms. This hardware-software co-optimization is the direct reason for achieving low latency and high real-time performance, making gesture control smooth and obstacle avoidance response timely.

[0060] (3) Enhanced the intelligence and adaptability of the system.

[0061] ① Problem solved: It solves the problem of rigid strategies and suboptimal decision-making in traditional obstacle avoidance systems based on fixed rules in dynamic and complex scenarios (such as densely populated areas).

[0062] ② Technical means and causes of effects:

[0063] Thanks to the introduction of "reinforcement learning decision-making" technology, this invention enables the system to autonomously learn optimal obstacle avoidance strategies (such as slowing down and detouring around people, and stopping abruptly when encountering walls) in simulated environments using algorithms such as PPO / DQN, with "obstacle avoidance success rate" and "movement efficiency" as reward functions. This means that the system's decision-making ability no longer relies on limited rules manually written by engineers, but rather possesses the ability to learn from data and self-optimize, thus making reasonable and efficient decisions in unprecedentedly complex and dynamic environments, thereby improving its intelligence level.

[0064] In summary, the beneficial effects of this invention do not stem from a breakthrough in a single technology, but rather from the organic combination and synergistic effect of a series of technical means, including "complementary fusion mechanism," "dynamic weighting algorithm," "spatiotemporal synchronization registration," "layered fusion architecture," and "software and hardware co-optimization." Through these specific means, this invention successfully achieves a "1+1>2" fusion effect between machine vision and ultrasonic sensors, ultimately forming a highly reliable, highly accurate, low-latency, and intelligently adaptive next-generation perception and interaction system. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram of the intelligent obstacle avoidance and gesture control method that integrates machine vision and ultrasound involved in this invention.

[0067] Figure 2 This is a flowchart illustrating the intelligent obstacle avoidance and gesture control system that integrates machine vision and ultrasound, as described in this invention.

[0068] Figure 3 This is a schematic diagram of the parallel data acquisition using dual sensors involved in this invention.

[0069] Figure 4 This is a schematic diagram illustrating the data calibration and alignment involved in this invention.

[0070] Figure 5 This is a schematic diagram of the multimodal fusion involved in this invention.

[0071] Figure 6 This is a schematic diagram of the decision output involved in this invention.

[0072] Figure 7 This is a schematic diagram illustrating the instruction execution and feedback involved in this invention.

[0073] Figure 8 This is a schematic diagram illustrating the key technical safeguards involved in this invention. Detailed Implementation

[0074] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0075] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0076] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0077] like Figure 1 As shown, this embodiment provides an intelligent obstacle avoidance and gesture control method that integrates machine vision and ultrasound, including:

[0078] Step 1. Data Acquisition: Acquire raw machine vision data, raw ultrasonic echo data, and ambient temperature data using machine vision sensors and ultrasonic sensors;

[0079] Step 2. Preprocessing and Spatiotemporal Synchronization:

[0080] Data preprocessing: The raw machine vision data is filtered, illumination compensated, and distortion corrected to obtain preprocessed vision data; the raw ultrasonic echo data is digitally filtered and the ranging value is calibrated in conjunction with ambient temperature data to obtain preprocessed ultrasonic data.

[0081] Time synchronization: Align the timestamps of the preprocessed visual data and the preprocessed ultrasonic data to obtain time-synchronized dual-modal data;

[0082] Spatial registration: Taking the time-synchronized bimodal data as the object, the spatial coordinate transformation parameters are optimized through a calibration algorithm to obtain the time-space synchronized bimodal data;

[0083] Step 3. Multimodal fusion:

[0084] Obstacle avoidance fusion: Based on the spatiotemporally synchronized dual-modal data, visual confidence and ultrasonic confidence are calculated. According to the relative difference between the two types of confidence, the three-dimensional perception results of obstacles are output through dynamic weighted fusion weights.

[0085] Gesture fusion: Fine-align the spatiotemporally synchronized bimodal data, extract visual semantic features and ultrasonic motion features and fuse them to output gesture recognition results;

[0086] Step 4. Decision Output: Generate obstacle avoidance action commands based on the 3D obstacle perception results; generate equipment control commands based on the gesture recognition results.

[0087] In some embodiments, the intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasound further includes step 5. Instruction execution and environmental feedback: executing obstacle avoidance action instructions and device control instructions, collecting environmental feedback data after instruction execution, and sending it back to step 1 as a supplement to a new round of machine vision raw data and ultrasound raw echo data.

[0088] In some embodiments, the intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasound further includes step 6. System stability assurance: periodically calibrating the clock to maintain the temporal accuracy of the bimodal data after spatiotemporal synchronization; monitoring the sensor status to ensure the effective acquisition of raw machine vision data and raw ultrasonic echo data; and re-performing spatial registration when the ambient temperature data changes beyond a threshold to ensure the spatial accuracy of the bimodal data after spatiotemporal synchronization.

[0089] In some embodiments, in step 1, the raw machine vision data includes RGB images and depth images, or ordinary RGB images; the machine vision sensor is a depth camera or a photosensitive module equipped with a monocular depth estimation algorithm.

[0090] In some embodiments, in step 1, the raw ultrasonic echo data includes distance data, echo intensity data, and echo signal-to-noise ratio data; the ultrasonic sensor supports external triggering, echo signal output, and temperature compensation functions.

[0091] In some embodiments, in step 2, the distortion correction of the data preprocessing is performed based on the intrinsic parameter matrix and distortion coefficients of the machine vision sensor, which are obtained through camera calibration experiments.

[0092] In some embodiments, in step 2, time synchronization is performed using either hardware-level synchronization or software-level time synchronization: hardware-level synchronization uses a microcontroller to generate a synchronization pulse to trigger a sensor and record a reference timestamp; software-level time synchronization uses a precision clock protocol to generate timestamp matching data.

[0093] In some embodiments, in step 3, the visual confidence level of obstacle avoidance fusion is determined based on image sharpness score and the number and distribution uniformity of feature points; the ultrasonic confidence level is determined based on echo signal-to-noise ratio and echo signal peak voltage.

[0094] In some embodiments, in step 3, the visual semantic features of gesture fusion are extracted through a pre-trained convolutional neural network, and the ultrasonic motion features include instantaneous distance, distance change rate and acceleration. The fusion adopts an attention model or feature splicing method.

[0095] In some embodiments, in step 4, the obstacle avoidance action command is generated by a reinforcement learning algorithm or a path planning algorithm. The reinforcement learning algorithm is selected from PPO or DQN algorithms, and the path planning algorithm is selected from dynamic window method or A* algorithm.

[0096] An intelligent obstacle avoidance and gesture control system integrating machine vision and ultrasound, applying any one of the above-mentioned intelligent obstacle avoidance and gesture control methods integrating machine vision and ultrasound, such as... Figure 2 As shown, the intelligent obstacle avoidance and gesture control system integrating machine vision and ultrasound includes:

[0097] The data acquisition module is used to collect raw machine vision data, raw ultrasonic echo data, and ambient temperature data; the dual-sensor parallel data acquisition process is as follows: Figure 3 As shown;

[0098] The preprocessing and spatiotemporal synchronization module is used for:

[0099] Data preprocessing: The raw machine vision data is filtered, illumination compensated, and distortion corrected to obtain preprocessed vision data; the raw ultrasonic echo data is digitally filtered and the ranging value is calibrated in conjunction with ambient temperature data to obtain preprocessed ultrasonic data.

[0100] Time synchronization: Align the timestamps of the preprocessed visual data and the preprocessed ultrasonic data to obtain time-synchronized dual-modal data;

[0101] Spatial registration: Taking the time-synchronized bimodal data as the object, the spatial coordinate transformation parameters are optimized through a calibration algorithm to obtain the time-space synchronized bimodal data;

[0102] The multimodal fusion module is used for:

[0103] Obstacle avoidance fusion: Based on the spatiotemporally synchronized dual-modal data, the confidence scores of vision and ultrasound are calculated, and the three-dimensional perception results of obstacles are output through weighted fusion.

[0104] Gesture fusion: Fine-align the spatiotemporally synchronized bimodal data, extract visual semantic features and ultrasonic motion features and fuse them to output gesture recognition results;

[0105] The decision output module is used to: generate obstacle avoidance action commands based on the 3D perception results of obstacles; and generate device control commands based on the gesture recognition results.

[0106] The instruction execution module is used to execute obstacle avoidance commands and equipment control commands;

[0107] The environmental feedback module is used to collect environmental feedback data after the command is executed, and send it back as a supplement to the new round of machine vision raw data and ultrasonic raw echo data.

[0108] The technical concept of this invention is as follows:

[0109] The core technology of this invention lies in the deep fusion and collaborative computing of machine vision and ultrasonic sensors, specifically covering the following key areas and technical aspects:

[0110] (1) Intelligent mobile robot platform technology

[0111] As the application platform for the system, this field focuses on the autonomous navigation and interactive control of mobile devices (such as service robots, drones, and unmanned vehicles). Key areas include:

[0112] ① High-reliability obstacle avoidance navigation: Combining the robot's kinematic model, the fused perception results (obstacle type, position, distance) are converted into obstacle avoidance actions (such as emergency stop, detour, and crossing) in real time. In particular, it relies on the stable distance information provided by ultrasonic waves in low light and textureless environments to make up for the shortcomings of pure vision positioning.

[0113] ② Natural human-computer interaction control: Based on the recognized gesture semantics (such as waving, clenching fist), generate device control commands (such as start, speed adjustment, hovering) to achieve intuitive and contactless human-computer interaction.

[0114] (2) Machine vision perception technology

[0115] As the core of environmental and gesture semantic understanding, it mainly includes:

[0116] ① Image enhancement and preprocessing: Image quality is improved by using methods such as filtering, illumination compensation, and distortion correction to cope with complex environments such as backlight and low light;

[0117] ② Deep learning visual recognition: CNN models are used to classify static gestures and identify obstacle types (such as distinguishing pedestrians and walls), and RNN or LSTM networks are used to process dynamic gesture temporal information (such as trajectory tracking).

[0118] ③ Visual localization and segmentation: Target pixel-level position and contour are extracted using algorithms such as YOLO and Mask R-CNN, providing a spatial basis for fusion with ultrasonic data.

[0119] (3) Dynamic calculation scheme based on real-time performance evaluation

[0120] The core of this technical solution lies in an advanced sensor fusion strategy. It abandons the traditional simple fusion method based on fixed rules or preset weights, and instead adopts a dynamic weighted fusion method based on real-time data quality assessment. This method constructs a built-in confidence assessment module to quantitatively analyze real-time data from visual and ultrasonic sensors, and autonomously and dynamically allocates fusion weights based on the analysis results, thereby achieving optimal performance of the sensing system in complex and changing environments. The specific implementation method is as follows:

[0121] ① Confidence assessment module

[0122] This system defines and calculates real-time confidence index (CI) for various sensors. This index is a continuous, quantified value used to objectively and directly reflect the reliability of sensor data under specific current conditions. The weight coefficients of each sub-index are determined through regression analysis of a large amount of experimental data or through the analytic hierarchy process (AHP) based on expert experience, and are then embedded in the system.

[0123] a. Visual confidence index (C_vision) calculation: The confidence assessment of the visual sensor is based on the quality and feature richness of its image data, mainly including two dimensions: image sharpness and feature availability, and is combined according to a preset weight (e.g., 6:4).

[0124] Image clarity quantification: Algorithms such as the Brenner gradient function (calculation window is 3x3 pixels) or the Tenengrad gradient function are used to calculate the sum of gradient magnitudes of the entire image or the region of interest (ROI) as the clarity score (S_clarity). A large number of samples are collected under different lighting conditions to statistically determine the threshold range for clarity: when S_clarity is below the empirical threshold T_low (e.g., 80), the image is considered blurry; when it is above the empirical threshold T_high (e.g., 200), the image is considered sharp. This score decreases significantly under conditions of strong light (overexposure leading to loss of image detail), low light (increased noise), or motion blur.

[0125] Feature Usability Analysis: Run lightweight ORB or Fast feature point detection algorithms. The discernibility of environmental content is determined by statistically analyzing the number of stable feature points (N_features) and their distribution uniformity (e.g., quantified by calculating the standard deviation σ of feature point locations). When the number of feature points N_features is below a threshold T_features (e.g., 50) or the distribution uniformity σ is above a threshold T_distribution (e.g., 10), it indicates sparse or concentrated features, making effective sensor identification and tracking difficult. This situation often occurs in textureless areas (e.g., solid-color walls), in front of transparent objects, or when the target is too far away.

[0126] Conclusion: The C_vision value is calculated by combining the above indicators, for example: Where w1 and w2 are weighting coefficients, and f() and g() are functions that normalize the original index to the [0,1] interval. The lower the value, the less reliable the visual data is in the current environment.

[0127] b. Calculation of the ultrasonic confidence index (C_ultrasonic):

[0128] The confidence evaluation of the ultrasonic sensor is based on the quality of its echo signal, mainly including:

[0129] Echo signal-to-noise ratio (SNR) analysis: Near the center frequency of the ultrasonic sensor (for example, within the 38kHz - 42kHz bandwidth of a 40kHz sensor), calculate the ratio (SNR) of the effective echo signal power to the ambient background noise power. Through experimental determination, when the SNR is below the threshold T_snr (for example, 10dB), it indicates that the ranging data is at high risk of being affected by co-frequency acoustic noise interference, multipath reflection, or adverse weather (such as rain, fog).

[0130] Echo intensity monitoring: Monitor the peak voltage (V_echo) of the received echo signal. Determine the normal operating range (V_min, V_max) of the echo intensity through sensor calibration experiments (for example, 0.5V to 2.0V). An echo signal that is too weak (V_echo < V_min) may mean that the target is sound-absorbing, the ranging distance is close to the limit, or the incident angle is not good; an echo that is too strong (V_echo > V_max) may mean signal saturation or the presence of a strong reflector in front. Abnormal echo intensities all indicate that the ranging result may be biased.

[0131] Conclusion: The value of C_ultrasonic is calculated comprehensively from the above indicators (such as weighted average). The lower its value, the less reliable the ultrasonic data is in the current environment.

[0132] ② Dynamic soft weight assignment mechanism

[0133] After obtaining the real-time confidence index, the system uses a normalization function (i.e., the binary special case of the Softmax function) to automatically calculate the final fusion weights of each sensor, rather than calling any pre-set lookup table or fixed threshold.

[0134] Weight calculation formula:

[0135] Weight of the vision sensor: ;

[0136] Weight of the ultrasonic sensor: ;

[0137] That is ;

[0138] Advantages and effects of the mechanism:

[0139] The dynamic weight allocation mechanism of this scheme is highly adaptive. Its output weights are dynamically determined entirely by the relative magnitudes of the real-time confidence indices of the visual and ultrasonic sensors, achieving fully automatic environmental adaptation without human intervention. Through Softmax normalization, the sum of the weights of the two sensors is always guaranteed to be 1, conforming to probabilistic principles and facilitating subsequent fusion calculations. The introduction of an exponential function amplifies the advantages of high-confidence sensors; when the data quality of one sensor is significantly superior to that of the other, its weight will infinitely approach 1, thus occupying an absolute dominant position in the fusion result. The final generated weight values ​​are entirely the natural result of the mathematical calculation process, completely avoiding the arbitrariness of subjective parameter presets and greatly enhancing the scientific rigor and reliability of the scheme.

[0140] (4) Spatiotemporal synchronization and data registration technology (data calibration and alignment process as follows) Figure 4 (As shown)

[0141] ① Hardware-level time synchronization system

[0142] The hardware synchronization system in this solution uses an STM32H743 microcontroller as the master clock source, generating a precise synchronization pulse with a rise time of <10ns and a width of 2us through its timer TIM1. This pulse simultaneously triggers an Intel RealSense D435i depth camera (connected to the GPIO_TRIGGER pin, configured with a 1.5ms trigger delay and 8ms global shutter exposure) and a MaxBotix MB1043 ultrasonic sensor (connected to the TRIG pin, activating a 160us receiving window). A DS18B20 temperature sensor is used to monitor the ambient temperature in real time, based on the sound speed compensation formula. Real-time calibration of ultrasonic ranging ensures hardware-level synchronization of visual and ultrasonic data acquisition at the microsecond level.

[0143] ② Software timestamp system

[0144] This system utilizes a high-precision clock architecture built upon the MCU's DWT period counter, providing timing accuracy up to 6ns (168MHz). It achieves microsecond-level time synchronization via a 32-bit monotonic clock and employs an NTP-based synchronization algorithm to control clock drift between master and slave devices to within 1µs / min. For timestamp management, the system records a reference timestamp T_trigger immediately upon the rising edge of the hardware trigger pulse in the interrupt service routine. A 1.2ms CSI-2 interface transmission delay is added to visual data (T_image = T_trigger + 1.2ms), and a 60ms measurement delay and temperature compensation delay (T_ultrasonic = T_trigger + 60ms + ΔT_temp) are added to ultrasonic data. Finally, a timestamp hash table indexed by the pulse sequence number ensures the time consistency of multimodal data.

[0145] ③ Spatial coordinate registration system

[0146] This system employs a customized black and white checkerboard calibration board (6×8 squares, 30mm×30mm specification) in conjunction with a high-precision laser rangefinder (±0.1mm) and a three-dimensional translation stage (control accuracy 0.01mm / 0.1°) to form a calibration device. The calibration process involves collecting 50 sets of data at each of 20 different locations (distance 0.5~5m, angle ±45°) and averaging the results to reduce random errors. A transformation model [R|t] (where R is the rotation matrix and t is the translation vector) from the visual coordinate system to the ultrasonic coordinate system is established. The Levenberg-Marquardt nonlinear optimization algorithm is used to minimize Σ(|P_u-(R·P_c+t)|²) as the objective function. Initial values ​​are obtained through SVD decomposition. Ultimately, a high-precision calibration result is achieved with a reprojection error of less than 0.3 pixels and a distance error of less than 1cm.

[0147] ④ Real-time data processing pipeline

[0148] This system efficiently receives visual CSI-2 data streams through a DMA double-buffering mechanism, while simultaneously acquiring ultrasonic ADC data at a 500kHz sampling rate and performing digital filtering. It employs a timestamp-based nearest neighbor matching algorithm (search window ±2ms) to achieve accurate alignment of multimodal data. In terms of anomaly handling, the system compensates for lost visual frames through historical data interpolation, uses Kalman filtering to predict missing ultrasonic signals, and establishes a sequence number-based resynchronization mechanism to handle timestamp anomalies, ensuring the integrity of data acquisition and system robustness.

[0149] ⑤ System fault tolerance mechanism

[0150] This system integrates three major safeguards to ensure long-term operational stability: maintaining time synchronization accuracy through a clock drift monitoring and correction algorithm every 10 seconds; employing a heartbeat mechanism with 500ms intervals to monitor sensor health in real time; and automatically triggering a recalibration process when ambient temperature changes exceed ±5°C, comprehensively ensuring the system's sensing reliability under various environmental conditions. Key technical safeguard processes are as follows: Figure 8 As shown.

[0151] ⑥ Dynamic Weighting Mechanism: The dynamic weighting process in this scheme does not require independent "startup conditions." Essentially, each time the system acquires a new set of sensor data, the confidence assessment module calculates a new set of C_vision and C_ultrasonic in real time, and then immediately outputs a new set of fusion weights, Weight_vision and Weight_ultrasonic, through the Softmax function. This process is synchronized with the data acquisition frame rate and is an inherent and continuous part of the system's data processing pipeline.

[0152] The "step size" of the weight adjustment is not a preset fixed value (such as 0.1), but is determined non-linearly and adaptively by the relative difference in confidence levels between the two sensors through an exponential function.

[0153] When one party's confidence level is significantly higher than the other's (e.g., C_vision >> C_ultrasonic), e C_vision It will be much larger than e C_ultrasonic This causes Weight_vision to approach 1 and Weight_ultrasonic to approach 0. At this point, the "step size" effect of weight adjustment becomes significant, and the system will quickly tend to trust the more reliable sensor.

[0154] When the confidence levels of both sides are similar (e.g., C_vision≈C_ultrasonic), the weight ratio will be close to 1:1. At this point, any small change in confidence will cause a smooth, minor adjustment in the weights, achieving a natural transition in weight changes and avoiding output jitter that may be caused by a fixed step size.

[0155] This normalization calculation based on the Softmax function is a perfect mechanism that combines "triggering" (any new confidence input will immediately trigger recalculation) and "step control" (the greater the difference, the greater the adjustment), thus achieving fully automatic, adaptive, and smooth dynamic weighted fusion.

[0156] This technical solution achieves high-precision spatiotemporal synchronization between visual and ultrasonic sensors through sophisticated hardware design, accurate clock management, and refined calibration algorithms, providing a reliable data foundation for subsequent fusion perception algorithms.

[0157] (5) A hierarchical fusion method for gesture recognition based on cross-modal attention mechanism and spatiotemporal synchronization

[0158] The core innovation of this method lies in constructing a three-layer fusion architecture of data, features, and decision-making, and introducing a cross-modal attention mechanism. This method first solves the spatiotemporal alignment problem between visual and ultrasonic sensors through hardware-level synchronization and coordinate mapping; then, at the feature layer, it dynamically weights the semantic features of vision and the motion / distance features of ultrasound using an attention model; finally, at the decision layer, it outputs the result through bimodal voting and scene-adaptive thresholding. The three-layer fusion process of "data-feature-decision" is as follows: Figure 5 As shown.

[0159] ① Visual semantic feature extraction

[0160] Visual semantic features are extracted using a pre-trained deep convolutional neural network (CNN). This network is pre-trained on a large-scale image dataset (such as ImageNet) and fine-tuned for a gesture recognition task using a specific dataset. Features are extracted from the network's deep semantic feature layers (e.g., for a ResNet-based Backbone architecture, the output of its layer 4 or layer 3 convolutional blocks can be extracted) to obtain feature representations rich in high-level semantic information.

[0161] The visual features, after being processed by global average pooling, are mapped into a fixed-length feature vector. The dimension of this vector is set according to the balance between model complexity and computational resources (e.g., 256-dimensional or 512-dimensional). This design ensures both the representational power of the features and controls the computational complexity, thus meeting real-time requirements.

[0162] In one specific embodiment, the output of layer 4 of a ResNet-34 model pre-trained on ImageNet is used, and after global average pooling, a 512-dimensional feature vector is obtained as the visual semantic feature.

[0163] ② Extraction of ultrasonic motion and distance features

[0164] Ultrasonic features consist of two components: instantaneous distance features and temporal motion features.

[0165] The instantaneous distance characteristic refers to the precise ranging value of the current frame after temperature and humidity compensation, in meters. This value is calibrated using the sound speed compensation formula v=331.4+0.6T (where T is in degrees Celsius) to ensure a ranging accuracy of ±1cm.

[0166] Temporal motion characteristics are dynamic parameters calculated by analyzing ranging data from multiple consecutive frames within a temporal sliding window. Specifically, within a set time window (e.g., 5 consecutive frames, corresponding to a time window length of approximately 150ms), the rate of change of distance (i.e., velocity, unit: m / s) and acceleration (unit: m / s²) are calculated. The window size can be dynamically adjusted according to the application's requirements for real-time performance and smoothness.

[0167] These features together constitute a low-dimensional ultrasonic feature vector (e.g., containing instantaneous distance, average velocity and acceleration over the past 4 frames, totaling 5-15 dimensions) used to accurately describe the target's motion state and position information.

[0168] ③ Cross-modal attention fusion mechanism

[0169] The cross-modal attention mechanism employed in this invention is based on calculating the relevance weights between visual semantic features (as the Query) and ultrasonic motion features (as the Key and Value). Specifically, it uses a ScaledDot-Product Attention model for calculation, expressed by the following formula:

[0170] ;

[0171] Where Q represents the query matrix after linear transformation of the visual feature vector, K and V represent the key matrix and value matrix after linear transformation of the ultrasonic feature vector, and d k Let be the dimension of the key vector. This is a scaling factor used to prevent the gradient of the softmax function from vanishing due to an excessively large dot product result.

[0172] In the calculation process, the 512-dimensional visual features are first mapped to a query vector Q through a linear transformation layer, and the 15-dimensional ultrasound features are mapped to a key vector K and a value vector V through different linear transformation layers, respectively. Then, the attention score matrix is ​​calculated, which reflects the degree of attention the visual features pay to different ultrasound feature elements. The final weighted fusion feature output integrates the complementary information of the two modalities, providing an enriched feature representation for subsequent classification decisions. The scenario-based instruction generation process (decision output) is as follows: Figure 6 As shown.

[0173] (6) Embedded systems and real-time control technology

[0174] As the foundation for system hardware implementation and efficient operation:

[0175] ① Heterogeneous computing architecture: adopts CPU+NPU / FPGA solutions (such as Jetson Nano, RK3588) to process visual inference, ultrasonic data processing and fusion decision-making tasks in parallel;

[0176] ② Real-time performance optimization: Through lightweight models (pruning, quantization), real-time operating systems (RT-Thread), and hardware acceleration, the end-to-end latency from perception to decision-making is ensured to be less than 100ms, meeting the real-time requirements of obstacle avoidance and interaction.

[0177] (7) Artificial intelligence and machine learning technologies

[0178] This invention employs reinforcement learning algorithms (such as PPO and DQN) to train an obstacle avoidance strategy model, enabling it to autonomously learn optimal behavior in complex dynamic scenarios within a simulated environment. The core of this training framework lies in a carefully designed multi-objective reward function.

[0179] Reward function design principle: The function consists of multiple reward and penalty terms linearly weighted, aiming to simultaneously optimize obstacle avoidance success rate, movement efficiency and safety.

[0180] During the training of the agent, we employ the following reward mechanism: A significant positive reward is given when the agent successfully avoids obstacles and reaches the target point; simultaneously, to encourage the selection of efficient and smooth paths, an efficiency reward is implemented, which is related to the stability of the agent's movement speed and the ratio of the actual path length to the optimal path; furthermore, significant negative penalties are imposed for behaviors such as colliding with obstacles, performing dangerous actions (e.g., sharp turns), or exceeding the task timeout. Instruction execution and environmental feedback: The closed-loop optimization process is as follows... Figure 7 As shown.

[0181] The weight coefficients of each reward item (such as α, β, γ) are determined by hyperparameter tuning methods such as grid search or Bayesian optimization to balance the priorities of different objectives.

[0182] Simulation Environment Construction: Training is conducted in a highly randomized simulation environment. The environment simulates various complex scenarios (such as narrow corridors, open spaces, and dynamic crowds), where the moving speed of dynamic obstacles is randomly generated within a set range (e.g., 0.5 m / s to 2.0 m / s), and the number, density, and distribution of obstacles also vary randomly to ensure that the trained policy has high generalization ability and robustness.

[0183] Training Execution: The training process utilizes an Experience Replay mechanism to break down correlations between data points and employs a fixed learning rate plan. Specific hyperparameters (such as learning rate, discount factor, replay pool size, and total training steps) follow community-recognized default or standard settings based on the chosen algorithm (PPO or DQN), and are fine-tuned through a small number of experiments. The trained policy network is ultimately deployed to an embedded platform for real-time obstacle avoidance decision-making.

[0184] In summary, this technical solution, through the deep integration and system optimization of multidisciplinary technologies, achieves highly reliable and low-latency environmental perception and interactive control in complex real-world scenarios, demonstrating significant advancement and practicality.

[0185] In some embodiments, different models or types of visual sensors are used: In the main embodiment of the present invention, the Intel RealSense D435i depth camera is used as the core sensor, while also providing a variety of alternatives. Other brands of RGB-D cameras such as the Orbbec Astra series and Microsoft Azure Kinect DK can be selected, as well as ordinary RGB cameras such as the Sony IMX series photosensitive modules, paired with monocular depth estimation algorithms such as MiDaS and LeReS. Regardless of the sensor solution used, the acquired image information will undergo subsequent image enhancement, feature extraction, and fusion processing to achieve obstacle recognition and gesture judgment functions. Although different sensors may have slight differences in depth accuracy, frame rate, or low-light performance, the fusion architecture proposed in this invention can still effectively solve the limitations of single visual perception, enabling the system to have basic environmental adaptability.

[0186] In some embodiments, different models or types of ultrasonic sensors are used: In the implementation of this invention, the MaxBotix MB1043 ultrasonic sensor is selected. Other alternatives can also be used, such as the Senix® TSPC-30S1 with a detection range of up to 10 meters, or the low-cost and commonly used HY-SRF05 ultrasonic ranging module. However, these alternative sensors must meet the key requirements of supporting external triggering and echo signal output and having a certain temperature compensation function. Whether it is the core sensor or a qualified alternative sensor, they can all provide distance information. They will participate in the fusion perception through the synchronous triggering mechanism and temperature compensation process of this system. Although different sensors may have differences in ranging accuracy, anti-interference ability, or maximum range, they can all effectively make up for the blind spots of vision in the recognition of transparent objects and perception in low light environments, and provide the system with the distance information required for basic obstacle avoidance.

[0187] In some embodiments, a non-attention mechanism feature fusion method is employed: This invention uses a cross-modal attention mechanism in the feature layer fusion stage, and also provides simpler alternative feature fusion strategies, including direct concatenation of visual feature vectors and ultrasonic feature vectors, element-wise addition fusion, and fusion through weighted average. When using these alternative strategies, the fused feature vector obtained after concatenation or computation can be directly input into subsequent classifiers such as fully connected networks for gesture recognition or obstacle judgment. This type of method can avoid the complex calculation process of the attention mechanism, greatly reducing the consumption of computing resources, and is therefore more suitable for embedded platforms with limited computing power. Although its fusion intelligence and adaptability in complex scenarios are not as good as the cross-modal attention mechanism, it can still comprehensively utilize visual and ultrasonic dual-modal information to effectively improve the robustness of single sensor recognition.

[0188] In some embodiments, a rule-based dynamic weight adjustment strategy is adopted: This invention dynamically calculates the fusion weights using the Softmax function combined with real-time confidence. An alternative is to manually adjust the weights based on preset environmental rules. For example, the system can incorporate a TI OPT3001 illuminance sensor. When the ambient illuminance is below 50 Lux, the ultrasonic weight is set to 0.8 and the visual weight to 0.2; when the illuminance is above 500 Lux, the visual weight is set to 0.8 and the ultrasonic weight to 0.2. When using this alternative, the system queries the current environmental state through a rule base and applies the corresponding weights for fusion. This method is simple to implement, requiring no complex exponential calculations or real-time confidence assessment modules. Although it cannot achieve the fine, smooth, and adaptive weight adjustment effect of this invention, and its response to sudden environmental changes is somewhat abrupt, it can still switch the dominant sensor based on macroscopic environmental changes, providing more reliable perception results than fixed-weight fusion in most typical scenarios.

[0189] In some embodiments, traditional planning algorithms are used instead of reinforcement learning for obstacle avoidance decision-making: In the obstacle avoidance strategy training, this invention uses reinforcement learning algorithms such as PPO and DQN, and can also use traditional robot path planning algorithms as alternatives, such as Dynamic Window (DWA), A* algorithm and its variants. These traditional algorithms can perform real-time path planning based on the obstacle positions output by the fusion perception module and existing map information (if available). In actual operation, the obstacle category, position, distance and other results obtained by fusion perception are used as input to the cost map. The traditional planning algorithm calculates a collision-free path that can reach the target point based on the cost map and outputs control commands such as linear velocity and angular velocity. This method does not rely on a large amount of data for training and simulation environment. It is not only highly deterministic but also easy to debug. Although its intelligence and flexibility are not as good as reinforcement learning models in extremely complex and dynamic environments, it can still achieve effective basic obstacle avoidance functions.

[0190] In some embodiments, software time synchronization is used instead of hardware synchronization: This invention uses hardware-level synchronization pulse triggering based on an STM32 microcontroller to achieve sensor data synchronization. Alternatively, an alternative solution that eliminates the need for a dedicated hardware synchronization circuit can be adopted. Specifically, in an environment such as Linux, a high-precision software timestamp is generated using the PTP precision clock protocol to align the visual frame with the ultrasonic measurement data. Time point matching is then completed at the software level through interpolation or extrapolation. In actual operation, the operating system assigns high-precision timestamps to the data collected by different sensors. The data fusion module then pairs the data based on these timestamps. This solution effectively reduces hardware complexity and cost. Although its synchronization accuracy (usually in the millisecond range) is slightly lower than that of hardware synchronization (in the microsecond range), it may introduce larger registration errors in high-speed motion scenarios. However, it can still meet the basic data fusion requirements for low- and medium-speed applications such as service robots and smart wheelchairs.

[0191] Key technical innovations of this invention:

[0192] (1) Machine vision and ultrasonic dynamic fusion to achieve all-domain obstacle avoidance

[0193] This invention pioneers a dual-sensor fusion mechanism that prioritizes complementarity and uses dynamic weighting as a supplement. It effectively compensates for the blind spots in machine vision in low-light, textureless, transparent, and low-obstacle scenarios using ultrasound, while simultaneously leveraging visual capabilities to accurately identify obstacle categories. Furthermore, it dynamically adjusts sensor confidence weights based on ambient light and occlusion characteristics (the dynamic weighted fusion mechanism is a continuously running, closed-loop optimization process; its weight allocation is based on real-time calculated sensor confidence indices (C_vision, C_ultrasonic) that change continuously and smoothly without requiring a specific trigger threshold). Combined with spatial coordinate registration and time synchronization technologies, it ultimately outputs a stable 3D perception result that fuses "category + location + precise distance," effectively solving the misjudgment problems of traditional solutions in scenarios such as glass doors and dimly lit corridors.

[0194] (2) A hierarchical fusion method for gesture recognition based on cross-modal attention mechanism and spatiotemporal synchronization

[0195] The core innovation of this method lies in constructing a three-layer fusion architecture of "data-feature-decision" and introducing a cross-modal attention mechanism. First, it solves the spatiotemporal alignment problem between visual and ultrasonic sensors through hardware-level synchronization and coordinate mapping; then, at the feature layer, it uses an attention model to dynamically weight the semantic features of vision and the motion / distance features of ultrasound (such as increasing the weight of ultrasound in low-light environments); finally, at the decision layer, it outputs results through bimodal voting and scene-adaptive thresholding, thereby achieving highly accurate and low-latency gesture recognition in complex scenarios such as low light and occlusion.

[0196] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values ​​or substitutions of equivalent elements should still fall within the scope of this invention.

[0197] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.

[0198] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

[0199] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0200] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of the invention. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to the invention. Such modifications, improvements, and corrections are suggested in this invention and therefore remain within the spirit and scope of the exemplary embodiments of the invention.

[0201] Furthermore, this invention uses specific terms to describe embodiments of the invention. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the invention. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the invention can be appropriately combined.

[0202] Furthermore, those skilled in the art will understand that aspects of the present invention can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of the present invention can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software. All of the above hardware or software can be referred to as "units," "modules," or "systems." Furthermore, aspects of the present invention can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0203] The computer program code required for the operation of each part of this invention can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, and Python; conventional procedural programming languages ​​such as C; Visual Basic, Fortran2103, Perl, COBOL2102, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0204] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this invention are not intended to limit the order of the processes and methods of this invention. Although the foregoing disclosure has discussed some currently considered useful embodiments of the invention through various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments of this invention. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.

[0205] Similarly, it should be noted that, in order to simplify the description of this invention and thus aid in the understanding of one or more embodiments, multiple features may sometimes be grouped into a single embodiment, drawing, or description thereof in the foregoing description of embodiments of the invention. However, the method of the present invention should not be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject of the invention should possess fewer features than in any single embodiment described above.

Claims

1. A method of intelligent obstacle avoidance and gesture control fusing machine vision and ultrasonic waves, characterized in that, The application relates to a method for multi-modal fusion of obstacle avoidance and gesture recognition, comprising the following steps: Step 1. Data acquisition: machine vision raw data, ultrasonic raw echo data and environmental temperature data are collected by machine vision sensors, ultrasonic sensors and temperature sensors respectively; Step 2. Preprocessing and time-space synchronization: Data preprocessing: the machine vision raw data is filtered, light compensation and distortion correction are performed, and preprocessed visual data is obtained; The ultrasonic raw echo data is digitally filtered, and the ranging value is calibrated in combination with the environmental temperature data, and preprocessed ultrasonic data is obtained; Time synchronization: the time stamps of the preprocessed visual data and the preprocessed ultrasonic data are aligned, and time-synchronized bimodal data is obtained; Space registration: taking the time-synchronized bimodal data as the object, the space coordinate conversion parameters are optimized through a calibration algorithm, and the time-space synchronized bimodal data is obtained; Step 3. Multi-modal fusion: Obstacle avoidance fusion: the visual confidence and the ultrasonic confidence are calculated based on the time-space synchronized bimodal data, the relative difference between the two types of confidence is obtained, the dynamic weighted fusion weight is obtained through dynamic weighting, and the three-dimensional obstacle perception result is output; the visual confidence is calculated based on the image definition score and the feature point quantity and distribution uniformity; the ultrasonic confidence is calculated based on the echo signal-to-noise ratio and the echo signal peak voltage; the dynamic weighted fusion weight is calculated through a Softmax function according to the relative difference between the two types of confidence; Gesture fusion: the data registration before fusion is performed on the time-space synchronized bimodal data, the visual semantic features and the ultrasonic motion features are extracted and fused, and the gesture recognition result is output; the visual semantic features are extracted through a pre-trained convolutional neural network, the ultrasonic motion features include instantaneous distance, distance change rate and acceleration, and the fusion adopts an attention model or a feature splicing mode; Step 4. Decision output: the obstacle avoidance action instruction is generated based on the three-dimensional obstacle perception result; the device control instruction is generated based on the gesture recognition result.

2. The intelligent obstacle avoidance and gesture control method fusing machine vision and ultrasonic wave according to claim 1, characterized in that, Further comprising step 5. Instruction execution and environmental feedback: the obstacle avoidance action instruction and the device control instruction are executed, the environmental feedback data after the instruction execution is collected, and the environmental feedback data is returned to step 1 as the supplement of the machine vision raw data and the ultrasonic raw echo data in the next round. 3.The intelligent obstacle avoidance and gesture control method of fusing machine vision and ultrasonic wave according to claim 2, wherein, Further comprising step 6. System stability guarantee: the clock is calibrated regularly to maintain the time accuracy of the time-space synchronized bimodal data; the sensor state is monitored to ensure the effective collection of the machine vision raw data and the ultrasonic raw echo data; when the environmental temperature data changes by more than a threshold value, the space registration is re-executed to guarantee the spatial accuracy of the time-space synchronized bimodal data.

4. The intelligent obstacle avoidance and gesture control method fusing machine vision and ultrasonic wave according to claim 1, characterized in that, In step 1, the machine vision raw data includes RGB images and depth images, or ordinary RGB images; the machine vision sensor selects a depth camera or a photosensitive module matched with a monocular depth estimation algorithm. 5.The intelligent obstacle avoidance and gesture control method fusing machine vision and ultrasonic wave according to claim 1, wherein, In step 1, the ultrasonic raw echo data includes distance data, echo intensity data and echo signal-to-noise ratio data; the ultrasonic sensor supports external triggering, echo signal output and temperature compensation functions. 6.The intelligent obstacle avoidance and gesture control method fusing machine vision and ultrasonic wave according to claim 1, wherein, In step 2, the distortion correction of data preprocessing is based on the intrinsic matrix and distortion coefficient of the machine vision sensor, and the intrinsic matrix and distortion coefficient are obtained through a camera calibration experiment.

7. The intelligent obstacle avoidance and gesture control method fusing machine vision and ultrasonic wave according to claim 1, characterized in that, In step 2, time synchronization is achieved through either hardware-level synchronization or software-level time synchronization: hardware-level synchronization uses a microcontroller to generate synchronization pulses to trigger sensors and record a reference timestamp; software-level time synchronization uses a precision clock protocol to generate timestamp matching data.

8. An intelligent obstacle avoidance and gesture control system fusing machine vision and ultrasonic waves, characterized in that, The intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasound, as described in any one of claims 1 to 7, comprises: The data acquisition module is used to collect raw machine vision data, raw ultrasonic echo data, and ambient temperature data. The preprocessing and spatiotemporal synchronization module is used for: Data preprocessing: The raw machine vision data is filtered, illumination compensated, and distortion corrected to obtain preprocessed vision data; the raw ultrasonic echo data is digitally filtered and the ranging value is calibrated in conjunction with ambient temperature data to obtain preprocessed ultrasonic data. Time synchronization: Align the timestamps of the preprocessed visual data and the preprocessed ultrasonic data to obtain time-synchronized dual-modal data; Spatial registration: Taking the time-synchronized bimodal data as the object, the spatial coordinate transformation parameters are optimized through a calibration algorithm to obtain the time-space synchronized bimodal data; The multimodal fusion module is used for: Obstacle avoidance fusion: Based on spatiotemporally synchronized dual-modal data, visual and ultrasonic confidence scores are calculated, and the three-dimensional perception results of obstacles are output through dynamic weighted fusion weights. The visual confidence score is calculated based on image sharpness score and the number and distribution uniformity of feature points; the ultrasonic confidence score is calculated based on echo signal-to-noise ratio and echo signal peak voltage; the dynamic weighted fusion weights are calculated using the Softmax function based on the relative difference between the two types of confidence scores. Gesture fusion: Data registration is performed on the spatiotemporally synchronized dual-modal data before fusion, visual semantic features and ultrasonic motion features are extracted and fused, and the gesture recognition result is output; the visual semantic features are extracted through a pre-trained convolutional neural network, and the ultrasonic motion features include instantaneous distance, distance change rate and acceleration; the fusion adopts an attention model or feature concatenation method. The decision output module is used to: generate obstacle avoidance action commands based on the 3D perception results of obstacles; and generate device control commands based on the gesture recognition results. The instruction execution module is used to execute obstacle avoidance commands and equipment control commands; The environmental feedback module is used to collect environmental feedback data after the command is executed, and send it back as a supplement to the new round of machine vision raw data and ultrasonic raw echo data.

Images