Enhanced multi-mode robot visual perception system

Abstract

The invention relates to an enhanced multi-mode robot visual perception system, and aims to remarkably improve the adaptability and perception capability of a robot to a complex environment by integrating various sensors and an advanced data fusion algorithm. The system comprises a high-resolution camera, a multi-line laser radar, an infrared sensor, a sonar, a touch sensor and the like so as to realize comprehensive perception of the environment. An innovative data fusion algorithm can process and integrate data from different sensors in real time, and a consistent and accurate environment model is provided. The system is also integrated with a machine learning module, so that the robot can continuously learn and adapt to new environment characteristics and changes through a deep learning algorithm. A real-time feedback and control mechanism ensures that the robot can quickly adjust behaviors and path planning according to environment changes. Through the design of a user interaction interface, the system state is monitored in real time, and intervention is carried out when necessary. The system provided by the invention adopts a modular design, so that the sensor configuration can be quickly adjusted and upgraded according to different application scenes.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot technology, and specifically relates to a multimodal robot visual perception system that integrates multiple sensors and data processing technologies. Background Technology

[0002] In the field of intelligent robotics, environmental perception is crucial for enabling autonomous operation and intelligent decision-making in robots. Traditional robot vision systems typically rely on a single type of sensor, such as a camera or LiDAR, to acquire environmental information. However, single sensors have limitations; for example, cameras perform poorly in low-light conditions, while LiDAR, although providing accurate distance measurements, is inadequate in color and texture recognition. These limitations restrict the adaptability and reliability of robots in varying environments.

[0003] With technological advancements, multimodal sensing systems have emerged, integrating various types of sensors, such as visual sensors, distance sensors, and thermal imaging sensors, to obtain more comprehensive environmental information. Nevertheless, existing multimodal systems still face challenges in data fusion, real-time processing, and intelligent decision-making. The synchronization, fusion, and processing of sensor data require complex algorithms, and effectively integrating this data to achieve accurate environmental understanding and rapid response remains a technical challenge.

[0004] Existing systems also have shortcomings in machine learning and adaptive capabilities. Robots need to be able to continuously learn and adapt to new environmental characteristics and changes to improve their long-term operational stability and efficiency. However, most existing systems lack sufficient intelligence to achieve autonomous learning and optimization. Therefore, developing a multimodal robot visual perception system that can integrate data from multiple sensors, possess efficient data processing capabilities, and continuously self-optimize through machine learning is of great significance for promoting the development of intelligent robot technology. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal robot visual perception system. This system, by integrating multiple advanced sensors and employing innovative data fusion and processing technologies, significantly improves the robot's perception, adaptability, and intelligent decision-making capabilities in complex environments. The following are the detailed contents of the invention:

[0006] This invention's multimodal robot visual perception system achieves comprehensive and in-depth environmental perception through a carefully designed sensor integration strategy. The system first uses a high-resolution color camera to capture color images of the environment, performing object recognition tasks. Simultaneously, a high-frame-rate monochrome camera is dedicated to capturing images of fast-moving objects, ensuring clear capture of dynamic targets. A multi-line LiDAR system operates independently, generating a high-precision 3D map of the surrounding environment, providing fundamental data for spatial cognition. Building upon this, an infrared thermal imaging sensor can identify objects in dark environments, providing temperature distribution information; a short-range infrared proximity sensor is responsible for detecting nearby obstacles, ensuring the robot can react promptly when approaching obstacles. A sonar sensor performs obstacle detection and distance measurement underwater or in the air, providing necessary environmental information for underwater or aerial robots.

[0007] Furthermore, pressure sensors detect the force exerted when the robot comes into contact with its environment, while a tactile sensor array provides fine tactile feedback and analysis of object surface characteristics, supporting the robot's physical interactions. Ambient light sensors monitor ambient light intensity, atmospheric pressure sensors detect changes in ambient pressure, and humidity sensors monitor ambient humidity; these sensors work together to ensure the robot can adapt to different environmental conditions. Magnetic field sensors detect the strength of the ambient magnetic field, providing the robot with auxiliary navigation and positioning information, ensuring precise movement in complex environments. All the data collected by these sensors is synchronized and integrated through a data fusion unit, employing advanced data fusion algorithms to generate a unified environmental perception model, providing comprehensive, accurate, and real-time environmental information for the robot's autonomous operation and intelligent decision-making.

[0008] Data fusion algorithms transform multi-source sensor data into a comprehensive environmental perception model through a series of precise computational steps. The algorithm first performs data preprocessing, using a filter such as a Gaussian filter G... σ To remove noise from sensor data, the formula is expressed as I smooth =G o *I, where I is the original image, I smooth This is the image after wave saturation. Next, the feature extraction step uses algorithms such as SIFT or ORB to identify and describe key points in the image, providing a benchmark time difference for subsequent data synchronization and fusion, ensuring the data focus remains consistent at the same reference time. The consistency verification step compares the correlation between the data from different sensors, such as using correlation coefficients.

[0009] ρ is used to evaluate the consistency between data, and the formula is: Among them I sensor1 and I sensor2 These are the data points from the two sensors, where μ is the mean and σ is the standard deviation.

[0010] The decision-level fusion step employs a weighted fusion algorithm F, assigning different weights w based on the importance and reliability of each sensor's data. The formula is F = w1·I sensor1 +w2·I sensor2 +...+w n ·I sensorN , where w n It is the weight of the nth sensor, I sensorN This is data from the Nth sensor. The weight w is dynamically adjusted based on sensor accuracy, environmental conditions, and data quality to generate the most accurate environmental perception model.

[0011] The machine learning module employs deep learning algorithms, particularly Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), to process and analyze the massive amounts of data obtained from sensor ensembles. CNNs enable the module to extract complex features from visual images for object recognition and scene understanding; while RNNs utilize time-series data to predict the robot's behavioral patterns in dynamic environments. First, the data preprocessing unit standardizes and normalizes the sensor data to adapt it to the input requirements of the deep learning model. The feature extraction unit uses CNNs to extract features from image and sensor data, forming feature vectors. These feature vectors are fed into a trained deep learning model for learning and optimizing behavioral patterns. Model training employs supervised or unsupervised learning, continuously adjusting network weights and biases based on environmental feedback and user input to improve prediction accuracy.

[0012] The module also includes an online learning component that uses an incremental learning algorithm, allowing the model to be updated in real time as new data arrives, without needing to retrain from scratch. This online learning mechanism updates parameters using mini-batch gradient descent, as expressed in the formula:

[0013]

[0014] Among them, w t This represents the model parameters at time step t, where η is the learning rate. The loss function J is the parameter w t The ladder was destroyed, w t+1 These are the updated parameters. The machine learning module can automatically adjust the robot's behavior strategy based on real-time data and historical experience, enabling it to better adapt to environmental changes and user needs, and achieve intelligent autonomous operation and decision-making.

[0015] The real-time feedback and control mechanism is a highly efficient and dynamic decision support system that ensures the robot can quickly adapt to environmental changes and user commands. The mechanism's workflow begins by receiving real-time data streams from the sensor integration unit, including multi-dimensional sensor readings from vision, distance, temperature, pressure, ambient light, atmospheric pressure, humidity, and magnetic fields. This data is preprocessed and feature extracted using a data fusion algorithm to generate a comprehensive environmental perception model. The real-time analysis unit performs in-depth analysis of the fused data, utilizing advanced statistical methods and machine learning prediction models to assess the environmental state and predict potential changes. Based on these analysis results, the behavior decision unit rapidly generates corresponding behavioral commands that consider the robot's safety, efficiency, and mission objectives. For example, if the analysis unit detects a potential collision risk, the decision unit will immediately trigger obstacle avoidance behavior. The path planning unit dynamically adjusts the robot's path based on the behavioral commands and the current environmental perception model. The path planning algorithms used here, such as the artificial potential field method or the Rapid Random Tree (RRT) algorithm, can quickly calculate an optimal path that avoids obstacles and reaches the target point. The path planning results are transmitted in real-time to the robot's actuators, such as motors and servos, through the control interface to adjust the robot's motion state. The mechanism also includes a user interface that allows operators to intervene in the decision-making process when necessary, guiding the robot's behavior through manual input or parameter adjustments. The entire real-time feedback and control mechanism is designed to ensure the robot's flexibility and responsiveness in complex environments, improving task execution success rates and the robot's autonomy.

[0016] The user interface uses graphical display technology to intuitively present the robot's real-time status and environmental information in the form of charts, maps, and indicators, allowing operators to easily monitor the robot's position, speed, and behavior. The interface includes control options, allowing operators to set desired speeds via numeric input boxes or set target points on the map, while also providing safety features such as an emergency stop button. Operators can also receive immediate information on obstacle avoidance triggers and path adjustments through a real-time feedback system, ensuring precise control over the robot's behavior. In emergencies, operators can manually guide the robot using joysticks or keyboard commands, while a log recording function records all user operations and system responses for subsequent analysis and optimization. The user interface is designed with intuitiveness and ease of use in mind, ensuring operators can quickly understand the robot's status and intervene effectively when necessary.

[0017] The visual perception system employs an innovative modular design concept, allowing for flexible adjustment of sensor configurations to suit different application scenarios and needs. Each sensor module, as an independent functional unit, has its own data processing and power management modules, enabling easy integration or removal without affecting other components. For example, if the robot needs to operate in low-light environments, an infrared thermal imaging sensor module can be quickly added or upgraded without interfering with the normal operation of other sensors. The modular design also allows for plug-and-play functionality between different sensor modules through software-defined interfaces. This means that when new sensor technologies or improved models emerge, older modules can be easily replaced without redesigning the entire system. For instance, if a new camera module with higher resolution or a wider field of view is introduced, performance improvements can be achieved simply by connecting it to the system and configuring its parameters via software. The modular design is also reflected in the system's software architecture, where each sensor module has corresponding drivers and data processing algorithms. These software components are designed as reusable modules that can be combined or replaced as needed. This design not only improves the system's flexibility and scalability but also reduces maintenance and upgrade costs, as individual modules can be updated independently without requiring large-scale modifications to the entire system. The modular design of this invention provides a scalable, upgradeable, and easy-to-maintain system architecture, enabling robots to quickly adapt to changing working conditions while maintaining technological sophistication and competitiveness.

[0018] The multimodal robot vision perception system of this invention includes a comprehensive data processing method that ensures the robot can intelligently understand and respond to its environment with precise and efficient steps. The initial steps of the method involve receiving data streams from all sensor modules within the system, including but not limited to image pixel values, distance measurements, temperature readings, pressure, and humidity levels. All these data streams are first synchronized to a common time reference to ensure data timeliness and consistency, with synchronization accuracy down to the microsecond level.

[0019] Next, the synchronized data enters the preprocessing stage, where advanced signal processing techniques are employed, such as Gaussian filter denoising, and a standard deviation σ = 1.5 is set to smooth the image data and reduce random noise. The normalization process scales the data to the [0, 1] interval so that data from different sensors can be compared and processed at the same scale. The feature extraction step utilizes algorithms such as SIFT or HOG to extract key points and descriptors from the image, providing rich environmental features for subsequent data fusion.

[0020] During the data fusion phase, a weighted average fusion algorithm is employed, based on the reliability and accuracy of each sensor. Different weights w can be assigned to high-reliability LiDAR data.LiDAR =0.6, while for relatively low-precision infrared sensor data, a weight w is assigned. 1R =0.2. The fusion algorithm integrates these weighted data to generate a comprehensive environmental perception model, which uses high precision... And rich details reflect the actual environment in which the robot is located.

[0021] A machine learning module intervenes, using trained neural networks to perform in-depth analysis of the fusion model, identifying environmental features and predicting behavioral patterns. Convolutional neural networks (CNNs) may be used to identify obstacles and paths from visual data, while loop neural networks (RNNs) may be used to predict dynamic changes in robot movement.

[0022] The analysis results are then fed into a real-time feedback and control mechanism, which dynamically adjusts the robot's behavior parameters and path planning based on the analysis results and possible user inputs (such as target position or speed settings). If a potential collision is predicted, the control mechanism will calculate a new obstacle avoidance path and adjust the robot's speed and steering parameters, such as reducing the speed from V = 1.2 m / s to V′ = 0.5 m / s, and changing the path to avoid the obstacle.

[0023] Ultimately, this adjusted behavior and path planning information is fed back to the robot's actuators, such as motor controllers, in numerical form, enabling precise speed control and direction adjustment. For example, the instructions received by the motors might include decelerating with an undetermined acceleration a = 0.3 m / s² and turning at a turning angle θ = 45° to execute the new path planning.

[0024] The design of the entire data processing method ensures the robot's autonomy and adaptability in complex environments, improving the accuracy and efficiency of task execution.

[0025] The technical solution of this invention can bring about the following significant technical effects:

[0026] 1. Enhanced Environmental Perception: By integrating multiple sensors, including a high-resolution color camera, a high-frame-rate monochrome camera, a multi-line LiDAR, and an infrared thermal imaging sensor, the system of this invention provides comprehensive environmental perception. The combined use of color and monochrome cameras enables the system to recognize the color and texture of objects, as well as capture images of fast-moving objects. The high-precision 3D map generated by the LiDAR, combined with infrared thermal imaging and a short-range infrared proximity sensor, allows the robot to accurately identify and avoid obstacles under various lighting conditions. This fusion of multi-sensor data significantly improves the robot's perception accuracy and robustness in complex environments.

[0027] 2. Improved Data Processing and Decision-Making Speed: The data fusion algorithm of this invention employs an efficient data processing workflow, including data preprocessing, feature extraction, data synchronization, and consistency verification, ensuring the real-time performance and accuracy of the data. Furthermore, the introduction of real-time feedback and control mechanisms enables the system to quickly respond to environmental changes and dynamically adjust behavioral decisions and path planning. This rapid data processing and decision-making capability is crucial for the robot's rapid response in emergency situations, significantly improving the robot's operational efficiency and safety.

[0028] 3. Intelligent Learning and Adaptive Capabilities: The machine learning module of this invention employs deep learning algorithms, enabling it to continuously learn and adapt to new environmental features and user behaviors. Over time, the robot automatically updates and optimizes its behavioral patterns through interaction with the environment to better adapt to changes. This adaptive learning capability allows the robot not only to handle pre-set tasks but also to cope with unknown or dynamically changing environmental conditions, significantly improving the robot's intelligence level and application scope. Attached Figure Description

[0029] Figure 1 This is a simplified diagram of the multimodal robot vision perception system module of the present invention.

[0030] Figure 2 This is a diagram illustrating the data processing steps of the multimodal robot vision perception system of the present invention. Detailed Implementation

[0031] The following will refer to the appendices in the embodiments of the present invention. Figure 1 and Figure 2 The technical solutions in the embodiments of the present invention are clearly and completely described herein. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0032] A multimodal robot visual perception system, the details of which are as follows:

[0033] This invention's multimodal robot visual perception system achieves comprehensive and in-depth environmental perception through a carefully designed sensor integration strategy. The system first uses a high-resolution color camera to capture color images of the environment, performing object recognition tasks. Simultaneously, a high-frame-rate monochrome camera is dedicated to capturing images of fast-moving objects, ensuring clear capture of dynamic targets. A multi-line LiDAR system operates independently, generating a high-precision 3D map of the surrounding environment, providing fundamental data for spatial cognition. Building upon this, an infrared thermal imaging sensor can identify objects in dark environments, providing temperature distribution information; a short-range infrared proximity sensor is responsible for detecting nearby obstacles, ensuring the robot can react promptly when approaching obstacles. A sonar sensor performs obstacle detection and distance measurement underwater or in the air, providing necessary environmental information for underwater or aerial robots.

[0034] Furthermore, pressure sensors detect the force exerted when the robot comes into contact with its environment, while a tactile sensor array provides fine tactile feedback and analysis of object surface characteristics, supporting the robot's physical interactions. Ambient light sensors monitor ambient light intensity, atmospheric pressure sensors detect changes in ambient pressure, and humidity sensors monitor ambient humidity; these sensors work together to ensure the robot can adapt to different environmental conditions. Magnetic field sensors detect the strength of the ambient magnetic field, providing the robot with auxiliary navigation and positioning information, ensuring precise movement in complex environments. All the data collected by these sensors is synchronized and integrated through a data fusion unit, employing advanced data fusion algorithms to generate a unified environmental perception model, providing comprehensive, accurate, and real-time environmental information for the robot's autonomous operation and intelligent decision-making.

[0035] Data fusion algorithms transform multi-source sensor data into a comprehensive environmental perception model through a series of precise computational steps. The algorithm first performs data preprocessing, using a filter such as a Gaussian filter G... σ To remove noise from sensor data, the formula is expressed as I smooth =G σ *I, where I is the original image, I smooth This is the image after wave saturation. Next, the feature extraction step uses algorithms such as SIFT or ORB to identify and describe key points in the image, providing a benchmark for subsequent data synchronization and fusion to ensure the consistency of data focus at the same reference time. The consistency verification step compares the correlation between data from different sensors, such as using the correlation coefficient ρ to evaluate the consistency between data, as shown in the formula... Among them I sensor1 and I sensor2 These are the data points from the two sensors, where μ is the mean and σ is the standard deviation.

[0036] The decision-level fusion step employs a weighted fusion algorithm F, assigning different weights w based on the importance and reliability of each sensor's data. The formula is F = w1·I sensor1 +w2·I sensor2 +...+w n ·I sensorN , where w n It is the weight of the nth sensor, I sensorN This is data from the Nth sensor. The weight w is dynamically adjusted based on sensor accuracy, environmental conditions, and data quality to generate the most accurate environmental perception model.

[0037] The machine learning module employs deep learning algorithms, particularly Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), to process and analyze the massive amounts of data obtained from sensor ensembles. CNNs enable the module to extract complex features from visual images for object recognition and scene understanding; while RNNs utilize time-series data to predict the robot's behavioral patterns in dynamic environments. First, the data preprocessing unit standardizes and normalizes the sensor data to adapt it to the input requirements of the deep learning model. The feature extraction unit uses CNNs to extract features from image and sensor data, forming feature vectors. These feature vectors are fed into a trained deep learning model for learning and optimizing behavioral patterns. Model training employs supervised or unsupervised learning, continuously adjusting network weights and biases based on environmental feedback and user input to improve prediction accuracy.

[0038] The module also includes an online learning component that uses an incremental learning algorithm, allowing the model to be updated in real time as new data arrives, without needing to retrain from scratch. This online learning mechanism updates parameters using mini-batch gradient descent, as expressed in the formula:

[0039]

[0040] Among them, w t This represents the model parameters at time step t, where η is the learning rate.

[0041] The loss function J is the parameter w t The ladder was destroyed, w t+1 These are the updated parameters. The machine learning module can automatically adjust the robot's behavior strategy based on real-time data and historical experience, enabling it to better adapt to environmental changes and user needs, and achieve intelligent autonomous operation and decision-making.

[0042] The real-time feedback and control mechanism of the multimodal robot's visual perception system precisely controls the robot's behavior and path planning through a series of numerical parameters. Real-time environmental data collected by the sensor integration unit, such as light intensity set at 300 lux, ambient temperature at 25 degrees Celsius, atmospheric pressure at 101.3 kPa, relative humidity at 50%, and magnetic field strength at 50 microtesla, is normalized by the data fusion unit to generate standardized feature vectors with values ​​ranging from 0 to 1. Based on these feature vectors and a preset risk assessment threshold of 0.8, the real-time analysis unit evaluates the environmental state using a machine learning model. Once the risk value exceeds the threshold, the behavior decision unit immediately initiates obstacle avoidance behavior, setting the obstacle avoidance distance to 2 meters and the obstacle avoidance speed to 1 m / s. The path planning unit calculates a new safe path based on the obstacle avoidance command and the environmental perception model, considering an obstacle avoidance radius of 1.5 meters, a target direction of 45 degrees, and a expected speed of 0.5 m / s. The path is then converted into specific motion commands, setting the speed to 0.5 m / s and the turning angle to 30 degrees. In addition, users can manually input the target position coordinates (10,5) meters and the desired speed of 1 meter per second through the interactive interface to directly intervene in the robot's control. This entire mechanism, through real-time adjustment of these numerical parameters, ensures that the robot can flexibly and accurately adapt to environmental changes and user commands, improving its autonomy and task execution efficiency.

[0043] The user interface, through numerical parameters and graphical display technology, provides an intuitive and efficient operating platform. Operators can use the simple controls on the interface to set the robot's desired speed in real time, inputting V... set = 1 m / s, and by clicking the map control on the interface, you can input the target coordinates, such as (x goal y goal ) = (10, 5) meters, directly specifying the distance D from the target point. tooal This enables operators to accurately grasp the robot's dynamics.

[0044] The operator can also adjust the sensor's sensitivity threshold by inputting values ​​or using a slider on the interface, such as setting the threshold of the infrared sensor to T. infrared =0.05 degrees Celsius to optimize the accuracy of environmental perception. In an emergency, the operator can immediately reduce the robot's speed to V using the one-button emergency stop function. stop =0 meters per second, ensuring operational safety. All user operations and system responses are recorded by the log system, including operation timestamps, operation types, and numerical changes in robot responses, such as from V... current To V new The transformation provides detailed data support for system analysis and optimization.

[0045] The multimodal robot vision perception system employs a modular design, providing high flexibility and scalability. This design allows operators to quickly adjust or upgrade sensor modules by configuring numerical parameters according to specific application scenarios, such as industrial inspection or outdoor exploration. For example, to enhance visual recognition capabilities in low-light environments, a high-resolution infrared thermal imaging sensor module with an ultra-low illumination sensitivity of 0.01 lux can be inserted, and its scanning frequency can be set to 10 Hz via software to update temperature images 10 times per second. To improve spatial perception capabilities, a multi-line LiDAR module with a 360-degree field of view and 0.1-degree angular resolution can be installed, configured to emit 100,000 pulses per second to obtain high-density 3D environmental data. Each sensor module has independent data processing capabilities and is connected to the system's central processing unit via a standardized communication interface. This processing unit dynamically adjusts the parameters of each module based on real-time analysis of the sensor data. For example, if the ambient light sensor detects a drop in light intensity to 5 lux, the system automatically reduces the exposure time of the color camera to 1 / 30 second and increases the gain of the infrared sensor to 2.5 times to ensure image clarity. The modular design also allows for software updates to optimize sensor performance, such as improving the signal processing algorithm of the sonar sensor through firmware upgrades to reduce false alarm rates, or optimizing the tactile sensor array to improve sensitivity to 0.1 Newtons for more accurate perception of object surface characteristics.

[0046] The multimodal robot vision perception system of this invention includes a comprehensive data processing method that ensures the robot can intelligently understand and respond to its environment with precise and efficient steps. The initial steps of the method involve receiving data streams from all sensor modules within the system, including but not limited to image pixel values, distance measurements, temperature readings, pressure, and humidity levels. All these data streams are first synchronized to a common time reference to ensure data timeliness and consistency, with synchronization accuracy down to the microsecond level.

[0047] Next, the synchronized data enters the preprocessing stage, where advanced signal processing techniques are employed, such as Gaussian filter denoising, and a standard deviation σ = 1.5 is set to smooth the image data and reduce random noise. The normalization process scales the data to the [0, 1] interval so that data from different sensors can be compared and processed at the same scale. The feature extraction step utilizes algorithms such as SIFT or HOG to extract key points and descriptors from the image, providing rich environmental features for subsequent data fusion.

[0048] During the data fusion phase, a weighted average fusion algorithm is employed, based on the reliability and accuracy of each sensor. Different weights w can be assigned to high-reliability LiDAR data. LiDAR=0.6, while for relatively low-precision infrared sensor data, a weight w is assigned. 1R =0.2. The fusion algorithm integrates these weighted data to generate a comprehensive environmental perception model, which uses high precision... And rich details reflect the actual environment in which the robot is located.

[0049] A machine learning module intervenes, using trained neural networks to perform in-depth analysis of the fusion model, identifying environmental features and predicting behavioral patterns. Convolutional neural networks (CNNs) may be used to identify obstacles and paths from visual data, while loop neural networks (RNNs) may be used to predict dynamic changes in robot movement.

[0050] The analysis results are then fed into a real-time feedback and control mechanism, which dynamically adjusts the robot's behavior parameters and path planning based on the analysis results and possible user inputs (such as target position or speed settings). If a potential collision is predicted, the control mechanism will calculate a new obstacle avoidance path and adjust the robot's speed and steering parameters, such as reducing the speed from V = 1.2 m / s to V′ = 0.5 m / s, and changing the path to avoid the obstacle.

[0051] Ultimately, this adjusted behavior and path planning information is fed back to the robot's actuators, such as motor controllers, in numerical form, enabling precise speed control and direction adjustment. For example, the instructions received by the motors might include accelerating at an undetermined rate a = 0.3 m / s². 2 Decelerate and turn at an angle θ = 45° to execute a new path plan.

[0052] The design of the entire data processing method ensures the robot's autonomy and adaptability in complex environments, improving the accuracy and efficiency of task execution.

[0053] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal robot visual perception system, characterized in that, include: A high-resolution color camera is used to capture color images of the environment and perform object recognition; A high frame rate monochrome camera is used to capture images of fast-moving objects; a multi-line lidar (LiDAR) is used to generate high-precision 3D maps of the surrounding environment. Infrared thermal imaging sensors are used to detect the temperature distribution of objects and identify objects in dark environments; short-range infrared proximity sensors are used for short-range obstacle detection; sonar sensors are used for underwater or air obstacle detection and distance measurement; pressure sensors are used to detect the contact force between the robot and the environment. A tactile sensor array is used to provide precise tactile feedback and analysis of object surface characteristics; A data fusion unit is used to integrate the data from the sensors and provide a unified environmental perception model. An ambient light sensor is used to detect the intensity of ambient light to adapt to different lighting conditions; Atmospheric pressure sensors are used to detect changes in ambient pressure to adapt to different climatic conditions; A humidity sensor detects ambient humidity to adapt to different humidity conditions; a magnetic field sensor detects the intensity of the ambient magnetic field to aid navigation and positioning. These sensor modules are tightly connected through a precisely designed hardware interface and communication protocol, forming a highly integrated sensing network. A high-resolution color camera and a high-frame-rate monochrome camera work together to capture static and dynamic environmental images, providing data support for object recognition and motion capture. A multi-line LiDAR independently generates a high-precision 3D map, which, combined with the infrared thermal imaging sensor and the short-range infrared proximity sensor, enables accurate detection of object temperature distribution and nearby obstacles. A sonar sensor is responsible for obstacle detection and distance measurement in water or air, while a pressure sensor and a tactile sensor array provide detailed information on physical contact and surface characteristics. Data collected by the sensors is integrated by a data fusion unit, using advanced algorithms to process and fuse the synchronized data, generating a unified environmental perception model. Auxiliary sensors such as ambient light sensors, atmospheric pressure sensors, humidity sensors, and magnetic field sensors monitor environmental parameters, ensuring the robot can adapt to different lighting, climate, and humidity conditions, and utilize magnetic field information to assist navigation and positioning. Through efficient data processing and intelligent algorithms, the entire system achieves comprehensive perception and rapid response to complex environments, providing strong support for the robot's autonomous operation and intelligent decision-making.

2. The system according to claim 1, characterized in that, The data fusion unit employs an innovative multi-source data fusion algorithm that can process and integrate data from different sensors in real time to improve the accuracy and robustness of environmental perception.

3. The system according to claim 2, characterized in that, The data fusion algorithm includes data preprocessing, feature extraction, data synchronization, consistency verification, and decision-level fusion steps.

4. The system according to claim 1 or 2, characterized in that, It further includes at least one machine learning module for optimizing the robot's behavior patterns and environmental adaptability through deep learning algorithms.

5. The system according to claim 4, characterized in that, The machine learning module can automatically update and optimize algorithms to adapt to constantly changing environmental conditions and user behavior.

6. The system according to any one of claims 1 to 5, characterized in that, It also includes a real-time feedback and control mechanism to quickly adjust the robot's behavior and path planning based on environmental changes and user input.

7. The system according to claim 6, characterized in that, The real-time feedback and control mechanism includes real-time analysis of sensor data, rapid response of behavioral decisions, and dynamic adjustment of path planning.

8. The system according to any one of claims 1 to 7, characterized in that, It also includes a user interface that allows operators to monitor the system status in real time and intervene when necessary.

9. The system according to claim 8, characterized in that, The user interface provides graphical displays, control options, and real-time feedback on system status to facilitate operator understanding and operation.

10. A data processing method for a multimodal robot visual perception system, characterized in that, Includes the following steps: Receive and synchronize data from image sensors, LiDAR, infrared sensors, sonar sensors, pressure sensors, tactile sensors, ambient light sensors, atmospheric pressure sensors, humidity sensors, and magnetic field sensors; The synchronized data is preprocessed, including denoising, normalization, and feature extraction. Data fusion algorithms are used to integrate preprocessed data and generate a unified environmental perception model. The machine learning module is used to further analyze the fusion model to identify environmental features and behavioral patterns; Based on the analysis results and user input, the robot's behavior and path planning are adjusted through real-time feedback and control mechanisms; The adjusted behavior and path planning information is fed back to the robot's actuator to achieve precise control.

Images