Multi-sensor fusion identification method for wheel-foot humanoid robot in industrial scene

Abstract

The invention discloses a wheel-foot humanoid robot multi-sensor fusion identification method in an industrial scene, particularly relates to the technical field of robots, and realizes time alignment of sensor units by constructing a multi-modal sensing system and establishing a synchronous triggering mechanism. Multi-source data is unified to a basic coordinate system by adopting multi-element external parameter calibration and real-time motion compensation, then data-level, feature-level and decision-level fusion is carried out based on a hierarchical fusion architecture, the fusion weight is dynamically adjusted according to the moving state of the robot and scene features, and finally a navigation path plan, an operation instruction and safety early warning are output in real time. The problems of data time delay and coordinate system deviation are solved through a high-precision time synchronization and motion compensation mechanism, the recognition precision and stability in a complex scene are improved by using a hierarchical fusion and weight adaptive strategy, and challenges such as illumination variation, dust interference and machine body shaking are effectively overcome. And the efficient sensing and intelligent decision-making capability of the wheel-foot humanoid robot in a dynamic industrial task is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot technology, more particularly, the present application relates to a wheel-legged robot multi-sensor fusion recognition method in an industrial scene. BACKGROUND

[0002] Industrial 4.0 era puts forward higher requirements for robot intelligence, wheel-legged robot combines the efficiency of wheeled movement and the strong adaptability of legged environment, and becomes an ideal platform for industrial automation. At present, industrial mobile robots mainly rely on single or limited types of sensors for environment perception and navigation, which has significant limitations in complex industrial application scenarios. Typical industrial environments usually contain challenging factors such as dynamic light changes, complex terrain structures, moving obstacles, and high-intensity electromagnetic interference, which put high requirements on the perception system of the robot.

[0003] Therefore, it can be found that the existing wheel-legged robot has the following defects in the industrial scene: there are adverse factors such as light changes, dust, electromagnetic interference in the industrial scene, which seriously affect the performance of the sensor; the vibration and posture change generated during the movement interfere with the sensor data acquisition; at the same time, industrial tasks such as precision assembly and equipment inspection require millimeter-level positioning accuracy, and single sensor is difficult to meet the requirements.

[0004] Therefore, in view of the above problems, a wheel-legged robot multi-sensor fusion recognition method in an industrial scene is proposed. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, the present application provides a wheel-legged robot multi-sensor fusion recognition method in an industrial scene to solve the problems raised in the background art.

[0006] To achieve the above purpose, the present application provides the following technical scheme: a wheel-legged robot multi-sensor fusion recognition method in an industrial scene, comprising the following steps: S1, constructing a sensor system: configuring a multi-modal sensing system composed of sensor units on the wheel-legged robot, establishing a synchronous triggering mechanism through the multi-modal sensing system to align the time of the sensor units; S2, data preprocessing and alignment: the original data collected by the multi-modal sensing system is preprocessed by image processing technology, the preprocessing includes two-stage optimization of joint bilateral filtering and Kalman filtering on depth image, and the multi-source data after preprocessing is unified to the base coordinate system of the wheel-legged robot; S3, dynamic fusion recognition: a hierarchical fusion architecture is constructed, which adjusts the fusion weight of the sensor unit according to the current wheeled or gait movement state of the wheeled biped robot and the characteristics of the industrial scene, realizes the recognition and positioning of environmental objects and obstacles; S4, adaptive output: according to the results of the recognition and positioning, the navigation path planning, operation instruction and safety warning of the wheeled biped robot are output in real time.

[0007] Preferably, in step S1, the sensor unit includes a vision sensor, a depth camera, a laser radar, an inertial measurement unit and a torque sensor, and the sensor unit is respectively installed on the trunk, head, end effector and joint part of the wheeled biped robot, the time alignment generates a unified time reference signal by using a master-slave synchronization architecture with high precision IMU as the master clock, the vision sensor includes a global camera for large range scene monitoring and an eye-hand camera for end fine operation guidance, and the depth camera preferably adopts time of flight technology.

[0008] Preferably, in step S2, a multi-element external parameter calibration method is used for space calibration of multiple sensors, and IMU data is used for real-time motion compensation of visual perception data caused by the motion of the wheeled biped robot itself.

[0009] Preferably, the steps of the multi-element external parameter calibration method are as follows: A1, mapping the initial detection information corresponding to the sensor unit to the preset coordinate system corresponding to the wheeled biped robot, to obtain the conversion detection information corresponding to the data of the sensor unit; A2, based on the spatial layout of the sensor unit, multiple sensor units are divided into at least one sensor group; A3, based on the conversion detection information corresponding to each sensor unit contained in the sensor group, cross-modal matching information of a predetermined type of object is constructed; A4, based on the cross-modal matching information corresponding to the sensor group, the external parameters of multiple sensor units are calibrated to obtain the relative position relationship between the sensor units.

[0010] Preferably, in step S3, the hierarchical fusion architecture is specifically as follows: Data level fusion: directly integrating the texture information of RGB image and the geometric information of depth camera to generate enhanced environment representation; Feature level fusion: extracting point cloud features, image features and inertial features from the data of the sensor unit, and performing weighted fusion through attention mechanism; Decision level fusion: using D-S evidence theory to comprehensively decide the independent recognition results of the sensor unit.

[0011] Preferably, in step S3, the adjustment of the fusion weight is adaptively completed by a pre-trained weight level prediction network based on the data quality evaluated by the sensor unit in real time and the illumination and dust interference factors of the environment.

[0012] Preferably, it further includes an anti-interference mechanism for the motion characteristics of the wheel-legged robot: when the wheel-legged robot is in gait movement, the filtering strength of the IMU data is increased to compensate for body jitter; when the wheel-legged robot is in wheeled movement, the real-time performance of front obstacle detection is preferentially ensured.

[0013] Preferably, the depth camera is installed at the joint of the wheel-legged robot, and the local perspective information obtained with the joint movement is fused with the global perspective information obtained by the sensor unit of the trunk or head of the wheel-legged robot.

[0014] Preferably, in step S4, the safety warning is automatically triggered when a potential collision risk or failure of the sensor unit is identified, and the data of the remaining effective sensor units is degraded to run, so that the basic function of the sensor unit is not affected.

[0015] Preferably, the industrial scene includes an automated assembly line, an intelligent warehouse area, and a logistics sorting station, and the identified objects include workpieces, equipment, personnel, and other mobile robots.

[0016] The technical effects and advantages of the present application are as follows: 1. Compared with the prior art, the wheel-legged robot multi-sensor fusion identification method in the industrial scene solves the problems of data time delay and coordinate system deviation through a high-precision time synchronization and motion compensation mechanism. High-precision IMU is used as the master clock to realize time alignment of the sensor units, and multi-element external parameter calibration and real-time motion compensation are combined to unify multi-source data to the basic coordinate system, so that data space consistency can be maintained during robot movement, and perception distortion caused by body jitter or movement is overcome.

[0017] 2. Compared with the prior art, the wheel-legged robot multi-sensor fusion identification method in the industrial scene improves the identification accuracy and stability in complex scenes through a hierarchical fusion architecture and a weight self-adaptive mechanism. The data level, feature level, and decision level are fused in layers, and the sensor fusion weight is dynamically adjusted based on the robot movement state and environmental interference factors, so that optimal perception performance can be maintained under different task requirements.

[0018] 3. Compared with existing technologies, the multi-sensor fusion recognition method for wheeled humanoid robots in this industrial scenario enhances the system's fault tolerance and scene adaptability through the fusion of local and global perspectives, anti-interference mechanisms, and degraded operation strategies. By coordinating depth cameras installed at the joints with torso / head sensors, it achieves complementary local fine perception and global monitoring. At the same time, when sensors fail or collision risks occur, the system can automatically trigger safety warnings and degrade operation based on effective sensors to ensure that basic functions are not affected. Thus, it can reliably identify and interact with workpieces, equipment, personnel, and other robots in typical industrial scenarios such as automated assembly and intelligent warehousing. Attached Figure Description

[0019] Fig. 1 This is a flowchart of the method of the present invention.

[0020] Fig. 2 This is a flowchart of the multivariate extrinsic parameter calibration method of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1 As attached Figs. 1-2 The multi-sensor fusion recognition method for wheeled humanoid robots in the industrial scenario shown includes the following steps: S1. Construct a sensor system: Configure a multimodal sensing system composed of sensor units on the wheel-legged humanoid robot, and establish a synchronous triggering mechanism through the multimodal sensing system to align the time of the sensor units; S2. Data preprocessing and alignment: The raw data collected by the multimodal sensing system is preprocessed using image processing technology. The preprocessing includes a two-stage optimization of the depth image using joint bilateral filtering and Kalman filtering, and the preprocessed multi-source data is unified to the basic coordinate system of the wheeled humanoid robot. S3. Dynamic Fusion Recognition: Construct a hierarchical fusion architecture. The hierarchical fusion architecture adjusts the fusion weights of the sensor units according to the current wheeled or gait movement state of the wheeled humanoid robot and the characteristics of the industrial scene to achieve the recognition and localization of environmental objects and obstacles. S4. Adaptive Output: Based on the identification and positioning results, output the navigation path planning, operation instructions and safety warnings of the wheeled humanoid robot in real time.

[0023] Specifically, first, when constructing the sensor system, a global camera and a depth camera using time-of-flight technology are installed on the torso and head of the wheeled-legged humanoid robot, laser radars, inertial measurement units and torque sensors are configured at the joints and actuators, a unified timestamp is generated by a high-precision IMU as a master clock, and all sensor units are time-aligned at the microsecond level in a hardware-triggered manner; In the data preprocessing stage, the original data collected by the depth camera is sequentially subjected to joint bilateral filtering to maintain edge details and suppress noise, Kalman filtering is used to smooth the motion of consecutive frames, the transformation matrix obtained by multi-element external parameter calibration is used to convert the visual, point cloud and inertial data to the robot base coordinate system, and the image sequence is real-time motion compensated based on the body motion measured by the IMU, so as to eliminate the pixel shift caused by the movement of the robot; In the dynamic fusion recognition stage, a hierarchical fusion architecture is constructed, the RGB image texture and depth point cloud geometric information are directly registered to generate a color point cloud environment representation at the data level, the point cloud features, CNN image features and IMU inertial features are dynamically weighted and fused through the attention mechanism at the feature level, the weight value is automatically adjusted by the pre-trained network according to the real-time data quality such as image clarity, point cloud density, and scene interference factors such as light intensity and dust concentration, and at the decision level, the independent recognition results of each sensor are integrated by using D-S evidence theory, and the fusion strategy is optimized according to the current movement mode of the robot, i.e., the IMU filtering is enhanced to suppress body jitter during gait movement, and the detection frequency of the front obstacles of the laser radar is preferentially guaranteed during wheeled movement; Finally, the system generates a navigation path and operation instruction in real time according to the recognition and positioning results, triggers a safety warning when a sensor failure or collision risk is detected, automatically switches to a degraded operation mode based on the remaining valid sensors, and ensures uninterrupted basic perception function; through time sequence synchronization, multi-level filtering, coordinate unification and state adaptive fusion mechanism, stable perception and interaction of workpieces, equipment and moving targets in dynamic industrial environment are realized.

[0024] As a preferred embodiment, in step S1, a global camera is installed on the front of the torso and the central position of the head of the wheeled-legged humanoid robot for large-range environment monitoring, an eye-hand camera is installed on the wrist of the end effector for guiding fine operation, a depth camera preferably uses time-of-flight technology to actively project infrared light and measure the time of flight to directly obtain high anti-interference depth information, a laser radar is arranged on the head or shoulder of the robot to obtain point cloud data with a large horizontal viewing angle, and an inertial measurement unit and a torque sensor are embedded in the main joints and leg actuators of the robot respectively for sensing the body motion posture and ground contact force; All sensor units adopt a master-slave synchronization architecture with high-precision IMU as the master clock. Precise pulse synchronization signals are generated by IMU hardware and directly distributed to visual sensors, depth cameras and lidar through dedicated hardware trigger lines, forcing each sensor to start data acquisition under a unified hardware time reference, thereby achieving microsecond-level time alignment. In the visual sensor, the global camera uses a wide-angle lens to cover a large fan-shaped area in front of the robot, and the eye-hand camera is bound to the kinematic model of the robot arm, which can track the end-effector in real time through coordinate transformation. The depth camera uses time-of-flight technology and has strong robustness to light changes. Its point cloud data and the RGB image of the global camera are frame-level paired through timestamps. The above-mentioned sensor units can use corresponding models of commonly used sensor types in reality, such as the G1-D model of the wheel-legged robot of Yushu Technology. This layout makes the sensors on the torso and head provide stable global environment perception, while the sensors on the joints and the end provide dynamic local perception perspective as the robot moves, providing a spatiotemporal consistent perception basis for subsequent multi-level data fusion.

[0025] As a preferred embodiment, after completing the sensor system construction and time synchronization in step S2, multi-element extrinsic calibration is first performed. The robot is controlled to move to a plurality of preset calibration board poses, and the data of all sensor units are synchronously collected. The two-dimensional image corners detected by the visual sensor, the three-dimensional point cloud planes generated by the depth camera, and the linear features scanned by the lidar are respectively converted into converted detection information in the robot base coordinate system. Then, according to the physical layout relationship of the sensors on the robot torso, head and joints, the sensors with overlapping fields of view or functional correlation are divided into sensor groups. For example, the global camera and the depth camera on the head are set as a group, and the lidar on the torso and the depth camera at the joint are set as another group. Then, based on the converted detection information generated by each sensor in the sensor group on the same calibration board or specific marker, the spatial correspondence between the point cloud and the image features is calculated, and the cross-modal matching information is constructed. The expression of this calculation is:

[0026] wherein, represents the two-dimensional pixel coordinates on the image, which is a vector containing u and v components; represents the intrinsic matrix of the visual sensor, which is a 3x3 matrix containing focal length, principal point coordinates and other camera-specific optical parameters, which determines how three-dimensional points are projected onto the two-dimensional image plane; and denotes a component of the extrinsic parameter to be calibrated, where R is a 3x3 rotation matrix representing the pose rotation relationship from the point cloud coordinate system to the visual sensor coordinate system; t is a 3x1 translation vector representing the position offset between the origins of the two coordinate systems; denotes the spatial coordinates of the three-dimensional point cloud provided by the depth camera or the laser radar in its own coordinate system, which is a homogeneous coordinate form vector; The matching information is used to solve the accurate rotation matrix and translation vector between each two sensors by a least square iterative optimization algorithm, to complete the extrinsic parameter calibration and establish the relative position relationship between all sensor units. In actual operation, to compensate for the influence of the robot's own motion on visual perception, the system continuously reads the high-frequency output IMU data, and obtains the displacement and rotation of the robot within the image exposure time period by integration, and then uses the calibrated extrinsic parameter matrix to compensate the motion amount to the corresponding image pixel coordinates or point cloud frames in reverse, for example, for each frame of image collected by the global camera, the body motion calculated from the IMU data is used to perform affine transformation correction, so as to eliminate the image blur and positioning deviation caused by the robot movement, and to provide stable and spatially consistent multi-source data basis for subsequent dynamic fusion recognition.

[0027] As a preferred embodiment, the steps of the multi-element extrinsic parameter calibration method are: A1, mapping the initial detection information corresponding to the sensor unit to the preset coordinate system corresponding to the wheel-legged humanoid robot, to obtain the conversion detection information corresponding to the data of the sensor unit; A2, dividing a plurality of sensor units into at least one sensor group based on the spatial layout of the sensor units; A3, constructing cross-modal matching information of a predetermined type of object based on the conversion detection information corresponding to each sensor unit included in the sensor group; A4, calibrating the extrinsic parameters of the plurality of sensor units based on the cross-modal matching information corresponding to the sensor group, to obtain the relative position relationship between the sensor units.

[0028] Specifically, in step A1, the system controls the robot to be in a static calibration posture, and synchronously acquires sensor data. For a two-dimensional image captured by a visual sensor, the pixel coordinates of the corner points of the calibration board are extracted by a corner detection algorithm. For a depth camera and a laser radar, the three-dimensional point cloud coordinates of the planar feature points of the calibration board are obtained. The initial transformation matrix of each sensor unit to the robot base coordinate system is used to uniformly convert the above-mentioned corner point coordinates and point cloud coordinates to the base coordinate system, to generate conversion detection information. In step A2, logical grouping is performed according to the physical installation position of the sensor on the robot body and the functional correlation, for example, a head global camera, a depth camera and a trunk laser radar that simultaneously face the front of the robot, whose fields of view overlap, are divided into one sensor group, and an eye-hand camera that focuses on end operation and a wrist depth camera are divided into another sensor group; In step A3, cross-modal matching information is constructed based on the conversion detection information of each unit in the sensor group, the core of which is to establish the correspondence of different sensor data to the same spatial physical feature such as the corner points of the calibration board, the spatial correspondence between the point cloud and the image feature is calculated through the coordinate projection model, and the correspondence is optimized by using the iterative closest point algorithm or the feature descriptor matching method, and a large number of accurate cross-modal matching point pairs are formed; In step A4, the above cross-modal matching information is used as a constraint, and a nonlinear optimization algorithm such as Levenberg-Marquardt method is used:

[0029] wherein, represents the parameter update amount to be solved, contains all the external parameter variables to be optimized, i.e. each rotation matrix, usually represented by Euler angles or quaternions, and converted into a vector form and a translation vector; represents the error function with respect to the to-be-optimized parameters , each row of the matrix corresponds to an observation data point, and each column corresponds to a to-be-optimized parameter; represents an approximation to the Hessian matrix of the error function; represents a damping factor; l represents a unit matrix, which has the same dimension as the length of the parameter vector ; The reprojection error or the spatial distance error between the observation data of different sensors in the group is minimized, and the optimal rotation and translation parameters between the sensors are iteratively solved, and finally the relative position relationship between all the sensor units is accurately calibrated, laying a solid spatial geometric foundation for the accurate fusion and unification of subsequent multi-source data.

[0030] As a preferred embodiment, in step S3, the hierarchical fusion architecture is specifically: Data-level fusion directly integrates the texture information of the RGB image and the geometric information of the depth camera to generate an enhanced environment representation; Feature-level fusion extracts point cloud features, image features and inertial features from the data of the sensor units, and performs weighted fusion through an attention mechanism; Decision-level fusion uses D-S evidence theory to comprehensively decide the independent recognition results of the sensor units.

[0031] Specifically, in the data-level fusion stage, the system first directly integrates the RGB images and depth camera point clouds that have completed spatio-temporal alignment. Through the calibrated camera intrinsic and extrinsic parameters, the pixel color information of the RGB images is accurately mapped to each corresponding point of the three-dimensional point cloud, generating a color point cloud with rich texture features, thereby constructing an enhanced environment representation with both geometric structure and surface appearance. In the feature-level fusion stage, the system extracts heterogeneous features from multi-source data in parallel. The PointNet++ network is used to extract geometric structure features from the lidar and depth camera point clouds, a convolutional neural network is used to extract texture and contour features from the RGB images, and the robot's motion inertia features are parsed from the IMU data. Then, the feature vectors of different modalities are mapped to the same high-dimensional space, and an attention mechanism is introduced to calculate the weight coefficients of each feature channel, which are determined by the current movement state of the robot and the real-time data quality. Finally, adaptive fusion of features is achieved through weighted summation. In the decision-level fusion stage, each sensor unit such as the visual sensor and lidar independently completes preliminary identification and classification of objects or obstacles in the environment based on its own data and outputs the identification results with confidence. The system uses the D-S evidence theory to synthesize these independent decisions by calculating the basic probability assignment function and combining different evidence bodies, effectively handling the uncertainty and conflicting information between sensors, and obtaining the final comprehensive recognition decision. This hierarchical architecture ensures the robustness and accuracy of the fusion process through step-by-step refinement from raw data to abstract decisions, providing reliable environmental perception results for robot navigation and operation.

[0032] As a preferred embodiment, in step S3, the quality of each sensor data is evaluated in real time and environmental interference factors are analyzed through a pre-trained weight level prediction network. The network takes multiple parallel sub-networks as input, one of which calculates quantitative indicators such as image sharpness, contrast, and noise level for visual sensor data, another of which calculates point cloud density and effective ranging point proportion for depth camera and lidar data, and another of which analyzes environmental light intensity sensor readings and dust concentration detection values. These real-time quantitative indicators are concatenated into a feature vector and input into the fully connected layer of the weight level prediction network. The network is pre-trained on a large amount of industrial scene data and can learn the complex mapping relationship between sensor reliability and its data indicators and environmental indicators under different working conditions. The network finally outputs a set of fusion weight coefficients corresponding to each sensor unit such as visual, depth, and lidar. These coefficients are normalized by the Softmax function to ensure that the sum of the weights is one. In the feature level and decision level stages of the hierarchical fusion architecture, the system applies this set of dynamic weights to weight and fuse feature vectors or independent recognition results from different sensors, for example, in strong light interference, automatically reducing the weight of the visual sensor and increasing the weight of the laser radar, and in a dusty environment, reducing the weight of the optical sensor and increasing the weight of the inertial navigation of the IMU; this adaptive adjustment mechanism based on data quality and environmental factors ensures that the fusion system always maintains optimal perception robustness in variable industrial scenarios.

[0033] As a preferred embodiment, by real-time analysis of joint encoder and IMU data to determine whether the robot is in gait movement or wheeled movement state, when detecting gait movement with alternating support of both feet, due to the periodic jitter of the body, the system dynamically enhances the filtering strength of the IMU data, uses an adaptive Kalman filter to expand the uncertainty range of state estimation by increasing the value of the attitude angle component in the process noise covariance matrix, thereby achieving strong attenuation of high-frequency jitter signals, while fusing joint torque sensor data to construct a full-body motion model, kinematically compensating the acceleration and angular velocity measured by the IMU, effectively suppressing the inertial measurement noise caused by limb swing; When detecting wheeled movement, the system prioritizes real-time obstacle detection in front, at this time appropriately reduces the IMU filtering strength to maintain the agility of the attitude response, and allocates higher data processing priority to the laser radar and depth camera, uses a sliding window method to quickly register and analyze the differences of the recent continuous multiple frames of point cloud, significantly improves the obstacle detection frequency on the premise of ensuring positioning accuracy; this mechanism enables the perception system to adapt to the motion characteristics of the robot body, and always maintains the stability and reliability of environmental perception in complex motion processes.

[0034] As a preferred embodiment, the depth camera is installed at the joint of the wheel-foot humanoid robot, and the local perspective information obtained with joint movement, the depth camera installed at the wrist or ankle joint of the robot changes the perspective with joint movement, and the local point cloud data obtained by the depth camera is converted to the global coordinate system in real time through the kinematic forward solution model and the pre-calibrated external parameter matrix from the joint to the robot base coordinate system. Meanwhile, the global camera and lidar mounted on the robot head or trunk provide a stable global environment map; the system spatiotemporally aligns the local view data with the global view data through unified timestamp, and fuses different view observations of the same spatial region, for example, when the robot arm performs operation close to the workpiece, the joint depth camera provides high-resolution local geometric details of the workpiece surface, while the head global camera provides the location context of the workpiece in the overall scene; during fusion, first identify the spatial region corresponding to the local view data in the global map, and then use point cloud registration algorithm to splice and fuse the local high-precision point cloud with the global point cloud, so as to dynamically update the geometric information of the region in the global map and enhance the detail resolution.

[0035] As a preferred embodiment, in step S4, the sensor state is diagnosed in real time by continuously monitoring the heartbeat signal and data stream validity of each sensor unit, when potential collision risks are identified, such as obstacles entering the safety threshold range or certain sensor units failing, such as laser radar data loss or long-time no valid output of visual sensor, the system automatically triggers a three-level safety warning mechanism, which can be artificially divided into primary acoustic-optical warning, intermediate operation delay and high-level emergency braking in actual use; At the same time, the system immediately starts the degraded operation mode, through the strategy of dynamically reconstructing the hierarchical fusion architecture, automatically excludes the data stream of the failed sensor and recalculates the remaining valid sensors, such as when the vision fails, the fusion weight of laser radar and IMU (i.e. inertial measurement unit) is increased, when the laser radar fails, the stereo perception of vision and depth camera is relied on and combined with IMU for motion calculation, to ensure that the basic environment recognition and obstacle detection functions are maintained; the degradation logic is built into the fusion decision core, which can adaptively adjust the parameters and output frequency of the perception algorithm according to the type and performance of the remaining sensors, so that the minimum necessary environmental perception information for the robot's navigation path planning and safety control can still be provided when some components fail, ensuring the basic operation safety and reliability of the system.

[0036] As a preferred embodiment, in the automatic assembly line scene, the system tracks the workpiece on the conveyor belt through the global camera, and uses the eye-hand camera mounted at the end of the robot arm for fine positioning at the grabbing station; the identified objects are mainly specific models of workpieces and assembly tools, whose texture features and three-dimensional models have been pre-stored in the database; In the intelligent warehouse area scene, the robot mainly uses wheeled movement, the system preferentially calls the laser radar and trunk depth camera for high-speed stack plate and shelf detection, and at the same time identifies the ground navigation markers and product markers through the visual sensor; the identified objects include pallets, shelves and personnel; In the logistics sorting station scenario, the robot needs to dynamically avoid other mobile robots. The system realizes real-time identification and tracking of personnel, package sorting equipment and other mobile robots by fusing panoramic scanning data of the head laser radar and low-view blind area detection data of the joint depth camera. The system adaptively adjusts the perception strategy according to the characteristics of different scenes. For example, in the assembly line, the high-weight visual fusion is adopted to emphasize the millimeter-level precision of the end operation; in the warehouse area, the high-weight point cloud fusion is adopted to emphasize the navigation efficiency; in the sorting station, the multi-sensor decision-level fusion is adopted to emphasize dynamic obstacle avoidance. This perception configuration based on scene semantics enables the robot to accurately adapt to the specific needs of typical industrial scenarios such as automated assembly, intelligent warehousing and logistics sorting, and realizes reliable identification and interaction with workpieces, equipment, personnel and other mobile robots.

[0037] The above is the working principle of the multi-sensor fusion identification method of the wheel-foot humanoid robot in the industrial scene.

Claims

1. A multi-sensor fusion recognition method for a wheeled-legged robot in an industrial scene, characterized in that, The method comprises the following steps: S1, constructing a sensor system: configuring a multi-modal sensing system composed of sensor units on a wheel-legged robot, establishing a synchronous triggering mechanism through the multi-modal sensing system to align the time of the sensor units; S2, data preprocessing and alignment: the original data collected by the multi-modal sensing system is preprocessed through image processing technology, the preprocessing includes two-stage optimization of joint bilateral filtering and Kalman filtering for depth images, and the preprocessed multi-source data is unified to the base coordinate system of the wheel-legged robot; S3, dynamic fusion recognition: a hierarchical fusion architecture is constructed, which adjusts the fusion weight of the sensor units according to the current wheeled or gait movement state of the wheel-legged robot and the characteristics of the industrial scene, and realizes the recognition and positioning of environmental objects and obstacles; S4, adaptive output: according to the results of the recognition and positioning, the navigation path planning, operation instruction and safety warning of the wheel-legged robot are output in real time.

2. The multi-sensor fusion recognition method for the wheel-legged robot in the industrial scene according to claim 1, characterized in that: In step S1, the sensor units include visual sensors, depth cameras, laser radars, inertial measurement units and torque sensors, and the sensor units are respectively installed on the torso, head, end effector and joint parts of the wheel-legged robot, the time alignment generates a unified time reference signal using a master-slave synchronization architecture with high-precision IMU as the master clock, the visual sensor includes a global camera for large-scale scene monitoring and an eye-hand camera for fine operation guidance, and the depth camera preferably uses time-of-flight technology.

3. The multi-sensor fusion recognition method for the wheel-legged robot in the industrial scene according to claim 1, characterized in that: In step S2, a multi-element external parameter calibration method is used to calibrate the space of multiple sensors, and IMU data is used to compensate the visual perception data caused by the motion of the wheel-legged robot in real time.

4. The multi-sensor fusion recognition method for the wheel-legged robot in the industrial scene according to claim 3, characterized in that: The steps of the multi-element external parameter calibration method are: A1, mapping the initial detection information corresponding to the sensor units to the preset coordinate system corresponding to the wheel-legged robot to obtain the conversion detection information corresponding to the data of the sensor units; A2, based on the spatial layout of the sensor units, divide multiple sensor units into at least one sensor group; A3, based on the conversion detection information corresponding to each sensor unit included in the sensor group, construct cross-modal matching information of a predetermined type of object; A4, based on the cross-modal matching information corresponding to the sensor group, calibrate the external parameters of multiple sensor units to obtain the relative position relationship between the sensor units.

5. The multi-sensor fusion recognition method for the wheeled-legged robot in the industrial scene according to claim 2, characterized in that: In step S3, the hierarchical fusion architecture is specifically: Data level fusion, directly integrating the texture information of RGB image and the geometric information of depth camera to generate enhanced environment representation; Feature level fusion, extracting point cloud features, image features and inertial features from the data of the sensor units, and performing weighted fusion through attention mechanism; Decision level fusion, using D-S evidence theory to comprehensively decide the independent recognition results of the sensor units.

6. The method of claim 1, wherein the method further comprises: In step S3, the adjustment of the fusion weight is adaptively completed by a pre-trained weight level prediction network based on the data quality evaluated by the sensor unit in real time and the light and dust interference factors of the environment.

7. The method of claim 1, wherein the method further comprises: It also includes an anti-interference mechanism for the motion characteristics of the wheel-legged robot: when the wheel-legged robot is in gait movement, the filtering strength of the IMU data is increased to compensate for body jitter; when the wheel-legged robot is in wheeled movement, the real-time performance of the front obstacle detection is prioritized.

8. The multi-sensor fusion recognition method of the wheeled-legged robot in the industrial scene according to claim 2, characterized in that: The depth camera is installed at the joint of the wheel-legged robot, and the local perspective information obtained with the joint movement is fused with the global perspective information obtained by the sensor unit of the trunk or head of the wheel-legged robot.

9. The method of claim 1, wherein the method further comprises: In step S4, the safety warning is automatically triggered when a potential collision risk or failure of the sensor unit is identified, and the data of the remaining effective sensor unit is degraded to run, so that the basic function of the sensor unit is not affected.

10. The method of claim 1, wherein the method further comprises: The industrial scene includes an automated assembly line, an intelligent warehouse area, and a logistics sorting station, and the identified objects include workpieces, equipment, personnel, and other mobile robots. The industrial scene includes an automated assembly line, an intelligent warehouse area, and a logistics sorting station, and the identified objects include workpieces, equipment, personnel, and other mobile robots.

Images