Identification method for intelligent sensing fastener of wheel-foot humanoid robot

Abstract

The invention relates to the technical field of intelligent sensing, and discloses a wheel foot humanoid robot intelligent sensing fastener identification method, which comprises the following steps that a multi-mode sensing system is arranged on a wheel foot humanoid robot body and an end effector; performing preprocessing on the multi-mode data; inputting the preprocessed data into a deep neural network model; on the basis of a motion error model of a robot end effector, the posture parameters obtained through recognition are compensated, and a corrected recognition result is obtained; self-adaptive optimization is carried out on a robot identification strategy through a reinforcement learning algorithm; fusing the visual features, the depth features and the haptic features based on a multi-modal feature fusion model; and further based on a confidence coefficient dynamic feedback mechanism. And a deep neural network model of the visual feature extraction sub-network and the time sequence modeling sub-network is used for performing dynamic weighting and key attitude capture on multi-frame time sequence features, performing high-precision identification on types, positions and attitudes of fasteners and extracting effects in a complex form.

Description

Technical Field

[0001] This invention relates to the field of intelligent sensing technology, specifically to a method for intelligent sensing and identification of fasteners by a wheeled humanoid robot. Background Technology

[0002] In the field of wheeled humanoid robot applications, intelligent perception and object recognition technologies have become key research directions, especially for the accurate identification of small objects, including fasteners, in complex environments. To achieve efficient operation, robots typically need multimodal perception capabilities, integrating different types of sensors, including vision, depth, and force sensors, to acquire multi-dimensional information about the environment. Through deep fusion and analysis of this multimodal data, robots can achieve high-precision object detection, localization, and pose estimation in dynamic environments.

[0003] With the continuous advancement of deep learning technology, neural network-based visual recognition systems have demonstrated significant advantages in processing and understanding multimodal data. Especially in visual feature extraction and temporal data modeling, by combining attention mechanisms with cross-modal feature mapping techniques, robots can more accurately capture the position and orientation of fasteners, thereby improving recognition accuracy and robustness.

[0004] In related technologies, the identification of small targets such as fasteners usually relies on single-frame images or single-modal features for judgment. The lack of sufficient modeling of temporal features and dynamic weight allocation of keyframes results in low recognition accuracy and stability when there are large changes in posture, complex lighting, or partial occlusion, making it difficult to achieve high-precision identification of fastener type, position, and posture. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for intelligently sensing and identifying fasteners in wheeled humanoid robots. This method solves the problem of low recognition accuracy and stability under conditions of large posture changes, complex lighting, or partial occlusion, making it difficult to achieve high-precision identification of fastener type, position, and posture.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for intelligently sensing and identifying fasteners in a wheel-legged humanoid robot, comprising the following steps: A multimodal perception system installed on the wheeled humanoid robot body and end effector acquires multimodal data including the target fastener; The multimodal data is preprocessed to obtain spatiotemporally aligned preprocessed data; The preprocessed data is input into a deep neural network model to extract and identify features of the fasteners, thereby obtaining the fastener's category information, spatial location, and attitude parameters. Based on the motion error model of the robot end effector, the identified posture parameters are compensated to obtain the corrected identification result; The robot recognition strategy is adaptively optimized using reinforcement learning algorithms to improve recognition stability under different environmental conditions. Based on a multimodal feature fusion model, visual features, depth features, and force features are fused to output the recognition results of fasteners; After obtaining the recognition results, the recognition model parameters are further fine-tuned online based on the confidence dynamic feedback mechanism.

[0007] Preferably, the multimodal sensing system includes a visible light camera, a structured light or TOF depth camera, and a six-dimensional force sensor; The multimodal data includes environmental image data, depth data, and force data; The multimodal sensing system further includes an adaptive sensing parameter adjustment module, which dynamically adjusts exposure, gain, and sampling frequency based on ambient light, reflectivity, and noise levels to adjust the quality of multimodal data.

[0008] Preferably, the preprocessing includes data registration, filtering and denoising, and normalization; The data registration in the preprocessing step includes multimodal data alignment based on timestamp synchronization and spatial coordinate calibration, which is used to unify the visual coordinate system and the force coordinate system; The preprocessing stage further includes a multi-scale feature enhancement mechanism, which uses a spatial pyramid filter to resample and fuse features at different scales to enhance the perception of complex fastener contours.

[0009] Preferably, the deep neural network model includes a visual feature extraction subnetwork and a temporal feature modeling subnetwork; The visual feature extraction subnetwork is used to extract spatial features of the image and depth information; The temporal feature modeling subnetwork is used to model the state changes between consecutive frames; The temporal feature modeling subnetwork further includes an attention mechanism module, which dynamically weights key frame features to identify the spatiotemporal correlation and key pose capture capabilities of the process.

[0010] Preferably, the motion error model is constructed based on the robot joint angular velocity, end-effector posture deviation, and external disturbance parameters, and is used to dynamically calculate the compensation matrix and correct the identified posture. The compensation model further includes historical statistics on task execution errors, and uses Bayesian estimation for dynamic error learning, which is used for attitude correction and adaptability.

[0011] Preferably, the reinforcement learning algorithm uses recognition accuracy, execution time, and stability as a comprehensive reward function, including training a policy network through state, action, and reward sequences to improve the adaptability of the recognition policy; The reinforcement learning model is a multi-objective reinforcement learning model based on policy gradient. Its reward function further includes an energy consumption constraint term and a posture error penalty term to balance recognition performance and robot operating energy efficiency.

[0012] Preferably, the multimodal feature fusion model is based on the principle of maximizing information entropy, and assigns weights according to the mutual information of each modality data for weighted fusion of visual features and force features; The fusion model further employs a cross-modal feature modeling method based on graph neural networks to construct an intermodal correlation graph structure to capture higher-order correlations between visual, depth, and force features.

[0013] Preferably, the recognition result is fed back to the reinforcement learning module for updating the policy parameters and continuously updating the recognition performance; The recognition model is fine-tuned online based on a confidence dynamic feedback mechanism. When the recognition confidence is lower than a set threshold, a local retraining process is triggered, and the model parameters are updated using a self-supervised contrastive learning strategy.

[0014] Preferably, the output recognition results include fastener type, three-dimensional coordinate position, attitude angle and confidence parameters, and are used to guide the robot to perform fastening or disassembly tasks; After outputting the recognition results, proceed with the following steps: After outputting the fastener recognition results, a task feedback signal is generated based on the success rate of the robot performing fastening or disassembly tasks. This signal is then fed back to the reinforcement learning module for joint optimization of the recognition strategy and the action strategy.

[0015] A system for intelligently sensing and identifying fasteners in a wheel-legged humanoid robot, the system comprising: The perception module is used to acquire image, depth, and force data of the target environment; The preprocessing module is used to register, filter, and normalize multimodal data; The identification module is used to identify the type, location, and orientation of fasteners based on a neural network model. The compensation module is used to dynamically correct the recognized posture based on the motion error model. The learning module is used to adjust the recognition strategy based on environmental conditions. The fusion module is used to fuse multimodal features based on the information entropy criterion and output high-confidence recognition results.

[0016] This invention provides a method for intelligently sensing and identifying fasteners in a wheel-legged humanoid robot. It has the following beneficial effects: 1. This invention uses a deep neural network model consisting of a visual feature extraction subnetwork and a temporal modeling subnetwork, combined with cross-modal feature mapping and attention mechanisms, to achieve dynamic weighting and key pose capture of multi-frame temporal features, high-precision identification of fastener categories, positions and poses, and extraction effects under complex shapes.

[0017] 2. This invention constructs a motion error model based on joint angular velocity, posture deviation, and external disturbances, and uses Bayesian estimation combined with historical error statistics for dynamic compensation. At the same time, it utilizes reinforcement learning algorithms to optimize the reward function with multiple objectives, including recognition accuracy, execution time, and stability. This achieves the effect of high accuracy in posture correction, strong execution stability, and continuous self-optimization capability for the robot under various working conditions.

[0018] 3. This invention allocates the weights of visual, depth, and force modalities by adopting the principle of maximizing information entropy, and uses graph neural networks to construct semantic and geometric association graph structures between modalities to achieve feature fusion. This achieves the effect of fully preserving key information of multiple modalities, significantly improving the confidence of fastener recognition, and recognition in complex environments.

[0019] 4. This invention integrates a visible light camera, a depth camera, and a six-dimensional force sensor into the wheeled humanoid robot body and end effector, and introduces an adaptive sensing parameter adjustment module to dynamically adjust exposure, gain, and sampling frequency according to ambient light, reflectivity, and noise levels. This achieves the effect of stably acquiring high signal-to-noise ratio and high structural clarity multimodal data even in complex lighting and noise environments. Attached Figure Description

[0020] Figure 1 This is a flowchart of the intelligent sensing fastener recognition method for wheeled humanoid robots according to the present invention; Figure 2 This is a system architecture diagram of the intelligent sensing fastener recognition system for wheeled humanoid robots according to the present invention. Detailed Implementation

[0021] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example: Please see the appendix Figure 1 This invention provides a method for intelligently sensing and identifying fasteners in a wheeled humanoid robot, comprising the following steps: A multimodal perception system installed on the wheeled humanoid robot body and end effector acquires multimodal data including the target fastener; Multimodal data is preprocessed to obtain spatiotemporally aligned preprocessed data; The preprocessed data is input into a deep neural network model to extract and identify features of fasteners, thereby obtaining fastener category information, spatial location, and attitude parameters. Based on the motion error model of the robot end effector, the identified posture parameters are compensated to obtain the corrected identification result; The robot recognition strategy is adaptively optimized using reinforcement learning algorithms to improve recognition stability under different environmental conditions. Based on a multimodal feature fusion model, visual features, depth features, and force features are fused to output the recognition results of fasteners; After obtaining the recognition results, the parameters of the recognition model are further fine-tuned online based on the confidence dynamic feedback mechanism.

[0023] Multimodal sensing systems include visible light cameras, structured light or TOF depth cameras, and six-dimensional force sensors; Multimodal data includes environmental image data, depth data, and force data; The multimodal sensing system further features an adaptive sensing parameter adjustment module, which dynamically adjusts exposure, gain, and sampling frequency based on ambient light, reflectivity, and noise levels to adjust the quality of multimodal data.

[0024] Specifically, the camera captures environmental images, the depth camera provides spatial distance information, and the force sensor measures the contact force and torque between the end effector and the object. An integrated adaptive sensing parameter adjustment module adjusts the parameters based on ambient lighting. Reflectivity With noise Dynamically adjust exposure Gain With depth sampling frequency

[0025] ; in These are the initial parameters. This is an empirical coefficient. It is a micro constant. To reference illumination values, closed-loop adjustment is used to maintain the signal-to-noise ratio and structural sharpness of the image and depth signals. After data acquisition, the system performs time synchronization, spatial registration, and filtering. Time synchronization combines hardware triggering and software timestamps, ensuring that visual, depth, and force data correspond to the same moment. Spatial registration unifies coordinates through a calibration matrix. , ; in Point at camera coordinates , The transformation matrix is ​​a rigid body. Gaussian and median filtering is used to reduce noise interference. A multi-scale spatial pyramid convolutional structure is introduced in the preprocessing stage to enhance feature resolution and preserve complex edge details. The preprocessed multimodal data is input into a deep neural network. The network consists of a visual feature extraction subnetwork and a temporal modeling subnetwork. The visual subnetwork is based on an improved ResNet structure and incorporates a cross-modal mapping layer to align RGB with depth features. The temporal subnetwork uses a bidirectional GRU structure to handle inter-frame variations and introduces an attention mechanism with the following weights: ; in For the first Frame hidden state, The weight matrix outputs fused features with both spatial and temporal semantics. To reduce the impact of motion errors, an error compensation model is introduced, which considers joint angular velocities. Posture deviation External disturbances The error estimation function is: ; in Given the weight matrix, Bayesian estimation is used to update the error parameters: ; To ensure real-time correction and stable recognition of the end-effector pose, a reinforcement learning model is introduced during the recognition stage for policy optimization. The reward function is defined as: ; in For accuracy, Because it takes time, For stability, For energy consumption, For attitude error, As weighting coefficients, the system maintains high recognition efficiency in changing environments through policy gradient optimization. Multimodal feature fusion follows the principle of maximizing information entropy, and computational vision... ,depth Sense of strength Inter-information The weights are: ; A modal correlation graph is constructed using a graph neural network, where nodes represent feature channels and edges represent correlations. Node features are updated through message passing to generate a fused feature vector. The final output includes fastener category, 3D coordinates, attitude angle, and confidence score. If the confidence score is below a threshold... The system automatically performs self-supervised retraining, optimizes the embedding space distribution through comparative learning, improves adaptive recognition, and feeds the recognition results back to the reinforcement learning module for closed-loop optimization.

[0026] Preprocessing includes data registration, filtering and denoising, and normalization; The data registration step in the preprocessing process includes multimodal data alignment based on timestamp synchronization and spatial coordinate calibration, which is used to unify the visual coordinate system and the force coordinate system; The preprocessing stage further includes a multi-scale feature enhancement mechanism, which uses a spatial pyramid filter to resample and fuse features at different scales to enhance the perception of complex fastener contours.

[0027] Specifically, preprocessing includes data registration, filtering and denoising, and normalization operations to eliminate temporal and spatial errors between different sensors and improve the consistency and usability of input data. Data registration includes multimodal alignment processing based on timestamp synchronization and spatial coordinate calibration. The time synchronization part uses hardware trigger signals combined with software timestamp correction to ensure that visual frames, depth frames, and force signals correspond to the physical state at the same moment. Spatial alignment uses coordinate calibration parameters to unify the visual coordinate system and the force coordinate system. Let the three-dimensional point in the camera coordinate system be... The corresponding points in the visual coordinate system and the force coordinate system are respectively and Then we have: ; in, , These are the rigid body transformation matrices from the camera coordinate system to the visual and force coordinate systems, respectively. This mapping relationship unifies the multimodal spatial coordinates, making the data from different modalities comparable and enabling fusion. The basic filtering and denoising section employs adaptive filtering algorithms tailored to the characteristics of each modal data. For visual and depth signals, a weighted fusion method of Gaussian filtering and median filtering is used to eliminate random noise. For force signals, sliding window and root mean square filtering are used to smooth out high-frequency disturbances caused by mechanical vibration or sensor drift. The filtered data is then normalized to unify the distribution of the data across modalities. This facilitates subsequent feature extraction and fusion in deep models; The preprocessing stage further includes a multi-scale feature enhancement mechanism, employing a spatial pyramid filter to perform multi-scale resampling and feature fusion on the input data. Let the original input be... At different scales Filtering and downsampling are then performed to obtain multi-layer features. The fused output can be represented as: ; in, The feature weights at each scale are determined by energy normalization, which can enhance the details and textures while maintaining the overall outline, giving the model higher resolution and stability in edge recognition of complex fasteners.

[0028] The deep neural network model includes a visual feature extraction subnetwork and a temporal feature modeling subnetwork; The visual feature extraction subnetwork is used to extract spatial features from images and depth information; The temporal feature modeling subnetwork is used to model the state changes between consecutive frames; The temporal feature modeling subnetwork further includes an attention mechanism module, which dynamically weights key frame features to identify the spatiotemporal correlation and key pose capture capabilities of the process.

[0029] Specifically, a deep neural network model, including a visual feature extraction subnetwork and a temporal feature modeling subnetwork, is used to extract and model spatial and temporal features in multimodal data to improve the recognition accuracy of fasteners. Through a cross-modal feature mapping layer, the visual subnetwork performs feature alignment on image data and depth information, so that data from different modalities can share the same feature space. The network structure consists of multiple convolutional layers and residual connections, effectively extracting the edges, textures, and other salient features of fasteners. The temporal feature modeling sub-network is specifically designed to handle state changes between consecutive frames, capturing the spatiotemporal correlations in dynamic processes. This sub-network, based on a bidirectional gated recurrent unit (GRU) model, effectively captures long-term dependencies between frames and improves the accuracy of state modeling by combining forward and reverse information flows. The bidirectional GRU not only processes information from the current frame but also understands the states of preceding and following frames through reverse information, thereby enhancing the expressive power of temporal features. The attention mechanism module dynamically weights keyframe features to strengthen attention to important spatiotemporal features by calculating the attention weights for each frame. The module dynamically weights the features of the current frame: ; in, For the first The hidden state of a frame. This is the weight matrix. This is the attention weight for that frame. In this way, the network can focus on key poses and frames with significant changes, improving the accuracy of fastener pose and position capture.

[0030] The motion error model is constructed based on the robot's joint angular velocity, end-effector posture deviation, and external disturbance parameters. It is used to dynamically calculate the compensation matrix and correct the identified posture. The compensation model further includes historical statistics on task execution errors, and uses Bayesian estimation for dynamic error learning, which is used for attitude correction and adaptability.

[0031] Specifically, the system collects the angular velocities of each joint of the robot. End effector attitude deviation and external disturbance forces And based on the linear weighted model, a motion error estimation function is established: ; in, Represents the combined motion error vector. , , The system uses an empirical weight matrix, corresponding to velocity, attitude, and external disturbance influence factors, respectively. Based on the real-time acquired state variables, it calculates the error distribution and generates a dynamic compensation matrix.

[0032] ; in, It is the identity matrix. This is the adaptive gain coefficient matrix, which is used to correct the pose estimation in the recognition result so that the coordinates of the end effector are consistent with the desired recognized pose. To enhance the system's self-learning and long-term accuracy maintenance capabilities, the compensation model further incorporates historical statistical information on task execution errors. The system models the distribution characteristics of recognition errors and attitude deviations in historical tasks to form error priors. And combined with current mission observation data Dynamic error learning is performed using Bayesian estimation: ; By continuously updating the posterior distribution of the error, the system can adaptively adjust the weights of each parameter, enabling dynamic optimization of the error correction strategy. This improves the robot's recognition stability and adaptability under different working conditions. After the compensation mechanism is applied, the robot's end-effector posture error is significantly reduced, and the 3D position and posture angle of the recognition output are more accurate, providing highly consistent input for subsequent reinforcement learning optimization and multimodal feature fusion.

[0033] Reinforcement learning algorithms use recognition accuracy, execution time, and stability as a comprehensive reward function, including training a policy network through state, action, and reward sequences to improve the adaptability of the recognition policy. The reinforcement learning model is a multi-objective reinforcement learning model based on policy gradient. Its reward function further includes an energy consumption constraint term and a posture error penalty term to balance recognition performance and robot operating energy efficiency.

[0034] Specifically, reinforcement learning algorithms use recognition accuracy, execution time, and stability as their main reward functions, defined as follows: ; in, To improve recognition accuracy, For execution time, For system stability, , , These are the weighting coefficients; This reward function comprehensively evaluates the robot's recognition performance in different environments and provides optimization directions for the system. The reinforcement learning model adopts a policy gradient-based approach, training the policy network through state, action, and reward sequences. Contains all important information about the current task environment, action space The reward function represents possible policy actions and guides the policy learning process. Through multiple interactions and feedback, the policy network gradually adjusts the recognition policy to improve recognition accuracy, reduce execution time, and enhance system stability. Considering the impact of energy efficiency and posture accuracy on robot performance, the reward function further includes energy consumption constraints and posture error penalty terms, as follows: ; in, Energy consumed during mission execution. For the attitude error of the end effector, , To adjust the coefficients, by introducing energy efficiency constraints, the system can optimize operating efficiency and extend robot working time while ensuring recognition accuracy. Meanwhile, the posture error penalty term ensures the robot's posture correction capability during task execution, reducing deviations in recognition accuracy.

[0035] The multimodal feature fusion model is based on the principle of maximizing information entropy and assigns weights according to the mutual information of each modality's data for the weighted fusion of visual and force features. The fusion model further employs a cross-modal feature modeling method based on graph neural networks to construct an intermodal correlation graph structure to capture higher-order correlations between visual, depth, and force features.

[0036] Specifically, the system first calculates the mutual information between the data of each modality to reflect the degree of information correlation between different sensory sources. Let visual features, depth features, and force features be respectively... Their mutual information Used to characterize the degree of information sharing among different modes. The modal weights are calculated based on mutual information.

[0037] ; in, For the first Mutual information between one mode and other modes This corresponds to the fusion weights. By maximizing the overall information entropy... The system maintains optimal information content during feature fusion, ensuring that effective features from each modality are fully preserved. This weighting mechanism enables the dynamic fusion of visual and force features, improving the model's recognition stability under different lighting, pose, and contact conditions. Multimodal features are mapped to nodes in a graph structure, and the edges between nodes represent the semantic and geometric relationships between different modal features. Let the feature node set be... Its association diagram can be represented as ,in This is a set of edges connecting features of different modalities. Graph neural networks update node features through a message-passing mechanism: ; in, Indicates the first Layer nodes Feature representation, For nodes The set of adjacent nodes, , These are the weight matrix and the bias term, respectively. Using a non-linear activation function, the model learns complex dependencies between different modalities through multi-layer message passing and feature aggregation, enabling efficient information fusion. The fused comprehensive feature vector is mapped through a fully connected layer to output the fastener's category, 3D position, attitude angle, and confidence score. If the recognition confidence score is lower than a set threshold δc, the system automatically triggers a self-supervised retraining process, updating the embedding space distribution through a contrastive learning mechanism, thereby improving feature representation ability and recognition accuracy.

[0038] The recognition results are fed back to the reinforcement learning module for updating policy parameters and continuously improving recognition performance; The recognition model is fine-tuned online based on a confidence dynamic feedback mechanism. When the recognition confidence is lower than a set threshold, a local retraining process is triggered, and the model parameters are updated using a self-supervised contrastive learning strategy.

[0039] Specifically, the recognition results are fed into the reinforcement learning module via a feedback mechanism to update policy parameters, continuously optimizing recognition performance. The system dynamically adjusts the policy network weights based on the accuracy and stability of the latest recognition results, ensuring the reinforcement learning model maintains the optimal recognition policy in multi-task environments. Simultaneously, the system establishes a dynamic feedback mechanism based on confidence level to fine-tune the recognition model online. When the recognition confidence level is low, reaching a set threshold... At this time, a local retraining process is triggered. This process is based on a self-supervised contrastive learning strategy, which adaptively updates the model parameters. By minimizing the similarity loss between feature embeddings, the model can still optimize the feature distribution even with unlabeled data. ; in, Represents the feature similarity function. For temperature parameters, As a feature representation of positive sample pairs, the model can self-optimize based on changes in confidence during real-time operation, enhancing its stability and generalization ability during recognition.

[0040] The output recognition results include fastener type, three-dimensional coordinate position, attitude angle and confidence parameters, and are used to guide the robot to perform fastening or disassembly tasks; After outputting the recognition results, proceed with the following steps: After outputting the fastener recognition results, a task feedback signal is generated based on the success rate of the robot performing fastening or disassembly tasks. This signal is then fed back to the reinforcement learning module for joint optimization of the recognition strategy and the action strategy.

[0041] Specifically, the system outputs recognition results including fastener type, 3D coordinate position, attitude angle, and confidence parameters, which guide the robot to perform fastening or disassembly tasks. The recognition results are directly mapped to the robot's operating coordinate system after coordinate transformation, ensuring seamless integration between recognition and execution. After outputting the recognition results, the system generates a task feedback signal based on the success rate of the robot performing the fastening or disassembly task. This signal comprehensively reflects the recognition accuracy and the action execution effect, and is used to update the reward item of the reinforcement learning module. By jointly modeling the task feedback and recognition confidence, the system simultaneously optimizes the recognition strategy and the action strategy, assuming the task success rate is... The confidence level of identification is The joint optimization reward function is then defined as: ; in, , These are weighting coefficients used to balance recognition accuracy and action execution performance. The reinforcement learning module dynamically adjusts the policy gradient based on feedback signals, enabling the robot to achieve a dual improvement in recognition and execution performance in subsequent tasks.

[0042] Please see the appendix Figure 2 A wheeled humanoid robot intelligent sensing and fastener recognition system, the system includes: The perception module is used to acquire image, depth, and force data of the target environment; The preprocessing module is used to register, filter, and normalize multimodal data; The identification module is used to identify the type, location, and orientation of fasteners based on a neural network model. The compensation module is used to dynamically correct the recognized posture based on the motion error model. The learning module is used to adjust the recognition strategy based on environmental conditions. The fusion module is used to fuse multimodal features based on the information entropy criterion and output high-confidence recognition results.

[0043] Specifically, the perception module collects image, depth, and force data of the target environment to obtain multimodal perception information; the preprocessing module registers, filters, and normalizes the collected data to ensure the consistency and stability of multimodal inputs; the recognition module identifies the type, position, and posture of fasteners based on a deep neural network, extracting spatial and temporal features to enable high-precision recognition of multi-source information; the compensation module dynamically corrects the recognition posture based on a motion error model, adjusts the recognition results in real time by combining robot joint states and external disturbance parameters, and uses historical error statistics for adaptive compensation; the learning module is based on a reinforcement learning algorithm, with recognition accuracy, execution stability, and energy efficiency as optimization goals, and continuously updates the policy network using task feedback to enable self-improvement of recognition and execution performance; The fusion module performs weighted fusion of visual, depth, and force features based on the principle of maximizing information entropy, and uses graph neural networks to model the intermodal relationships to output high-confidence recognition results. Through multimodal perception, dynamic compensation, and reinforcement learning self-optimization, the robot can achieve high-precision recognition and adaptive control of fasteners, significantly improving the stability and intelligence level of operation in complex environments.

[0044] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for intelligently sensing and identifying fasteners in a wheel-legged humanoid robot, characterized in that, Includes the following steps: A multimodal perception system installed on the wheeled humanoid robot body and end effector acquires multimodal data including the target fastener; The multimodal data is preprocessed to obtain spatiotemporally aligned preprocessed data; The preprocessed data is input into a deep neural network model to extract and identify features of the fasteners, thereby obtaining the fastener's category information, spatial location, and attitude parameters. Based on the motion error model of the robot end effector, the identified posture parameters are compensated to obtain the corrected identification result; The robot recognition strategy is adaptively optimized using reinforcement learning algorithms to improve recognition stability under different environmental conditions. Based on a multimodal feature fusion model, visual features, depth features, and force features are fused to output the recognition results of fasteners; After obtaining the recognition results, the recognition model parameters are further fine-tuned online based on the confidence dynamic feedback mechanism.

2. The method for intelligent sensing and identifying fasteners in a wheel-legged humanoid robot according to claim 1, characterized in that: The multimodal sensing system includes a visible light camera, a structured light or TOF depth camera, and a six-dimensional force sensor; The multimodal data includes environmental image data, depth data, and force data; The multimodal sensing system further includes an adaptive sensing parameter adjustment module, which dynamically adjusts exposure, gain, and sampling frequency based on ambient light, reflectivity, and noise levels to adjust the quality of multimodal data.

3. The method for intelligent sensing and identifying fasteners in a wheel-legged humanoid robot according to claim 1, characterized in that: The preprocessing includes data registration, filtering and denoising, and normalization. The data registration in the preprocessing step includes multimodal data alignment based on timestamp synchronization and spatial coordinate calibration, which is used to unify the visual coordinate system and the force coordinate system; The preprocessing stage further includes a multi-scale feature enhancement mechanism, which uses a spatial pyramid filter to resample and fuse features at different scales to enhance the perception of complex fastener contours.

4. The method for intelligently sensing and identifying fasteners in a wheel-legged humanoid robot according to claim 1, characterized in that: The deep neural network model includes a visual feature extraction subnetwork and a temporal feature modeling subnetwork; The visual feature extraction subnetwork is used to extract spatial features of the image and depth information; The temporal feature modeling subnetwork is used to model the state changes between consecutive frames; The temporal feature modeling subnetwork further includes an attention mechanism module, which dynamically weights key frame features to identify the spatiotemporal correlation and key pose capture capabilities of the process.

5. The method for intelligent sensing and identifying fasteners in a wheel-legged humanoid robot according to claim 1, characterized in that: The motion error model is constructed based on the robot's joint angular velocity, end-effector posture deviation, and external disturbance parameters. It is used to dynamically calculate the compensation matrix and correct the identified posture. The compensation model further includes historical statistics on task execution errors, and uses Bayesian estimation for dynamic error learning, which is used for attitude correction and adaptability.

6. The method for intelligently sensing and identifying fasteners in a wheel-legged humanoid robot according to claim 1, characterized in that: The reinforcement learning algorithm uses recognition accuracy, execution time, and stability as a comprehensive reward function, including training a policy network through state, action, and reward sequences to improve the adaptability of the recognition policy. The reinforcement learning model is a multi-objective reinforcement learning model based on policy gradient. Its reward function further includes an energy consumption constraint term and a posture error penalty term to balance recognition performance and robot operating energy efficiency.

7. The method for intelligent sensing and identifying fasteners in a wheel-legged humanoid robot according to claim 1, characterized in that: The multimodal feature fusion model is based on the principle of maximizing information entropy and assigns weights according to the mutual information of each modality data for the weighted fusion of visual features and force features. The fusion model further employs a cross-modal feature modeling method based on graph neural networks to construct an intermodal correlation graph structure to capture higher-order correlations between visual, depth, and force features.

8. The method for intelligently sensing and identifying fasteners in a wheel-legged humanoid robot according to claim 1, characterized in that: The recognition results are fed back to the reinforcement learning module for updating policy parameters and continuously updating recognition performance; The recognition model is fine-tuned online based on a confidence dynamic feedback mechanism. When the recognition confidence is lower than a set threshold, a local retraining process is triggered, and the model parameters are updated using a self-supervised contrastive learning strategy.

9. The method for intelligent sensing and identifying fasteners in a wheel-legged humanoid robot according to claim 1, characterized in that: The output recognition results include fastener type, three-dimensional coordinate position, attitude angle and confidence parameters, and are used to guide the robot to perform fastening or disassembly tasks. After outputting the recognition results, proceed with the following steps: After outputting the fastener recognition results, a task feedback signal is generated based on the success rate of the robot performing fastening or disassembly tasks. This signal is then fed back to the reinforcement learning module for joint optimization of the recognition strategy and the action strategy.

10. A system for identifying fasteners intelligently sensed by a wheeled humanoid robot, applied to the method for identifying fasteners intelligently sensed by a wheeled humanoid robot as described in any one of claims 1-9, characterized in that, The system includes: The perception module is used to acquire image, depth, and force data of the target environment; The preprocessing module is used to register, filter, and normalize multimodal data; The identification module is used to identify the type, location, and orientation of fasteners based on a neural network model. The compensation module is used to dynamically correct the recognized posture based on the motion error model. The learning module is used to adjust the recognition strategy based on environmental conditions. The fusion module is used to fuse multimodal features based on the information entropy criterion and output high-confidence recognition results.

Images