A mechanical hand cross-modal perception control system for chemical emergency disposal

Abstract

The application relates to the technical field of mechanical arms and relates to a mechanical arm cross-modal perception control system for chemical emergency disposal, which comprises an environment scanning module, a visual positioning module, an obstacle avoidance planning module, a pose compensation module, a decision mapping module and a grabbing updating module. The working area of the mechanical arm is pre-scanned to obtain environment state information. The operation visual angle image of the mechanical arm is collected, and the visual features of the operation visual angle image are calculated to obtain pose information. The motion trajectory of the mechanical arm is corrected to avoid obstacles to obtain a collision-free approaching path. The operation visual angle image and the multi-modal micro-tactile signal are fused to compensate the error of the pose information and obtain accurate grabbing pose. The object surface physical properties perceived by the multi-modal micro-tactile signal are mapped to obtain adaptive grabbing instructions. The adaptive grabbing instructions are used to control the mechanical arm to grab and update the environment state information. The application can improve the accuracy of the cross-modal perception control of the mechanical arm.

Description

Technical Field

[0001] This invention relates to the field of robotic arm technology, and in particular to a cross-modal perception and control system for a robotic arm for chemical emergency response. Background Technology

[0002] In chemical emergency response scenarios, robotic arms are core equipment for remote operations. The perception and control performance of robotic arms directly determines the efficiency and safety of the response operation. Existing robotic arm control systems are mostly based on single visual perception to achieve pose recognition and trajectory planning. In complex chemical operation environments, it is difficult to accurately capture the three-dimensional spatial features of the work area by relying solely on visual information. There are obvious limitations in the recognition and avoidance of obstacles, which can easily lead to deviations in motion trajectory planning and make it impossible to form a stable collision-free operation path, thus restricting the adaptability of robotic arms in chemical emergency scenarios.

[0003] Meanwhile, traditional robotic arms lack multimodal perception fusion and dynamic pose compensation mechanisms. During the grasping process, they cannot adaptively adjust the grasping strategy by combining the physical properties of the object's surface. They only complete the grasping action by preset parameters, which easily leads to problems such as the accumulation of pose errors and improper matching of grasping force and speed. This not only reduces the grasping success rate, but may also cause secondary safety risks such as chemical material leakage and equipment collision due to improper operation. In addition, the environmental status information cannot be updated in real time during the operation, making it difficult to adapt to the dynamic changes in the environment during chemical emergency response. The overall control accuracy and the level of operation intelligence cannot meet the actual technical requirements of chemical emergency response. Summary of the Invention

[0004] To achieve the above objectives, this invention provides a cross-modal perception and control system for a robotic arm in chemical emergency response, characterized in that the system includes an environmental scanning module, a visual positioning module, an obstacle avoidance planning module, a pose compensation module, a decision mapping module, and a grasping and updating module, wherein:

[0005] The environmental scanning module is used to perform a three-dimensional environmental pre-scan of the working area of ​​the robot to obtain the environmental status information of the robot.

[0006] The visual positioning module is used to acquire the operation view image of the robot based on the environmental state information, and to calculate the visual features of the operation view image to obtain the pose information of the robot.

[0007] The obstacle avoidance planning module is used to correct the motion trajectory of the robotic arm based on the pose information and the environmental state information, so as to obtain the collision-free approach path of the robotic arm.

[0008] The pose compensation module is used to fuse the operation view image and the multimodal micro-tactile signals collected by the robot hand based on the collision-free approach path, so as to perform error compensation on the pose information and obtain the precise grasping pose of the robot hand.

[0009] The decision mapping module is used to perform decision mapping on the physical properties of the object surface perceived by the multimodal micro-tactile signals based on the precise grasping pose, so as to obtain the adaptive grasping command of the robot arm.

[0010] The grasping and updating module is used to control the robotic arm to grasp using the adaptive grasping command, and to update the environmental status information according to the grasping status.

[0011] In a preferred embodiment, when the environmental scanning module performs a three-dimensional environmental pre-scan of the robot's working area to obtain the robot's environmental state information, it is specifically used for:

[0012] The robot arm carries a 3D laser scanner to scan the robot arm's working area and obtain the original point cloud data of the working area.

[0013] The original point cloud data is subjected to noise removal processing to obtain the denoised point cloud data of the working area;

[0014] The denoised point cloud data is reconstructed in three dimensions to obtain a three-dimensional point cloud distribution map of the work area;

[0015] The three-dimensional point cloud distribution map is used as the environmental state information of the robotic arm.

[0016] In a preferred embodiment, when the visual positioning module performs the tasks of acquiring the operating view image of the robot based on the environmental state information, and calculating the visual features of the operating view image to obtain the pose information of the robot, it is specifically used for:

[0017] Based on the environmental status information, the acquisition angle of the vision sensor mounted on the robot arm is adjusted so that the field of view of the vision sensor covers the working area, so as to acquire the operating perspective image of the robot arm;

[0018] The image from the operating perspective is subjected to grayscale transformation, and the transformed image is binarized to obtain the binarized image of the manipulator.

[0019] Edge detection is performed on the binarized image to obtain the contour feature image of the robotic arm;

[0020] Based on the contour feature image, the spatial position and posture angle of the robot are calculated to obtain the pose information of the robot.

[0021] In a preferred embodiment, when the obstacle avoidance planning module performs obstacle avoidance correction on the motion trajectory of the robotic arm based on the pose information and the environmental state information to obtain a collision-free approach path for the robotic arm, it is specifically used for:

[0022] Based on the pose information, the current pose information of the robotic arm and the target grasping position are used to perform trajectory planning to obtain the initial motion trajectory of the robotic arm;

[0023] Based on the distribution of obstacles in the environmental state information, the initial motion trajectory is segmented for detection to obtain the collision risk trajectory segment of the robotic arm;

[0024] Based on the collision risk trajectory segment, select an avoidance midpoint outside the obstacle;

[0025] Based on the avoidance midpoint, obstacle avoidance correction is performed on the collision risk trajectory segment to obtain the corrected trajectory segment of the robotic arm;

[0026] By sequentially connecting the non-collision risk segment in the initial motion trajectory with the corrected trajectory segment, a collision-free approach path for the robotic arm is obtained.

[0027] In a preferred embodiment, when the pose compensation module executes the operation view image and the multimodal micro-tactile signals collected by the robotic arm based on the collision-free approach path, it is specifically used for:

[0028] Control the robotic arm to move along the collision-free approach path to the target object;

[0029] The miniature tactile sensor array integrated into the fingertips of the robotic arm collects the contact force signal and vibration signal of the target object, thereby obtaining the multimodal micro-tactile signal of the target object;

[0030] The visual-tactile perception data of the robotic arm is obtained by aligning the operation view image with the multimodal micro-tactile signal in time.

[0031] In a preferred embodiment, when the pose compensation module performs error compensation on the pose information to obtain the precise grasping pose of the robotic arm, it is specifically used for:

[0032] Extract the edge contour of the target object from the operational viewpoint image to obtain the visual contour features of the target object;

[0033] The contact point distribution and local geometric features of the target object surface are extracted from the multimodal micro-tactile signals to obtain the tactile geometric features of the target object;

[0034] The visual contour features and the tactile geometric features are spatially reconstructed to obtain the fused feature map of the robotic hand;

[0035] Based on the fused feature map, residual compensation is performed on the pose information to obtain the precise grasping pose of the robotic arm.

[0036] In a preferred embodiment, when the pose compensation module performs residual compensation on the pose information based on the fused feature map to obtain the precise grasping pose of the robotic arm, it is specifically used for:

[0037] Based on the fused feature map, local feature points of the target object are extracted to construct the measured feature point set of the target object;

[0038] Based on the pose information, determine the theoretical feature point set corresponding to the measured feature point set of the target object in the theoretical pose.

[0039] The measured feature point set and the theoretical feature point set are analyzed by difference to obtain the residual vector set of the manipulator;

[0040] The weighted average of the residual vectors in the residual vector group is used to obtain the comprehensive residual vector of the robot arm. The formula for calculating the comprehensive residual vector is as follows: ;

[0041] In the formula, Let be the composite residual vector. The number of feature points in the measured feature point set. For the first Preset weight coefficients for each feature point For the set of measured feature points, the first... The position coordinates of each measured feature point For the set of theoretical feature points, the first The position coordinates of the theoretical feature points;

[0042] Based on the comprehensive residual vector, the pose information is optimized and compensated to obtain the precise grasping pose of the robotic arm.

[0043] In a preferred embodiment, when the decision mapping module performs decision mapping on the physical properties of the object surface sensed by the multimodal micro-tactile signals based on the precise grasping pose to obtain the adaptive grasping command of the robotic arm, it is specifically used for:

[0044] Based on the precise grasping posture, the expected contact area between the robotic arm and the target object is determined;

[0045] The contact force distribution information and surface roughness information related to the expected contact area are extracted from the multimodal micro-tactile signal to obtain the surface physical properties of the target object;

[0046] The surface physical properties are bidirectionally matched with the gripping parameters in the preset gripping strategy library to obtain the gripping force and gripping speed of the robotic arm.

[0047] The gripping force and the gripping speed are encoded into adaptive gripping commands for the robotic arm.

[0048] In a preferred embodiment, when the decision mapping module performs bidirectional matching of the surface physical properties with gripping parameters in a preset gripping strategy library to obtain the gripping force and gripping speed of the robotic arm, it is specifically used for:

[0049] Based on the surface physical properties, candidate grasping force ranges and candidate grasping speed ranges from the preset grasping strategy library are selected.

[0050] An initial grasping force is selected from the candidate grasping force range to control the robotic arm to apply pre-contact pressure to the surface of the target object, and the rate of change of contact force in the multimodal microtactile signal is monitored simultaneously.

[0051] Based on the rate of change of the contact force, the deformation state of the target object under the initial grasping force is determined, and the deformation response characteristics of the target object are obtained.

[0052] Based on the deformation response characteristics, the initial grasping speed of the candidate grasping speed range is determined;

[0053] Based on the initial grasping force and the initial grasping speed, the robotic arm is controlled to perform a trial grasping action on the target object, and the slippage signal during the trial grasping process is collected in real time.

[0054] Based on the intensity of the slip signal, the initial gripping force and the initial gripping speed are synchronously corrected to obtain the gripping force and gripping speed of the robotic arm.

[0055] In a preferred embodiment, when the grasping update module executes the adaptive grasping command to control the robotic arm to grasp, and updates the environmental state information according to the grasping state, it is specifically used for:

[0056] The adaptive grasping command is sent to the execution terminal of the robotic arm to control the robotic arm to grasp the target object;

[0057] During the grasping operation, the contact state between the robotic arm and the target object, as well as the positional changes of the target object, are monitored in real time to obtain the grasping state information of the robotic arm.

[0058] Based on the grasping status information, a judgment is made regarding the grasping of the target object;

[0059] When the capture is successful, the target object is removed from its original position in the environmental state information;

[0060] If the capture fails, the position information of the target object in the environmental state information is updated according to the current position of the target object.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. This invention achieves cross-modal perception and precise control of a chemical emergency response robot through a multi-module collaborative technical architecture. It utilizes 3D environment scanning to accurately model the work area, combines visual positioning to rapidly calculate the robot's pose, and then employs a segmented detection and trajectory correction module for obstacle avoidance planning. This efficiently plans a collision-free approach path, significantly improving the accuracy and efficiency of the robot's motion trajectory planning in complex chemical work environments, ensuring the stability and safety of the robot's movement. Simultaneously, the pose compensation module integrates visual and multimodal micro-tactile signals to accurately compensate for pose errors. Through differential analysis and weighted average calculation of feature point sets, it achieves precise solution of the residual vector, effectively improving the positioning accuracy of the robot's grasping pose. This allows the robot to accurately conform to the grasping position of the target object, laying a precise pose foundation for subsequent grasping operations.

[0063] 2. This invention utilizes a decision mapping module to achieve adaptive matching and dynamic correction of the grasping strategy based on the physical properties of the object's surface. Relying on physical attribute information extracted from multimodal micro-tactile signals, and combining this with deformation response and slip signals from trial grasping, it achieves precise control of grasping force and speed, significantly improving the adaptability and accuracy of the robotic arm's grasping actions and effectively increasing the grasping success rate. The grasping update module updates environmental status information in real time based on the grasping status, allowing the system to dynamically adapt to real-time environmental changes in chemical emergency response, continuously ensuring the accuracy of environmental perception for subsequent operations. This comprehensively improves the robotic arm's operational intelligence and continuous operation capability in chemical emergency response scenarios, achieving a comprehensive improvement in cross-modal perception and control efficiency, and fully adapting to the operational technical requirements of chemical emergency response. Attached Figure Description

[0064] Figure 1 This is a system architecture diagram of a cross-modal perception and control system for a robotic arm used in chemical emergency response, provided by an embodiment of the present invention.

[0065] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments belong to some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “said” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0068] Depending on the context, the word "if" or "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0069] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0070] In practice, the server-side equipment deployed in a cross-modal perception and control system for robotic arms in chemical emergency response may consist of one or more devices. This system can be implemented as a business instance, a virtual machine, or hardware devices. For example, it can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, it can be understood as software deployed on a cloud node, providing a cross-modal perception and control system for robotic arms in chemical emergency response to various user terminals. Alternatively, it can be implemented as a virtual machine deployed on one or more devices in a cloud node, with application software installed to manage various user terminals. Or, it can also be implemented as a server-side system composed of numerous identical or different types of hardware devices, with one or more hardware devices configured to provide a cross-modal perception and control system for robotic arms in chemical emergency response to various user terminals.

[0071] In terms of implementation, the cross-modal perception and control system for a robotic arm in chemical emergency response and the user terminal are mutually compatible. Specifically, if the cross-modal perception and control system for a robotic arm in chemical emergency response is implemented as an application installed on a cloud service platform, then the user terminal acts as a client establishing a communication connection with that application; or if the system is implemented as a website, then the user terminal acts as a webpage; or if it is implemented as a cloud service platform, then the user terminal acts as a mini-program within an instant messaging application.

[0072] like Figure 1 The figure shown is a system architecture diagram of a cross-modal perception and control system for a robotic arm for chemical emergency response, provided by an embodiment of the present invention.

[0073] The cross-modal perception and control system for a robotic arm used in chemical emergency response, as described in this invention, can be hosted on a cloud server. In terms of implementation, it can function as one or more service devices, or as an application installed in the cloud (e.g., a mobile service operator's server, server cluster, etc.), or it can be developed into a website. Depending on the functions implemented, the cross-modal perception and control system for a robotic arm used in chemical emergency response may include an environmental scanning module, a visual positioning module, an obstacle avoidance planning module, a pose compensation module, a decision mapping module, and a capture and update module. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by an electronic device's processor and perform a fixed function, stored in the electronic device's memory.

[0074] In this embodiment of the invention, a cross-modal sensing and control system for a robotic arm in chemical emergency response can be implemented independently and called upon other modules. This "calling" can be understood as one module connecting to multiple modules of another type and providing corresponding services to those connected modules. This embodiment of the invention provides a cross-modal sensing and control system for a robotic arm in chemical emergency response, where the applicable scope of the system architecture can be adjusted by adding modules and directly calling them without modifying the program code. This achieves cluster-based horizontal expansion, enabling quick and flexible expansion of the system. In practical applications, these modules can be located in the same or different devices, or in virtual devices, such as service instances on a cloud server.

[0075] The following describes, with reference to specific embodiments, each component and its specific workflow of a cross-modal perception control system for a chemical emergency response robot:

[0076] The environmental scanning module is used to perform a three-dimensional environmental pre-scan of the working area of ​​the robot to obtain the environmental status information of the robot.

[0077] In this embodiment of the invention, when the environmental scanning module performs a three-dimensional environmental pre-scan of the robot's working area to obtain the robot's environmental state information, it is specifically used for:

[0078] The robot arm carries a 3D laser scanner to scan the robot arm's working area and obtain the original point cloud data of the working area.

[0079] The original point cloud data is subjected to noise removal processing to obtain the denoised point cloud data of the working area;

[0080] The denoised point cloud data is reconstructed in three dimensions to obtain a three-dimensional point cloud distribution map of the work area;

[0081] The three-dimensional point cloud distribution map is used as the environmental state information of the robotic arm.

[0082] The environmental scanning module sends control commands to the robotic arm, instructing it to start the 3D laser scanner carried at its end effector. Simultaneously, the module controls the robotic arm to move slowly along a preset scanning trajectory, ensuring that the 3D laser scanner can cover the entire working area of ​​the robotic arm. During its movement, the 3D laser scanner continuously emits laser beams into the working area. When the laser beams come into contact with the surfaces of various objects in the working area, they are reflected. The 3D laser scanner receives the reflected laser signals in real time and collects the position information of each spatial point in the working area based on the time difference between the emission and reception of the laser signals and the direction of propagation. After all the collected spatial point information is summarized, the original point cloud data of the working area is formed.

[0083] The collected raw point cloud data was checked point by point, with a focus on identifying points in the raw point cloud data that did not match the spatial distribution characteristics of the main structure of the work area. These points mainly included discrete points caused by external light interference and air dust blocking the laser signal, as well as abnormal offset points caused by slight errors in the laser scanner itself. After identifying these invalid points, they were all removed from the raw point cloud data to completely eliminate the interference of invalid data on subsequent processing. After the complete point checking and removal operation, the denoised point cloud data of the work area was obtained.

[0084] The spatial coordinates of all valid points in the denoised point cloud data are extracted, and the relative positional relationship of each valid point in the robot's working area is analyzed one by one to clarify the spatial distance and distribution pattern between each point. Then, according to the actual spatial coordinates of each valid point, the scattered valid point cloud data are systematically stitched together, and the point cloud data collected from different scanning positions are accurately integrated together. During the stitching and integration process, the spatial correspondence of each point is strictly guaranteed to be without deviation. Through this systematic stitching and integration operation, the real spatial outline, structural form and distribution of various objects in the robot's working area are completely restored, and finally a three-dimensional point cloud distribution map of the working area is constructed.

[0085] The integrity of the constructed 3D point cloud distribution map is verified to confirm that it can comprehensively and accurately reflect all spatial structural features of the robot's working area, including key information such as the boundary range of the working area and the shape and distribution of objects inside. After verification, the 3D point cloud distribution map is directly determined as the robot's environmental state information, providing accurate and reliable environmental data support for the robot's path planning and posture adjustment for subsequent grasping, handling and other operations.

[0086] The beneficial effects are that 3D laser scanning combined with subsequent processing can accurately restore the 3D spatial features and object distribution of the chemical emergency operation area, breaking through the limitations of traditional single visual perception in capturing the 3D features of complex environments and improving the robot's perception accuracy of the working environment.

[0087] Noise removal effectively filters out invalid data generated by scanning interference, ensuring the authenticity and validity of point cloud data, providing a high-quality data foundation for subsequent 3D reconstruction, and avoiding environmental modeling deviations caused by invalid data.

[0088] The 3D point cloud distribution map obtained through 3D reconstruction serves as environmental status information, providing comprehensive and accurate environmental data support for subsequent modules such as visual positioning and obstacle avoidance planning, thus laying a reliable environmental perception foundation for the precise operation of each module.

[0089] This enables precise 3D modeling of the robotic arm's operating area, allowing the robotic arm to clearly perceive key information such as the distribution of environmental obstacles in chemical emergency scenarios, ensuring the accuracy of environmental judgments for subsequent trajectory planning, pose positioning, and other operations.

[0090] The visual positioning module is used to acquire the operation view image of the robot based on the environmental state information, and to calculate the visual features of the operation view image to obtain the pose information of the robot.

[0091] In this embodiment of the invention, when the visual positioning module performs the tasks of acquiring the operating view image of the robot based on the environmental state information, and calculating the visual features of the operating view image to obtain the pose information of the robot, it is specifically used for:

[0092] Based on the environmental status information, the acquisition angle of the vision sensor mounted on the robot arm is adjusted so that the field of view of the vision sensor covers the working area, so as to acquire the operating perspective image of the robot arm;

[0093] The image from the operating perspective is subjected to grayscale transformation, and the transformed image is binarized to obtain the binarized image of the manipulator.

[0094] Edge detection is performed on the binarized image to obtain the contour feature image of the robotic arm;

[0095] Based on the contour feature image, the spatial position and posture angle of the robot are calculated to obtain the pose information of the robot.

[0096] The visual positioning module acquires the environmental state information of the generated robotic arm. By analyzing the 3D point cloud distribution map of the work area contained in the environmental state information, it clarifies the spatial boundaries of the work area, the location of the core operation area, and the surrounding environmental structure. Then, it sends an angle adjustment command to the robotic arm, controlling the robotic arm to drive its onboard visual sensor to rotate slowly. During the adjustment process, it acquires the current field of view of the visual sensor in real time and compares this field of view with the work area range in the environmental state information. It continuously adjusts the vertical and horizontal acquisition angles of the visual sensor until the field of view of the visual sensor can completely cover the entire work area, and the center of the field of view is accurately aligned with the core operation position of the work area, ensuring that all movements of the robotic arm and the surrounding environment can be clearly captured during the operation. At this time, it controls the visual sensor to start image acquisition, continuously capturing real-time images of the robotic arm during the operation, and finally acquires the image of the robotic arm's operating perspective.

[0097] The acquired images of the robot's operating perspective are imported into the image processing unit of the vision positioning module. Grayscale transformation is performed on these images, adjusting the brightness of each pixel to reduce the differences in light and shadow caused by uneven illumination, eliminating interference from irrelevant light and shadow on image details, and maintaining uniform overall brightness. This allows for a preliminary distinction between the robot's outline and the background of the work area. The transformed image is then binarized. A fixed grayscale judgment standard is set, and each pixel in the transformed image is judged one by one. Pixels with grayscale values ​​matching the standard are identified as target pixels and given a uniform solid color display state, while pixels with grayscale values ​​not matching the standard are identified as background pixels and given a different uniform solid color display state. This clear differentiation process thoroughly removes redundant information from the image, allowing the robot's outline to stand out clearly, ultimately resulting in a binarized image of the robot.

[0098] The obtained binarized image of the robotic arm is imported into the edge detection unit of the vision positioning module. Comprehensive and detailed edge detection is performed on the binarized image. By scanning the binarized image row by row and column by column, the display status of adjacent pixels is compared one by one to capture all pixels where the display status changes abruptly. These pixels are the boundaries between the robotic arm outline and the background. Then, these pixels with abrupt display status changes are connected in an orderly manner according to their distribution order in the image to form continuous lines. At the same time, breaks, stray lines and isolated pixels caused by image interference are removed from the lines to ensure that the connected lines are coherent and complete, and can accurately delineate the overall shape outline of the robotic arm and the specific shape of each joint, finally obtaining the outline feature image of the robotic arm.

[0099] Extract all continuous contour lines from the contour feature image of the robotic arm, meticulously analyze the direction, curvature, length, and interconnections of each contour line, clarify the specific distribution of the contours of each part of the robotic arm in the image, and combine the acquired environmental state information of the robotic arm with the 3D point cloud distribution map in the environmental state information as a spatial coordinate reference. By analyzing the spatial projection relationship of the robotic arm contour in the contour feature image, determine the corresponding position of the robotic arm contour in 3D space, and then calculate the specific spatial position of the robotic arm in the working area. At the same time, based on the curvature angle, direction, and the angle between each contour line, determine the rotation amplitude and direction of each joint of the robotic arm, and then calculate the posture angle of the robotic arm. Integrate the calculated spatial position and posture angle of the robotic arm to clarify the current spatial placement and action posture of the robotic arm, and finally obtain the pose information of the robotic arm.

[0100] The beneficial effects are that the visual sensor's acquisition angle can be precisely adjusted based on environmental condition information to ensure that the field of view completely covers the work area, guarantee the comprehensiveness and effectiveness of image acquisition from the operating perspective, and provide a high-quality image foundation for subsequent visual feature calculation.

[0101] By removing redundant information from the image through grayscale transformation and binarization, the contour features of the robotic arm are effectively highlighted, and the influence of interference factors such as light and shadow and background on feature extraction is eliminated, thereby improving the clarity of contour recognition.

[0102] Edge detection is performed on the binarized image to form a contour feature image, which accurately delineates the overall shape and joint morphology of the robot, providing accurate and intuitive visual feature basis for pose estimation.

[0103] Based on contour feature images combined with the three-dimensional spatial coordinates of the work area, pose estimation is completed, enabling rapid and accurate calculation of the robot's spatial position and attitude angle. This breaks through the limitations of traditional single visual perception, providing accurate basic pose data for subsequent trajectory planning and pose compensation, and improving the pose recognition accuracy and operational adaptability of the robot in complex chemical emergency environments.

[0104] The obstacle avoidance planning module is used to correct the motion trajectory of the robotic arm based on the pose information and the environmental state information, so as to obtain the collision-free approach path of the robotic arm.

[0105] In this embodiment of the invention, when the obstacle avoidance planning module performs obstacle avoidance correction on the motion trajectory of the robotic arm based on the pose information and the environmental state information to obtain a collision-free approach path for the robotic arm, it is specifically used for:

[0106] Based on the pose information, the current pose information of the robotic arm and the target grasping position are used to perform trajectory planning to obtain the initial motion trajectory of the robotic arm;

[0107] Based on the distribution of obstacles in the environmental state information, the initial motion trajectory is segmented for detection to obtain the collision risk trajectory segment of the robotic arm;

[0108] Based on the collision risk trajectory segment, select an avoidance midpoint outside the obstacle;

[0109] Based on the avoidance midpoint, obstacle avoidance correction is performed on the collision risk trajectory segment to obtain the corrected trajectory segment of the robotic arm;

[0110] By sequentially connecting the non-collision risk segment in the initial motion trajectory with the corrected trajectory segment, a collision-free approach path for the robotic arm is obtained.

[0111] The obstacle avoidance planning module acquires the pose information of the robotic arm generated by the visual positioning module, clarifies the current spatial position and attitude angle of the robotic arm, and obtains the preset target grasping position of the robotic arm. The current pose information of the robotic arm is used as the starting reference for trajectory planning, and the target grasping position is used as the endpoint of trajectory planning. Combining the motion characteristics of the robotic arm and the spatial constraints of the working area, a continuous motion path from the current pose to the target grasping position is planned. This path can ensure that the robotic arm has a stable posture and smooth movements during the movement, which conforms to the motion law of the robotic arm, and finally obtains the initial motion trajectory of the robotic arm.

[0112] The system retrieves the environmental status information of the robotic arm, analyzes the 3D point cloud distribution map of the work area contained in the environmental status information, identifies the specific distribution location of all obstacles in the work area from the 3D point cloud distribution map, clarifies the shape outline and spatial occupation range of each obstacle, then divides the initial motion trajectory of the robotic arm into multiple continuous trajectory segments according to a preset length, and detects each trajectory segment one by one, comparing the spatial path of each trajectory segment with the distribution location of the obstacles to determine whether the trajectory segment will spatially overlap with the obstacles, and filters out all trajectory segments that spatially overlap with the distribution location of the obstacles and may cause the robotic arm to collide, finally obtaining the collision risk trajectory segments of the robotic arm.

[0113] For the identified collision risk trajectory segment of the robotic arm, the obstacle avoidance planning module further analyzes the obstacle distribution location in the environmental state information, clarifies the outer boundary of the obstacle corresponding to the collision risk trajectory segment, and selects a suitable spatial point at the outer boundary of the obstacle as the avoidance midpoint, based on the size of the robotic arm and the movement clearance. During the selection process, it is ensured that a sufficient safe distance is maintained between the avoidance midpoint and the obstacle to avoid collision with the obstacle when the robotic arm moves to this point. At the same time, it is ensured that a continuous path can be formed between the avoidance midpoint and the two endpoints of the collision risk trajectory segment, providing a reasonable transition for subsequent trajectory correction. Finally, the avoidance midpoint is selected on the outer perimeter of the obstacle.

[0114] Using the selected avoidance midpoint as the core, the obstacle avoidance planning module corrects the collision risk trajectory segment of the robotic arm. First, it determines the starting and ending points of the collision risk trajectory segment, connects the starting point to the avoidance midpoint, and plans a path that smoothly transitions from the starting point to the avoidance midpoint. Then, it connects the avoidance midpoint to the ending point of the collision risk trajectory segment, and plans a path that smoothly transitions from the avoidance midpoint to the ending point. During the connection of the two paths, it ensures that the path is continuous and without breaks, and that the curvature of the path matches the robotic arm's motion capabilities, avoiding sharp turns that the robotic arm cannot complete. Through this connection correction, the original collision risk trajectory segment is replaced with a collision-free path segment, and finally, the corrected trajectory segment of the robotic arm is obtained.

[0115] The initial motion trajectory of the robotic arm is broken down, and non-collision risk segments that are not identified as collision risk segments are separated. The starting order and connection relationship of the non-collision risk segments are determined. Then, according to the original motion direction of the initial motion trajectory, the non-collision risk segments are sequentially connected with the corrected trajectory segments. During the connection process, it is ensured that the paths of adjacent segments are smoothly connected without obvious jams or sudden angle changes, so as to ensure that the robotic arm moves with a stable posture and continuous movements. After the connection is completed, a complete motion path that can avoid all obstacles is formed, and finally, the collision-free approach path of the robotic arm is obtained.

[0116] The beneficial effect is that the initial motion trajectory is planned based on the pose information, so that the trajectory planning fits the current pose of the robot and the target grasping position, which is in line with the motion characteristics of the robot and lays a reasonable trajectory foundation for subsequent obstacle avoidance correction.

[0117] By combining environmental status information with segmented detection of the initial trajectory, collision risk trajectory segments can be accurately identified, enabling precise avoidance and prediction of obstacles in the work area, and improving the pertinence and accuracy of risk identification.

[0118] By selecting an avoidance midpoint around the perimeter of the obstacle and correcting the risk section, the obstacle can be effectively avoided. At the same time, the corrected trajectory segment can be made in line with the robot's motion capabilities, thus avoiding any motion actions that cannot be performed.

[0119] By connecting the non-collision risk sections with the corrected trajectory sections in an orderly manner, the resulting non-collision approach path is smoothly connected, ensuring that the robot arm's posture is stable and its movements are continuous, thus completely avoiding the risk of collision.

[0120] This improves the accuracy and efficiency of robotic arm trajectory planning in complex obstacle environments during chemical emergencies, ensuring that the robotic arm can safely and smoothly approach the target object and provide reliable motion path support for subsequent grasping operations.

[0121] The pose compensation module is used to fuse the operation view image and the multimodal micro-tactile signals collected by the robot hand based on the collision-free approach path, so as to perform error compensation on the pose information and obtain the precise grasping pose of the robot hand.

[0122] In this embodiment of the invention, when the pose compensation module executes the operation view image and the multimodal micro-tactile signals collected by the robotic arm based on the collision-free approach path, it is specifically used for:

[0123] Control the robotic arm to move along the collision-free approach path to the target object;

[0124] The miniature tactile sensor array integrated into the fingertips of the robotic arm collects the contact force signal and vibration signal of the target object, thereby obtaining the multimodal micro-tactile signal of the target object;

[0125] The visual-tactile perception data of the robotic arm is obtained by aligning the operation view image with the multimodal micro-tactile signal in time.

[0126] When the pose compensation module performs error compensation on the pose information to obtain the precise grasping pose of the robotic arm, it is specifically used for:

[0127] Extract the edge contour of the target object from the operational viewpoint image to obtain the visual contour features of the target object;

[0128] The contact point distribution and local geometric features of the target object surface are extracted from the multimodal micro-tactile signals to obtain the tactile geometric features of the target object;

[0129] The visual contour features and the tactile geometric features are spatially reconstructed to obtain the fused feature map of the robotic hand;

[0130] Based on the fused feature map, residual compensation is performed on the pose information to obtain the precise grasping pose of the robotic arm.

[0131] When the pose compensation module performs residual compensation on the pose information based on the fused feature map to obtain the precise grasping pose of the robotic arm, it is specifically used for:

[0132] Based on the fused feature map, local feature points of the target object are extracted to construct the measured feature point set of the target object;

[0133] Based on the pose information, determine the theoretical feature point set corresponding to the measured feature point set of the target object in the theoretical pose.

[0134] The measured feature point set and the theoretical feature point set are analyzed by difference to obtain the residual vector set of the manipulator;

[0135] The weighted average of the residual vectors in the residual vector group is used to obtain the comprehensive residual vector of the robot arm. The formula for calculating the comprehensive residual vector is as follows:

[0136] ;

[0137] In the formula, Let be the composite residual vector. The number of feature points in the measured feature point set. For the first Preset weight coefficients for each feature point For the set of measured feature points, the first... The position coordinates of each measured feature point For the set of theoretical feature points, the first The position coordinates of the theoretical feature points;

[0138] Based on the comprehensive residual vector, the pose information is optimized and compensated to obtain the precise grasping pose of the robotic arm.

[0139] The system obtains the collision-free approach path of the robotic arm generated by the obstacle avoidance planning module, analyzes the overall direction of the collision-free approach path, the connection relationship between path segments, and the motion requirements of each segment, and then sends motion control commands to the robotic arm to control it to move smoothly according to the preset direction and rhythm of the collision-free approach path. During the movement, the system obtains the current pose information of the robotic arm in real time and compares it with the preset path of the collision-free approach path, and adjusts the movement speed and attitude angle of the robotic arm in a timely manner to ensure that the robotic arm always moves along the collision-free approach path without deviation, until the robotic arm moves to the location of the target object, completing the movement operation towards the target object along the collision-free approach path.

[0140] After the robotic arm moves to the location of the target object, the pose compensation module controls the micro-tactile sensor array integrated in the fingertip of the robotic arm to start working. The micro-tactile sensor array makes full contact with the surface of the target object. During the contact process, each tactile sensing unit in the sensor array senses the force exerted by the target object on the robotic fingertip in real time, captures the contact force signal generated when the target object and the fingertip come into contact, and senses the vibration signal generated by the surface texture and hardness differences of the target object and the slight movements of the robotic arm during the contact process. The captured contact force signal and vibration signal are collected and summarized synchronously, and invalid interference signals generated during the collection process are eliminated to ensure that the collected signals are true and complete, and finally the multimodal micro-tactile signal of the target object is obtained.

[0141] The acquired manipulator's operational viewpoint images and the target object's multimodal micro-tactile signals were retrieved separately. The acquisition sequence of the operational viewpoint images was analyzed to determine the acquisition time corresponding to each frame of the operational viewpoint image. At the same time, the acquisition sequence of the multimodal micro-tactile signals was analyzed to determine the acquisition time corresponding to each segment of contact force signal and vibration signal. Then, based on the manipulator's motion sequence, the operational viewpoint images and multimodal micro-tactile signals were matched according to the same acquisition time, so that the operational viewpoint images and multimodal micro-tactile signals acquired at the same time form a one-to-one correspondence, completing the timing alignment operation between the two. After alignment, the correlated images and signal data were integrated to finally obtain the manipulator's visual-tactile perception data.

[0142] The system retrieves the acquired images from the robot's operating perspective and processes them to reduce background interference and highlight the target object. Then, it performs a full scan of the processed images, identifying the boundaries between the target object and the background area point by point. All pixels at these boundaries are captured and connected sequentially to form continuous lines. These lines accurately delineate the overall shape of the target object, including its edge direction, curvature, and overall contour. Simultaneously, it removes noise and breaks caused by image interference to ensure clear and continuous contour lines. Finally, the system extracts the visual contour features of the target object from the operating perspective images.

[0143] Multimodal microtactile signals of the target object are acquired, and these signals are then sorted and filtered to remove invalid interference signals generated during the acquisition process, retaining only the true and valid contact force and vibration signals. Subsequently, by analyzing the distribution of the contact force signals, all contact points when the mechanical fingertip contacts the surface of the target object are identified, and the specific distribution pattern of each contact point is determined. At the same time, by analyzing the variation pattern of the vibration signals, the unevenness of the contact points on the surface of the target object is perceived, and the local morphological features of the target object surface are captured. By integrating the distribution of contact points and local morphological features, the tactile geometric features of the target object are finally extracted from the multimodal microtactile signals.

[0144] Visual contour features and tactile geometric features of the target object are extracted separately. Using the three-dimensional space of the robot's working area as a reference, the overall shape of the target object reflected by the visual contour features is fused with the distribution of contact points and local geometric shapes on the surface of the target object reflected by the tactile geometric features. The visual contour features are used as the overall framework, and the tactile geometric features are filled into the corresponding contour areas to compensate for the inability of the visual contour features to reflect the local details of the target object's surface. At the same time, the deviation of the visual contour features is calibrated by the tactile geometric features to ensure that the fused features can comprehensively and accurately reflect the overall contour and local details of the target object. After the complete fusion process, the spatial reconstruction of the target object is completed, and the fused feature map of the robot is finally obtained.

[0145] The system retrieves the acquired pose information of the robotic arm and the generated fused feature map. By analyzing the fused feature map, the precise spatial shape, contour dimensions, and surface details of the target object are determined. The actual spatial state of the target object reflected in the fused feature map is compared and analyzed with the current pose information of the robotic arm to identify the deviation between the current pose information and the actual spatial state of the target object, i.e., the residual. Then, based on the identified residual, the pose information of the robotic arm is adjusted in a targeted manner to correct the deviation in the pose information, ensuring that the adjusted pose information can accurately match the actual spatial state of the target object and meet the accuracy requirements of the robotic arm's grasping operation. After a complete residual compensation operation, the precise grasping pose of the robotic arm is finally obtained.

[0146] The generated fused feature map of the robotic arm is retrieved and thoroughly scanned and analyzed. The focus is on identifying representative points in the fused feature map that reflect the details of the target object's local structure. These points accurately represent the surface undulations, edge transitions, and other local features of the target object. During the scanning process, these representative local feature points are marked one by one, clarifying the specific location of each local feature point in the fused feature map and the corresponding local structure of the target object. All marked local feature points are summarized and organized to form a complete set of feature points that can reflect the actual local structure of the target object. Finally, the measured feature point set of the target object is extracted.

[0147] The process involves acquiring existing pose information of the robotic arm, analyzing the current spatial position and attitude angle reflected in this pose information, and combining the relative positional relationship between the target object and the robotic arm contained in the pose information to determine the spatial morphology and structural distribution of the target object in the theoretical state corresponding to this pose information. Subsequently, based on the feature point selection criteria of the measured feature point set, points corresponding one-to-one with each feature point in the measured feature point set are selected on the theoretical spatial morphology of the target object. The selection positions and feature types of these theoretical points are completely consistent with those of the measured feature points, and can reflect the local structural features of the target object in the theoretical pose. All selected theoretical points are summarized to finally determine the theoretical feature point set of the target object corresponding to the measured feature point set.

[0148] The constructed set of measured feature points is compared one-to-one with the determined set of theoretical feature points. For each measured feature point, its corresponding theoretical feature point is found, and the difference between the two in spatial position is analyzed. The positional offset direction and degree between each corresponding feature point pair are clarified. The positional difference of each corresponding feature point pair is transformed into a residual vector that can reflect the offset state. Each residual vector corresponds to a set of positional deviations between measured and theoretical feature points. Then, all the obtained residual vectors are sorted in order to form a complete set of vectors that can reflect the positional deviations of all feature points. Finally, the residual vector set of the robot is obtained by differential analysis between the set of measured feature points and the set of theoretical feature points.

[0149] The obtained residual vector set of the robot is comprehensively analyzed to clarify the importance of the feature points corresponding to each residual vector. Residual vectors that can significantly affect the robot's grasping accuracy and reflect the key structural deviations of the target object are assigned higher weights, while residual vectors with less impact and corresponding to non-critical feature points are assigned lower weights. After assigning weights, each residual vector is multiplied by its corresponding weight to obtain the weighted value of each residual vector. Then, the weighted values ​​of all residual vectors are summed and divided by the sum of all weights to complete the weighted average calculation. Finally, the comprehensive residual vector of the robot that can comprehensively reflect the positional deviations of all feature points is obtained.

[0150] Based on the obtained comprehensive residual vector, the overall offset direction and degree of the robot's pose reflected by the comprehensive residual vector are analyzed to clarify the comprehensive deviation between the current pose information of the robot and the actual position and posture of the target object. Then, based on this comprehensive deviation, the pose information of the robot is optimized and adjusted in a targeted manner to correct the offset part of the pose information that does not match the actual situation. During the adjustment process, it is ensured that the corrected pose information can accurately match the actual spatial state of the target object, eliminate the grasping error caused by the position deviation of all feature points, and enable the pose of the robot to meet the requirements of accurately grasping the target object. After a complete optimization and compensation operation, the precise grasping pose of the robot is finally obtained.

[0151] The data source for the comprehensive residual vector is the result of weighted averaging of the residual vectors in the residual vector group. Its value is obtained by calculating each parameter in the formula and is the core parameter reflecting the overall deviation between the robot's pose information and the actual situation.

[0152] The data source for the number of feature points in the measured feature point set is the local feature points of the target object extracted based on the fused feature map. Specifically, the pose compensation module parses the fused feature map, extracts local feature points with obvious recognizability on the surface of the target object, integrates them into the measured feature point set, and then counts all feature points in the measured feature point set to obtain the specific value. This value is completely consistent with the total number of feature points contained in the measured feature point set and can accurately reflect the total number of feature points involved in the calculation.

[0153] No. The data source for the preset weight coefficients of each feature point is a preset weight allocation rule. This rule is preset based on the importance of each feature point in the target object grasping process. Before performing weighted averaging on the residual vector, the pose compensation module assigns a corresponding weight coefficient to each feature point in the measured feature point set according to this preset rule. Feature points corresponding to key grasping positions are assigned higher weight coefficients, while feature points corresponding to non-key grasping positions are assigned lower weight coefficients. Once the weight coefficient corresponding to each feature point is set, it remains fixed during this calculation process.

[0154] The first in the set of measured feature points The data source for the position coordinates of the measured feature point is the local feature points of the target object extracted based on the fused feature map. When the pose compensation module parses the fused feature map to extract local feature points and constructs the set of measured feature points, it simultaneously records the spatial position of each measured feature point in the fused feature map and converts this spatial position into the corresponding position coordinates, which is the position coordinates of the measured feature point set. Each measured feature point has a unique set of coordinates, accurately corresponding to the spatial position of the actual surface of the target object.

[0155] The first theoretical feature point set The data source for the position coordinates of the theoretical feature points is the theoretical pose of the target object determined based on the pose information, combined with the feature point set extracted from the preset 3D model of the target object. When determining the theoretical feature point set, the pose compensation module determines the theoretical pose of the target object based on the pose information, and then finds the feature point in the preset 3D model of the target object that matches the measured feature point set. For each measured feature point, a theoretical feature point is correspondingly extracted. Simultaneously, the spatial position of this theoretical feature point in the 3D model is extracted and converted into its corresponding position coordinates, which is the theoretical feature point set. The position coordinates of the theoretical feature point and the measured feature point set The position coordinates of each measured feature point correspond one-to-one.

[0156] The significance of this formula lies in calculating a comprehensive residual vector that fully reflects the robot's pose deviation. Specifically, it involves weighting the positional deviations of each measured feature point and its corresponding theoretical feature point, then averaging the results to integrate the positional deviation information of all feature points. This eliminates the influence of individual feature point deviations on the overall pose judgment, resulting in parameters that comprehensively and accurately reflect the overall offset between the robot's current pose and the actual state of the target object. This provides the core calculation basis for the subsequent pose compensation module to optimize and compensate the robot's pose information based on the comprehensive residual vector, thereby obtaining a precise grasping pose. This ensures the accuracy and rationality of pose compensation and meets the requirements for the robot to accurately grasp the target object.

[0157] The beneficial effects are that it controls the robotic arm to move along a collision-free approach path, ensuring that it accurately and safely reaches the target object, avoiding collisions with obstacles during movement, and laying a stable positional foundation for subsequent sensing data collection.

[0158] By collecting contact force and vibration signals through a micro-tactile sensor array to form multimodal micro-tactile signals, the shortcomings of single visual perception in capturing the physical features of the target object's surface are overcome, thus enriching the robot's perception dimensions of the target object.

[0159] By aligning the operational viewpoint image with multimodal micro-tactile signals in a timely manner, visual-tactile perception data is obtained, which realizes the effective fusion of visual and tactile cross-modal perception data. This allows the two types of data to be accurately matched in the time dimension, providing comprehensive, accurate and synchronous perception data support for subsequent pose error compensation, and improving the robot's perception accuracy and completeness of target objects.

[0160] Visual contour features and tactile geometric features are extracted from the operational perspective image and multimodal micro-tactile signals, respectively, to achieve accurate extraction of visual and tactile features. This makes up for the deficiency that single visual perception cannot capture the local physical features of the target object's surface and enriches the feature perception dimensions of the target object.

[0161] By spatially reconstructing visual contour features and tactile geometric features, a fused feature map is obtained, which integrates the overall shape contour of the target object with the local geometric details of the surface. At the same time, the deviation of the visual contour features is calibrated by tactile geometric features, so that the feature map can comprehensively and accurately reflect the actual spatial form of the target object.

[0162] By performing residual compensation on the original pose information based on the fused feature map, the robot can accurately identify and correct pose errors, effectively eliminate the cumulative error of pose information, significantly improve the positioning accuracy of the robot's grasping pose, and enable the robot to accurately fit the grasping position of the target object, laying a reliable pose foundation for subsequent accurate grasping.

[0163] The entire feature extraction, fusion, and compensation process enables deep fusion of cross-modal perception data at the feature layer, providing a more comprehensive and accurate basis for pose compensation. This significantly improves the pose control accuracy of the robotic arm in complex chemical emergency scenarios, meeting the precision operation requirements of chemical emergency response.

[0164] Local feature points are extracted from the fused feature map to construct a set of measured feature points. Combined with pose information, the corresponding theoretical feature point set is determined, providing a precise feature point reference for pose error analysis and enabling a refined comparison between the actual and theoretical poses of the target object.

[0165] By performing differential analysis on the measured and theoretical feature point sets, a residual vector group is obtained, which accurately quantifies the pose deviation corresponding to each feature point, realizing fine-grained decomposition of pose error and providing accurate deviation data for subsequent overall compensation.

[0166] The weighted average of the residual vector groups yields the comprehensive residual vector. By using weights to distinguish the importance of feature points to the capture, the bias of a single feature point can be avoided from affecting the overall judgment. This accurately integrates the overall bias of pose information, providing a scientific and unified basis for compensation.

[0167] By optimizing and compensating the original position and pose information based on the comprehensive residual vector, the robot arm can accurately and quantitatively correct the pose error, effectively eliminate the problem of pose error accumulation, and significantly improve the positioning accuracy of the grasping pose, allowing the robot arm to accurately fit the target object grasping position.

[0168] The entire residual compensation process enables precise solution and targeted correction of pose error, making pose compensation more scientific and accurate, significantly improving the pose control accuracy of the robot in complex chemical emergency scenarios, and providing reliable pose assurance for subsequent precise grasping.

[0169] The decision mapping module is used to perform decision mapping on the physical properties of the object surface perceived by the multimodal micro-tactile signals based on the precise grasping pose, so as to obtain the adaptive grasping command of the robot arm.

[0170] In this embodiment of the invention, when the decision mapping module performs decision mapping on the physical properties of the object surface sensed by the multimodal micro-tactile signals based on the precise grasping pose to obtain the adaptive grasping command of the robotic arm, it is specifically used for:

[0171] Based on the precise grasping posture, the expected contact area between the robotic arm and the target object is determined;

[0172] The contact force distribution information and surface roughness information related to the expected contact area are extracted from the multimodal micro-tactile signal to obtain the surface physical properties of the target object;

[0173] The surface physical properties are bidirectionally matched with the gripping parameters in the preset gripping strategy library to obtain the gripping force and gripping speed of the robotic arm.

[0174] The gripping force and the gripping speed are encoded into adaptive gripping commands for the robotic arm.

[0175] When the decision mapping module performs bidirectional matching between the surface physical properties and the grasping parameters in the preset grasping strategy library to obtain the grasping force and grasping speed of the robotic arm, it is specifically used for:

[0176] Based on the surface physical properties, candidate grasping force ranges and candidate grasping speed ranges from the preset grasping strategy library are selected.

[0177] An initial grasping force is selected from the candidate grasping force range to control the robotic arm to apply pre-contact pressure to the surface of the target object, and the rate of change of contact force in the multimodal microtactile signal is monitored simultaneously.

[0178] Based on the rate of change of the contact force, the deformation state of the target object under the initial grasping force is determined, and the deformation response characteristics of the target object are obtained.

[0179] Based on the deformation response characteristics, the initial grasping speed of the candidate grasping speed range is determined;

[0180] Based on the initial grasping force and the initial grasping speed, the robotic arm is controlled to perform a trial grasping action on the target object, and the slippage signal during the trial grasping process is collected in real time.

[0181] Based on the intensity of the slip signal, the initial gripping force and the initial gripping speed are synchronously corrected to obtain the gripping force and gripping speed of the robotic arm.

[0182] The precise grasping pose of the robotic arm generated by the pose compensation module is obtained. The relative spatial position, attitude angle, and contact posture of the robotic arm end effector and the target object reflected by the precise grasping pose are analyzed to determine the specific area in which the robotic arm end effector fingertip can contact the target object. Combined with the shape and size of the robotic arm end effector fingertip and the grasping posture, the area in which the robotic arm will inevitably contact the target object when performing the grasping action is locked. This area can ensure that the robotic arm can stably contact the target object and complete the grasping action. Finally, the expected contact area between the robotic arm and the target object is determined based on the precise grasping pose.

[0183] The multimodal micro-tactile signals of the target object are retrieved and combined with the determined expected contact area. The multimodal micro-tactile signals are then filtered to remove contact force signals and vibration signals that are irrelevant to the expected contact area, retaining only the signal data related to the expected contact area. By analyzing the filtered contact force signals, the magnitude and distribution of contact force at each contact point within the expected contact area are determined, and the contact force distribution information of the expected contact area is integrated. At the same time, by analyzing the filtered vibration signals, the smoothness of the target object surface within the expected contact area is perceived. The surface undulation and texture density are judged based on the variation law of the vibration signals, forming the surface roughness information of the expected contact area. The contact force distribution information and surface roughness information are integrated to finally obtain the surface physical properties of the target object.

[0184] The system invokes a pre-defined grasping strategy library, which stores the matching relationships between different surface physical properties and corresponding grasping parameters. Each surface physical property corresponds to a unique and suitable grasping parameter. The system then compares the obtained surface physical properties of the target object with all surface physical property types in the grasping strategy library to find a match that is completely consistent with the surface physical properties of the target object. The grasping parameter corresponding to this match is extracted. At the same time, the extracted grasping parameter is compared with the surface physical properties of the target object to confirm that the grasping parameter can adapt to the contact force distribution information and surface roughness information of the target object, ensuring the compatibility between the grasping parameter and the surface physical properties. After confirming that the two-way comparison is correct, the corresponding grasping force and grasping speed are separated from the grasping parameters, and finally the grasping force and grasping speed of the robot arm are obtained.

[0185] The gripping force and gripping speed obtained by the robotic arm are encoded. According to the instruction encoding rules that the robotic arm control system can recognize and execute, the specific state corresponding to the gripping force is converted into an coded signal that the control system can recognize. At the same time, the specific state corresponding to the gripping speed is also converted into a corresponding coded signal to ensure that the encoded signal can accurately reflect the specific requirements of gripping force and gripping speed. Then, the two coded signals are integrated and arranged according to the preset instruction format to form a complete instruction signal that can be directly executed by the robotic arm. Finally, the gripping force and gripping speed are encoded into the robotic arm's adaptive gripping instruction.

[0186] The system acquires the surface physical properties of the extracted target object, which include contact force distribution information and surface roughness information of the expected contact area. Then, it calls a preset grasping strategy library, which pre-stores the correspondence between different surface physical properties and corresponding candidate grasping force ranges and candidate grasping speed ranges. The obtained surface physical properties of the target object are compared one by one with all the preset surface physical properties in the grasping strategy library. Candidate grasping force ranges and candidate grasping speed ranges that are completely compatible with the surface physical properties of the target object are selected. During the selection process, it is ensured that the candidate grasping force range can adapt to the contact force distribution information of the target object, and the candidate grasping speed range can adapt to the surface roughness information of the target object. Finally, the candidate grasping force ranges and candidate grasping speed ranges in the preset grasping strategy library are selected.

[0187] An initial gripping force suitable for the physical properties of the target object's surface is selected from the candidate gripping force range obtained through screening. During the selection process, the surface roughness information and contact force distribution information of the target object are combined to ensure that the initial gripping force is not too large to cause damage to the target object, nor too small to fail to form effective pre-contact. After the selection is completed, a control command is sent to the robot arm to control the robot arm to apply pre-contact pressure to the surface of the target object according to the initial gripping force, so that the fingertip of the robot arm makes light contact with the expected contact area of ​​the target object. At the same time, real-time monitoring of multimodal micro-tactile signals is started simultaneously to continuously capture the changes in contact force in the multimodal micro-tactile signals during the pre-contact process. By analyzing the change law of contact force over time, the rate of change of contact force in the multimodal micro-tactile signals is obtained.

[0188] The rate of change of contact force is obtained from monitoring. Combined with the surface physical properties of the target object, the rate of change of contact force is comprehensively analyzed to determine whether the target object deforms under the pre-contact pressure applied by the initial gripping force, and the specific nature of the deformation. If the rate of change of contact force remains stable and within a preset range, it indicates that the target object has not undergone significant deformation. If the rate of change of contact force fluctuates significantly or changes continuously, it indicates that the target object has deformed. At the same time, based on the magnitude and trend of the change of contact force, the degree and speed of the target object's deformation are determined. By integrating this deformation-related information, the deformation response characteristics of the target object are finally obtained.

[0189] The deformation response characteristics of the target object are obtained, and combined with the previously selected candidate grasping speed ranges, the candidate grasping speed ranges are further filtered and determined. If the deformation response characteristics indicate that the target object is not easily deformed, a relatively fast speed is selected from the candidate grasping speed range as the initial grasping speed to ensure grasping efficiency. If the deformation response characteristics indicate that the target object is easily deformed, a relatively slow speed is selected from the candidate grasping speed range as the initial grasping speed to avoid damage to the target object due to excessive speed. Throughout the selection process, the initial grasping speed is always closely matched with the deformation response characteristics of the target object to ensure that the initial grasping speed is completely adapted to the deformation response characteristics. Finally, the initial grasping speed of the candidate grasping speed range is determined based on the deformation response characteristics.

[0190] The selected initial grasping force and determined initial grasping speed are used as trial grasping parameters. Trial grasping control commands are sent to the robotic arm to control the robotic arm to perform trial grasping actions on the target object according to the initial grasping force and initial grasping speed. During the trial grasping action, the posture of the robotic arm is kept consistent with the precise grasping posture to ensure stable contact between the robotic arm's fingertip and the expected contact area of ​​the target object. At the same time, real-time acquisition of sliding signals is initiated. Through the miniature tactile sensor array integrated into the robotic arm's fingertip, it continuously captures whether relative sliding occurs between the target object and the robotic fingertip during the trial grasping process. The sliding signals generated during the sliding process are collected in real time to ensure that the collected sliding signals can accurately reflect the sliding state during the trial grasping.

[0191] The intensity of the real-time acquired slip signal is analyzed to clarify the degree of sliding between the target object and the robotic fingertip, as reflected by the slip signal intensity. If the slip signal intensity is large, it indicates that the initial gripping force is insufficient or the initial gripping speed is too fast, causing significant relative sliding between the target object and the robotic fingertip. In this case, the initial gripping force is increased and the initial gripping speed is decreased. If the slip signal intensity is small, it indicates that the initial gripping force and initial gripping speed are basically matched, requiring only minor adjustments. If no slip signal is detected, it indicates that the initial gripping force may be too large or the initial gripping speed may be too slow. In this case, the initial gripping force is decreased and the initial gripping speed is increased. During the adjustment process, the change in the slip signal intensity is monitored simultaneously until the slip signal intensity is within a reasonable range. Finally, the initial gripping force and initial gripping speed are synchronously corrected to obtain the gripping force and gripping speed of the robotic arm.

[0192] The beneficial effects are that by determining the expected contact area based on the precise grasping posture, the extraction of surface physical properties becomes more targeted, accurately focusing on the contact area actually grasped by the robotic arm, avoiding interference from invalid information, and improving the accuracy of physical property extraction.

[0193] By extracting the contact force distribution and surface roughness information of the expected contact area from multimodal micro-tactile signals, the core physical properties of the grasping part of the target object can be accurately obtained, providing real and effective data basis for grasping strategy matching.

[0194] It performs bidirectional matching with a preset grasping strategy library to achieve precise adaptation between surface physical properties and grasping force and speed, eliminating the drawbacks of traditional fixed grasping parameters, making the grasping parameters fit the actual characteristics of the target object, and improving the rationality of the grasping strategy.

[0195] The matched grasping force and speed are encoded into adaptive grasping instructions, which can be directly recognized and executed by the robot's execution terminal. This enables the efficient conversion of grasping strategies into control instructions, providing standardized and adaptable control basis for precise grasping by the robot, and improving the accuracy and adaptability of grasping actions.

[0196] By filtering candidate grasping force and speed ranges based on surface physical properties, the grasping parameters are matched within a clear range, avoiding blind setting, improving the initial rationality of parameter matching, and adapting to the basic physical characteristics of the target object.

[0197] By monitoring the change rate of contact force through pre-contact pressure, deformation response characteristics can be identified, and the deformation characteristics of the target object can be accurately perceived. This provides a basis for determining the grasping speed that is consistent with the actual situation of the object, and avoids damage to the object due to deformation caused by improper speed.

[0198] By combining initial force and speed to perform trial grasping and collect sliding signals, and using actual grasping feedback as the basis for adjustment, parameter correction becomes more practical and effectively avoids the deviation between theoretical matching and actual grasping.

[0199] By synchronously correcting the gripping force and speed based on the slip signal intensity, the gripping parameters are dynamically optimized, accurately adapting to target objects with different surface characteristics and deformation characteristics, avoiding gripping slippage or object damage, and significantly improving the matching accuracy of gripping parameters.

[0200] The entire two-way matching process realizes closed-loop optimization of grasping parameters from initial screening to dynamic correction, abandoning the traditional fixed parameter mode, and making the grasping force and speed highly compatible with the actual situation of the target object, significantly improving the stability and success rate of the robotic arm's grasping action, and adapting to the precise grasping needs of chemical emergency response.

[0201] The grasping update module is used to control the robotic arm to grasp objects using the adaptive grasping command, and to update the environmental state information according to the grasping state.

[0202] In this embodiment of the invention, when the grasping update module executes the adaptive grasping command to control the robotic arm to grasp, and updates the environmental state information according to the grasping state, it is specifically used for:

[0203] The adaptive grasping command is sent to the execution terminal of the robotic arm to control the robotic arm to grasp the target object;

[0204] During the grasping operation, the contact state between the robotic arm and the target object, as well as the positional changes of the target object, are monitored in real time to obtain the grasping state information of the robotic arm.

[0205] Based on the grasping status information, a judgment is made regarding the grasping of the target object;

[0206] When the capture is successful, the target object is removed from its original position in the environmental state information;

[0207] If the capture fails, the position information of the target object in the environmental state information is updated according to the current position of the target object.

[0208] The grasping update module acquires the adaptive grasping instructions of the robotic arm generated by the decision mapping module. These instructions contain encoded information related to the grasping force and speed of the robotic arm. A communication connection is then established with the robotic arm's execution terminal, and the complete adaptive grasping instructions are sent to the execution terminal. Upon receiving the instructions, the execution terminal decodes them to determine the specific requirements for grasping force and speed. It then sends corresponding control signals to each joint of the robotic arm to maintain a precise grasping posture. Following the grasping force and speed specified in the adaptive grasping instructions, the terminal fingertip moves towards the target object, making stable contact with the expected contact area and applying the corresponding force to complete the grasping operation.

[0209] Throughout the entire grasping operation, the grasping update module activates a real-time monitoring mechanism. Through a miniature tactile sensor array integrated into the fingertips of the robotic arm, it continuously collects contact signals between the robotic arm and the target object, determining whether the contact is stable and whether the contact force remains within the range specified by the adaptive grasping command. This monitors the contact state between the robotic arm and the target object. Simultaneously, a vision sensor captures the image of the target object in real time. Combined with the original position of the target object in the environmental information, the module compares and analyzes the positional changes of the target object in the image to determine whether the target object has shifted, and the direction and magnitude of the shift. The monitored contact state information and the target object positional change information are integrated to finally obtain the grasping state information of the robotic arm.

[0210] The grasping update module comprehensively analyzes the grasping status information obtained by the robotic arm, focusing on the contact state and changes in the target object's position contained in the grasping status information. It determines whether the robotic arm's grasp of the target object has reached a stable state. If the contact state is stable, the contact force is maintained within the specified range, and the target object can move with the movement of the robotic arm without slipping or falling off, the target object is judged to have been grasped successfully. If the contact state is unstable, the contact force deviates from the specified range, or the target object does not move with the robotic arm, or there is obvious slippage or even falling off, causing an unexpected change in the target object's position, the target object is judged to have failed to be grasped. Through this targeted analysis, the judgment of the target object grasping status is completed.

[0211] When the grasping status information indicates successful grasping, the grasping update module retrieves the existing environmental status information of the robotic arm. This environmental status information includes a 3D point cloud distribution map of the work area. It analyzes the original position information of the target object in the 3D point cloud distribution map, clarifies the spatial coordinates and contour range of the target object's original position in the 3D point cloud distribution map, and then modifies the 3D point cloud distribution map to completely remove the point cloud data corresponding to the original position of the target object from the 3D point cloud distribution map. This ensures that the 3D point cloud distribution map can accurately reflect the distribution status of the remaining objects in the work area after removal, thus completing the operation of removing the target object from its original position in the environmental status information.

[0212] When the grasping status information indicates a grasping failure, the grasping update module stops the robotic arm's grasping operation. Using a visual sensor and a micro-tactile sensor array, it collects the target object's current position information in real time, clarifying the target object's specific spatial location, attitude angle, and relative positional relationship with other objects in the work area after the grasping failure. Then, it retrieves existing environmental status information, finds the original position information of the target object recorded in the environmental status information, replaces the original position information with the collected current position information of the target object, and simultaneously updates the point cloud data of the corresponding area in the 3D point cloud distribution map of the environmental status information. This ensures that the environmental status information accurately reflects the target object's current actual position, completing the update of the target object's position information in the environmental status information.

[0213] The beneficial effect is that the adaptive grasping command is directly sent to the execution terminal, realizing precise control of the grasping action, allowing the robot to complete the grasping strictly according to the force and speed adapted to the target object's properties, thereby improving the execution accuracy of the grasping operation.

[0214] During the grasping process, the system monitors the contact state and changes in the target object's position in real time and generates grasping status information, providing accurate and comprehensive real-time data for determining the success or failure of the grasp and ensuring the accuracy of the judgment results.

[0215] Accurately determine the success or failure of the capture based on the capture status information, achieve efficient identification of the capture results, and provide a clear basis for targeted updates of subsequent environmental status information.

[0216] When the capture is successful, the original position information of the target object is removed; when it fails, its current position is updated, so as to realize the dynamic real-time update of environmental status information and ensure that the environmental data always keeps up with the actual changes in the chemical emergency response scenario.

[0217] Dynamically updated environmental status information can provide accurate environmental data support for subsequent operations such as secondary grasping and path replanning of the robotic arm, adapt to the dynamic operation requirements of chemical emergency scenarios, improve the continuous operation capability and operation intelligence level of the robotic arm, and avoid operational errors caused by lag in environmental data.

[0218] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0219] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A cross-modal perception and control system for a robotic arm for chemical emergency response, characterized in that, The system includes an environment scanning module, a visual positioning module, an obstacle avoidance planning module, a pose compensation module, a decision mapping module, and a capture and update module, wherein: The environmental scanning module is used to perform a three-dimensional environmental pre-scan of the working area of ​​the robot to obtain the environmental status information of the robot. The visual positioning module is used to acquire the operation view image of the robot based on the environmental state information, and to calculate the visual features of the operation view image to obtain the pose information of the robot. The obstacle avoidance planning module is used to correct the motion trajectory of the robotic arm based on the pose information and the environmental state information, so as to obtain the collision-free approach path of the robotic arm. The pose compensation module is used to fuse the operation view image and the multimodal micro-tactile signals collected by the robot hand based on the collision-free approach path, so as to perform error compensation on the pose information and obtain the precise grasping pose of the robot hand. The decision mapping module is used to perform decision mapping on the physical properties of the object surface sensed by the multimodal micro-tactile signals based on the precise grasping pose, to obtain the adaptive grasping instructions of the robotic arm, including: Based on the precise grasping pose, the expected contact area between the robotic arm and the target object is determined; The contact force distribution information and surface roughness information related to the expected contact area are extracted from the multimodal micro-tactile signal to obtain the surface physical properties of the target object; The surface physical properties are bidirectionally matched with gripping parameters in a preset gripping strategy library to obtain the gripping force and gripping speed of the robotic arm, including: Based on the surface physical properties, candidate grasping force ranges and candidate grasping speed ranges from the preset grasping strategy library are selected. An initial grasping force is selected from the candidate grasping force range to control the robotic arm to apply pre-contact pressure to the surface of the target object, and the rate of change of contact force in the multimodal microtactile signal is monitored simultaneously. Based on the rate of change of the contact force, the deformation state of the target object under the initial grasping force is determined, and the deformation response characteristics of the target object are obtained. Based on the deformation response characteristics, the initial grasping speed of the candidate grasping speed range is determined; Based on the initial grasping force and the initial grasping speed, the robotic arm is controlled to perform a trial grasping action on the target object, and the slippage signal during the trial grasping process is collected in real time. Based on the intensity of the slip signal, the initial gripping force and the initial gripping speed are synchronously corrected to obtain the gripping force and gripping speed of the robotic arm; The grasping force and the grasping speed are encoded into adaptive grasping commands for the robotic arm; The grasping and updating module is used to control the robotic arm to grasp using the adaptive grasping command, and to update the environmental status information according to the grasping status.

2. The cross-modal perception and control system for a robotic arm in chemical emergency response as described in claim 1, characterized in that, When the environmental scanning module performs a three-dimensional environmental pre-scan of the robot's working area to obtain the robot's environmental state information, it is specifically used for: The robot arm carries a 3D laser scanner to scan the robot arm's working area and obtain the original point cloud data of the working area. The original point cloud data is subjected to noise removal processing to obtain the denoised point cloud data of the working area; The denoised point cloud data is reconstructed in three dimensions to obtain a three-dimensional point cloud distribution map of the work area; The three-dimensional point cloud distribution map is used as the environmental state information of the robotic arm.

3. The cross-modal perception and control system for a robotic arm in chemical emergency response as described in claim 1, characterized in that, When the visual positioning module performs operations based on the environmental state information, acquires the operating view image of the robotic arm, and calculates the visual features of the operating view image to obtain the pose information of the robotic arm, it is specifically used for: Based on the environmental status information, the acquisition angle of the vision sensor mounted on the robot arm is adjusted so that the field of view of the vision sensor covers the working area, so as to acquire the operating perspective image of the robot arm; The image from the operating perspective is subjected to grayscale transformation, and the transformed image is binarized to obtain the binarized image of the manipulator. Edge detection is performed on the binarized image to obtain the contour feature image of the robotic arm; Based on the contour feature image, the spatial position and posture angle of the robot are calculated to obtain the pose information of the robot.

4. The cross-modal perception and control system for a robotic arm in chemical emergency response as described in claim 1, characterized in that, When the obstacle avoidance planning module performs obstacle avoidance correction on the robot's motion trajectory based on the pose information and the environmental state information to obtain a collision-free approach path for the robot, it is specifically used for: Based on the pose information, the current pose information of the robotic arm and the target grasping position are used to perform trajectory planning to obtain the initial motion trajectory of the robotic arm; Based on the distribution of obstacles in the environmental state information, the initial motion trajectory is segmented for detection to obtain the collision risk trajectory segment of the robotic arm; Based on the collision risk trajectory segment, select an avoidance midpoint outside the obstacle; Based on the avoidance midpoint, obstacle avoidance correction is performed on the collision risk trajectory segment to obtain the corrected trajectory segment of the robotic arm; By sequentially connecting the non-collision risk segment in the initial motion trajectory with the corrected trajectory segment, a collision-free approach path for the robotic arm is obtained.

5. The cross-modal perception and control system for a robotic arm in chemical emergency response as described in claim 1, characterized in that, When the pose compensation module executes the operation view image and the multimodal micro-tactile signals collected by the robotic arm based on the collision-free approach path, it is specifically used for: Control the robotic arm to move along the collision-free approach path to the target object; The miniature tactile sensor array integrated into the fingertips of the robotic arm collects the contact force signal and vibration signal of the target object, thereby obtaining the multimodal micro-tactile signal of the target object; The visual-tactile perception data of the robotic arm is obtained by aligning the operation view image with the multimodal micro-tactile signal in time.

6. The cross-modal perception and control system for a robotic arm in chemical emergency response as described in claim 5, characterized in that, When the pose compensation module performs error compensation on the pose information to obtain the precise grasping pose of the robotic arm, it is specifically used for: Extract the edge contour of the target object from the operational viewpoint image to obtain the visual contour features of the target object; The contact point distribution and local geometric features of the target object surface are extracted from the multimodal micro-tactile signals to obtain the tactile geometric features of the target object; The visual contour features and the tactile geometric features are spatially reconstructed to obtain the fused feature map of the robotic hand; Based on the fused feature map, residual compensation is performed on the pose information to obtain the precise grasping pose of the robotic arm.

7. A cross-modal perception and control system for a robotic arm in chemical emergency response as described in claim 6, characterized in that, When the pose compensation module performs residual compensation on the pose information based on the fused feature map to obtain the precise grasping pose of the robotic arm, it is specifically used for: Based on the fused feature map, local feature points of the target object are extracted to construct the measured feature point set of the target object; Based on the pose information, determine the theoretical feature point set corresponding to the measured feature point set of the target object in the theoretical pose. The measured feature point set and the theoretical feature point set are analyzed by difference to obtain the residual vector set of the manipulator; The weighted average of the residual vectors in the residual vector group is used to obtain the comprehensive residual vector of the robot arm. The formula for calculating the comprehensive residual vector is as follows: ； In the formula, For the composite residual vector, The number of feature points in the measured feature point set. For the first Preset weight coefficients for each feature point For the set of measured feature points, the first... The position coordinates of each measured feature point For the set of theoretical feature points, the first The position coordinates of the theoretical feature points; Based on the comprehensive residual vector, the pose information is optimized and compensated to obtain the precise grasping pose of the robotic arm.

8. The cross-modal perception and control system for a robotic arm in chemical emergency response as described in claim 1, characterized in that, When the grasping and updating module executes the adaptive grasping command to control the robotic arm to grasp, and updates the environmental state information according to the grasping state, it is specifically used for: The adaptive grasping command is sent to the execution terminal of the robotic arm to control the robotic arm to perform a grasping operation on the target object; During the grasping operation, the contact state between the robotic arm and the target object, as well as the positional changes of the target object, are monitored in real time to obtain the grasping state information of the robotic arm. Based on the grasping status information, a judgment is made regarding the grasping of the target object; When the capture is successful, the target object is removed from its original position in the environmental state information; If the capture fails, the position information of the target object in the environmental state information is updated according to the current position of the target object.

Images