Chess interaction device adaptive grabbing control method based on multi-modal perception

Abstract

The present application relates to the technical field of electric digital data processing, in particular to a chess interactive device adaptive grabbing control method based on multi-modal perception, comprising: acquiring multi-modal information of target chess pieces and adjacent chess pieces, establishing spatial relationship between the target chess pieces and the adjacent chess pieces, determining the grabbing posture and the entering path of the end effector; controlling the end effector to approach the target chess pieces along the entering path, dynamically adjusting the motion state and the grabbing parameters according to the real-time sensing results; grabbing according to the grabbing parameters, and moving the target chess pieces out. The present application fuses two-dimensional images and three-dimensional point cloud information to establish the three-dimensional spatial relationship of the target chess pieces and the adjacent chess pieces, and adaptively determines the grabbing posture and the entering path in combination with the structural features of the end effector; in the approaching and grabbing process, the motion state, the grabbing posture and the clamping parameters are dynamically adjusted according to the real-time sensing results, the collision and the false grabbing risk are reduced, and the grabbing success rate, the stability and the safety in the dense chess environment are improved.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and specifically to an adaptive grasping control method for a chess-like interactive device based on multimodal perception. Background Technology

[0002] With the development of intelligent robots, machine vision, and intelligent control technologies, robots are increasingly being applied to scenarios such as chess games, chess game reproduction, and human-computer interaction. In chess-playing scenarios, robots typically need to identify, grasp, and move target pieces on the chessboard using an end effector. Existing piece-grabbing methods usually determine the grasping position based on two-dimensional visual positioning or pre-set piece coordinates, then control the robotic arm to approach the target piece along a fixed trajectory and complete the grasping process using a gripping mechanism.

[0003] However, in actual chess-playing interactions, pieces are typically densely distributed on the board surface, with small gaps between adjacent pieces, and variations in their outlines, sizes, and postures are possible. When a target piece is situated within a narrow space surrounded by multiple pieces, determining the grasping position solely based on its two-dimensional position is insufficient to accurately reflect the three-dimensional spatial relationship between the target piece and its surrounding pieces. The end effector is prone to collisions or interference with adjacent pieces as it approaches the target piece, leading to grasping deviations, collisions, or even misgrabbing. Furthermore, existing methods typically employ fixed grasping postures, clamping openings, and clamping forces after the end effector enters the target grasping area. This makes it difficult to adaptively adjust based on the target piece's actual position, posture, and dimensions, easily resulting in inaccurate clamping positions, insufficient clamping force causing piece slippage, or excessive clamping force causing piece deviation. Summary of the Invention

[0004] (a) Purpose of the invention The purpose of this invention is to provide an adaptive grasping control method for a chess-like interactive device based on multimodal perception. By fusing two-dimensional images and three-dimensional point cloud information, a three-dimensional spatial relationship between the target chess piece and adjacent chess pieces is established, and the grasping posture and entry path are adaptively determined by combining the structural features of the end effector. During the approach and grasping process, the motion state, grasping posture and clamping parameters are dynamically adjusted according to the real-time perception results to reduce the risk of collision and misgrabbing, and improve the grasping success rate, stability and security in dense chess game environments.

[0005] (II) Technical Solution To address the above problems, this invention provides an adaptive grasping control method for a chess-like interactive device based on multimodal perception, comprising: Acquire multimodal information of the target piece and its adjacent pieces, and establish spatial relationships between the target piece and its adjacent pieces. The spatial relationships include relative distance, relative orientation, spatial height difference, and outline gap. Based on the spatial relationship and the structural characteristics of the end effector, the grasping posture and entry path of the end effector are determined; The end effector is controlled to approach the target piece along the entry path, and its motion state is dynamically adjusted based on real-time sensing results. When the end effector reaches the grasping stage, the grasping parameters are adjusted based on the real-time sensing results. The target piece is moved out based on the specified capture parameters.

[0006] In another aspect of the present invention, preferably, the multimodal information includes two-dimensional image data and three-dimensional point cloud data; the step of acquiring multimodal information of the target piece and adjacent pieces, and establishing the spatial relationship between the target piece and adjacent pieces, includes: Acquire two-dimensional image data and three-dimensional point cloud data of the target piece and its adjacent pieces; The two-dimensional image data is identified to determine the outline regions of the target chess piece and each adjacent chess piece; Based on the coordinate transformation relationship between the two-dimensional image data and the three-dimensional point cloud data, the contour region is mapped to the three-dimensional point cloud region corresponding to the three-dimensional point cloud data to obtain the three-dimensional point cloud corresponding to the target chess piece and each adjacent chess piece. Spatial reconstruction is performed on the three-dimensional point cloud corresponding to the target chess piece and each adjacent chess piece to obtain the three-dimensional model of the target chess piece and the three-dimensional model of each adjacent chess piece. Based on the 3D model of the target piece and the 3D models of each adjacent piece, establish the spatial relationship between the target piece and each adjacent piece.

[0007] In another aspect of the present invention, preferably, establishing the spatial relationship between the target piece and each adjacent piece based on the three-dimensional model of the target piece and the three-dimensional models of each adjacent piece includes: The three-dimensional centroid of the target chess piece's three-dimensional model and the three-dimensional centroids of each adjacent chess piece's three-dimensional model are determined respectively, and a local three-dimensional coordinate system of the target chess piece is established with the three-dimensional centroid of the target chess piece as the reference point; The relative distance between the target piece and each of its adjacent pieces is calculated based on the coordinate difference between the three-dimensional centroid of the target piece and the three-dimensional centroid of each of its adjacent pieces in the local three-dimensional coordinate system of the target piece. Based on the projection direction of the coordinate difference onto the horizontal plane of the local three-dimensional coordinate system of the target piece, determine the relative orientation of each adjacent piece relative to the target piece; Based on the vertical coordinate difference between the target piece's three-dimensional centroid and the three-dimensional centroids of each adjacent piece, calculate the spatial height difference between the target piece and each adjacent piece; Multiple first surface points are obtained by sampling from the outer surface of the target chess piece's 3D model, and multiple second surface points are obtained by sampling from the outer surface of the corresponding adjacent chess piece's 3D model. The minimum surface distance between the target chess piece and the corresponding adjacent chess piece is determined based on the 3D Euclidean distance between each first surface point and each second surface point, and the minimum surface distance is used as the contour gap in the corresponding direction. Based on the relative distance, relative orientation, spatial height difference, and outline gap of each adjacent piece, the spatial relationship between the target piece and each adjacent piece is formed.

[0008] In another aspect of the present invention, preferably, the structural features of the end effector include the size of the clamping portion and the range of motion of the clamping portion; determining the grasping posture and entry path of the end effector based on the spatial relationship and the structural features of the end effector includes: Based on the relative position and outline gap between the target piece and each adjacent piece, determine the first candidate entry direction of the target piece in the circumferential direction; Based on the size of the clamping part and the range of motion of the clamping part of the end effector, a second candidate entry direction is obtained. The second candidate entry direction represents the first candidate entry direction that can accommodate the clamping part without interfering with adjacent pieces. Based on the contour gap and spatial height difference corresponding to each of the second candidate entry directions, determine the available grabbing space corresponding to each of the second candidate entry directions; Based on the degree of matching between the available grasping space and the structural dimensions of the end effector, the accessibility of each second candidate entry direction is evaluated to obtain the accessibility evaluation result; Based on the reachability evaluation results, the grasping posture and entry path of the end effector are determined.

[0009] In another aspect of the present invention, preferably, determining the grasping posture and entry path of the end effector based on the reachability evaluation result includes: The target entry direction is determined based on the reachability evaluation result, and the attitude of the gripping part of the end effector toward the target entry direction is determined as the grasping attitude; Using the target's direction of entry as a constraint, the target grabbing position corresponding to the grabbing posture is determined based on the three-dimensional model of the target piece; An entry path consisting of an approach segment and a grasping segment is generated based on the target grasping position, the target entry direction, and the current position of the end effector.

[0010] In another aspect of the present invention, preferably, the real-time sensing result includes the real-time distance between the end effector and each adjacent piece; controlling the end effector to approach the target piece along the entry path and dynamically adjust its movement state according to the real-time sensing result includes: During the approach segment movement of the end effector along the entry path, real-time image data and real-time depth data of the target piece and each adjacent piece are acquired at a preset sampling period. The real-time spatial relationships between the target chess piece and its adjacent chess pieces are determined based on the real-time image data and real-time depth data. The real-time spatial relationship is compared with the spatial relationship to obtain the comparison result; Based on the comparison results, the real-time distance between the end effector and each adjacent chess piece is calculated; The motion state of the end effector is dynamically adjusted based on the relationship between the real-time distance and the preset safe distance.

[0011] In another aspect of the present invention, preferably, the step of dynamically adjusting the motion state of the end effector based on the relationship between the real-time distance and the preset safety distance includes: The real-time distance between the end effector and each adjacent chess piece is compared with a first safe distance threshold and a second safe distance threshold, wherein the first safe distance threshold is greater than the second safe distance threshold; When the real-time distance is greater than or equal to the first safe distance threshold, the end effector is controlled to move at a preset approach speed according to the entry path; When the real-time distance is less than the first safe distance threshold and greater than or equal to the second safe distance threshold, reduce the approach speed of the end effector; When the real-time distance is less than the second safe distance threshold, the end effector is controlled to pause its movement, and the local obstacle avoidance direction is re-determined according to the real-time spatial relationship between the target piece and adjacent pieces. The end effector is controlled to correct its position along the local obstacle avoidance direction and move toward the target piece.

[0012] In another aspect of the present invention, preferably, adjusting the grasping parameters based on real-time sensing results when the end effector reaches the grasping stage includes: Based on the outer contour dimensions of the target chess piece and the structural dimensions of the clamping part of the end effector, the target clamping position and target opening amount of the clamping part relative to the target chess piece are determined, and the target clamping position and target opening amount are grasping parameters; Based on the target clamping position and the target opening amount, determine the real-time grabbing distance between each adjacent piece; Adjust the target clamping position and target opening amount based on the real-time grasping distance.

[0013] In another aspect of the present invention, preferably, adjusting the grasping parameters based on real-time sensing results when the end effector reaches the grasping stage further includes: Based on the adjusted target clamping position and target opening, the target chess piece is clamped; The contact force and relative displacement of the clamping part are obtained, and the contact state of the target chess piece is determined based on the contact force and the relative displacement. The target clamping force is adjusted according to the contact state, and the target clamping force is a gripping parameter.

[0014] In another aspect of the present invention, preferably, the step of grasping according to the grasping parameters and removing the target piece includes: The target chess piece is held according to the grasping parameters; Update the spatial relationships around the clamping part to generate a safe movement path; Move the target piece out according to the safe movement path.

[0015] (III) Beneficial Effects The above-described technical solution of the present invention has the following beneficial technical effects: This invention establishes a three-dimensional spatial relationship between the target piece and its neighboring pieces by fusing two-dimensional image data and three-dimensional point cloud data. This allows for accurate acquisition of relative distance, relative orientation, spatial height difference, and contour gap information, providing more accurate spatial constraints for the end effector's grasping posture and entry path planning. Furthermore, by combining the end effector's gripping part dimensions and range of motion, candidate entry directions are screened and their accessibility evaluated. Collision detection is used to determine a suitable entry path, reducing the risk of interference between the end effector and neighboring pieces in dense chessboard environments. Simultaneously, real-time images and depth information are continuously acquired as the end effector approaches the target piece. The movement speed and direction are dynamically adjusted based on the real-time distance between the end effector and neighboring pieces, improving dynamic obstacle avoidance capabilities in complex chessboard environments. During the grasping phase, grasping parameters are adaptively determined, enabling the grasping action to adapt to different piece states, improving gripping stability and grasping success rate, reducing collision and misgrabbing risks, and enhancing the grasping success rate, stability, and safety in dense chessboard environments. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of one embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0018] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0020] Example 1 An adaptive grasping control method for a chess-like interactive device based on multimodal perception is disclosed. This method is applicable to chess-like interactive scenarios where chess pieces are densely distributed on the board and multiple adjacent pieces surround the target piece, especially irregularly shaped pieces. The chess-like interactive device may include a visual perception module, a 3D depth perception module, a robotic arm, an end effector, and a control module. The visual perception module acquires 2D image information of the chessboard and pieces; the 3D depth perception module acquires depth information or 3D point cloud information of the chessboard area and pieces; the robotic arm drives the end effector to move, and the end effector grasps the target piece; the control module processes the perceived data and controls the robotic arm and end effector to perform corresponding actions. Figure 1 An overall flowchart of one embodiment of the present invention is shown, as follows: Figure 1 As shown, it includes: Acquire multimodal information of the target piece and its adjacent pieces, and establish spatial relationships between the target piece and its adjacent pieces. These spatial relationships include relative distance, relative orientation, spatial height difference, and contour gaps. The specific content of the multimodal information is not limited here. Optionally, in this embodiment, the multimodal information includes two-dimensional image data and three-dimensional point cloud data. The two-dimensional image data can be obtained through a visual perception module, which can be an industrial camera. The three-dimensional point cloud data is obtained through a three-dimensional depth perception module, which can be a depth camera, a binocular vision device, or other three-dimensional measurement equipment. In this embodiment, acquiring multimodal information of the target piece and its adjacent pieces and establishing spatial relationships between them includes: The system acquires 2D image data and 3D point cloud data of the target chess piece and its adjacent pieces. It then synchronizes the 2D and 3D point cloud data in time, ensuring that the 2D image data and 3D point cloud data obtained at the same sampling time correspond to each other. The coordinate systems of the visual perception module and the 3D depth perception module are jointly calibrated to obtain the coordinate transformation relationship between the 2D image coordinate system and the 3D point cloud coordinate system. This coordinate transformation relationship can be determined by camera intrinsic and extrinsic parameters, as well as the rotation matrix and translation vector between the coordinate systems, ensuring that any pixel position in the 2D image corresponds to its spatial position in the 3D point cloud coordinate system. The 2D image data includes the outline and position of the chess piece in the image plane, while the 3D point cloud data includes the 3D coordinate information of each sampling point on the surface of the chess piece.

[0021] The two-dimensional image data is identified to determine the outline regions of the target chess piece and its adjacent pieces. First, the two-dimensional image can be denoised, brightness compensated, and background separated to reduce the impact of chessboard texture, reflections, and changes in ambient lighting on piece recognition. Then, based on the color features, edge features, texture features, or a pre-established chess piece template, target detection and instance segmentation are performed on the chess pieces in the image to distinguish the target chess piece from other pieces, obtaining the outline regions of the target chess piece and its adjacent pieces. Adjacent pieces can be determined based on the position of the target chess piece. For example, using the center position of the target chess piece as a reference, a preset neighborhood region is established around the target chess piece, and pieces located within this neighborhood region that have a potential spatial interference relationship with the target chess piece are identified as adjacent pieces. The neighborhood region can be determined based on the maximum range of motion of the end effector, the size of the gripping part, and a preset safety distance, thereby avoiding including pieces that are far from the target chess piece and will not affect the grasping process in subsequent spatial relationship calculations.

[0022] Based on the coordinate transformation relationship between the two-dimensional image data and the three-dimensional point cloud data, the contour region is mapped to the three-dimensional point cloud region corresponding to the three-dimensional point cloud data to obtain the three-dimensional point cloud corresponding to the target chess piece and each adjacent chess piece; for the pixels in the two-dimensional contour region, based on their coordinates in the two-dimensional image coordinate system and the corresponding depth information, they are transformed to the three-dimensional point cloud coordinate system, thereby filtering out the three-dimensional points located at the corresponding positions in the contour region of the target chess piece and the contour regions of each adjacent chess piece.

[0023] Spatial reconstruction is performed on the 3D point clouds corresponding to the target chess piece and each adjacent chess piece to obtain a 3D model of the target chess piece and each adjacent chess piece. Outlier removal and noise filtering are performed on the 3D point clouds corresponding to each chess piece. Then, the point cloud topology is established based on the spatial distance between adjacent points, and the outer surface of the chess piece is generated using surface reconstruction. For cases where the point cloud density is uneven or there are local gaps, interpolation and hole repair can be performed based on the spatial continuity of adjacent points to obtain a continuous outer surface model of the chess piece. In this embodiment, the 3D model retains geometric information such as the outer contour, bottom height, top height, and sidewall shape of the chess piece. For chess pieces with different shapes, a uniform regular geometric model is not used as a substitute. Instead, 3D reconstruction is performed based on the actual collected point clouds, so that the obtained 3D model can reflect the actual external contour of the target chess piece and adjacent chess pieces. For target chess pieces with asymmetrical structures, their actual spatial posture can also be determined based on the 3D point cloud, and the spatial posture is retained in the 3D model.

[0024] Based on the 3D model of the target piece and the 3D models of each adjacent piece, establish the spatial relationship between the target piece and each adjacent piece, including: The three-dimensional centroid of the target chess piece's three-dimensional model and the three-dimensional centroids of each adjacent chess piece's three-dimensional model are determined respectively. A local three-dimensional coordinate system for the target chess piece is established with the target chess piece's three-dimensional centroid as the reference point. The local three-dimensional coordinate system can take the plane where the chessboard is located as the horizontal reference plane, take the direction perpendicular to the chessboard plane as the vertical direction, and set a preset reference axis in the horizontal plane.

[0025] Based on the coordinate difference between the three-dimensional centroid of the target piece and the three-dimensional centroids of each adjacent piece in the local three-dimensional coordinate system of the target piece, the relative distance between the target piece and each adjacent piece is calculated. The relative distance is used to represent the overall spatial proximity between the target piece and its corresponding adjacent piece. The relative distance can be the Euclidean distance between the three-dimensional centroids of the target piece and the corresponding adjacent pieces.

[0026] Based on the projection direction of the coordinate difference onto the horizontal plane of the local three-dimensional coordinate system of the target piece, the relative position of each adjacent piece relative to the target piece is determined, and the relative position is represented by the angle between the projection direction and the preset reference axis of the local three-dimensional coordinate system of the target piece. This allows us to determine whether the adjacent pieces are located in front of, behind, to the left, to the right, or in other circumferential areas of the target piece.

[0027] Based on the vertical coordinate difference between the target piece's three-dimensional centroid and the three-dimensional centroids of each adjacent piece, the spatial height difference between the target piece and each adjacent piece is calculated. The spatial height difference is used to reflect the height position relationship between adjacent pieces and the target piece, thereby determining whether the end effector has space to pass through from above, to the side, or in other directions of the piece in a certain entry direction.

[0028] Multiple first surface points are sampled from the outer surface of the target chess piece's 3D model, and multiple second surface points are sampled from the outer surface of the corresponding adjacent chess piece's 3D model. Based on the 3D Euclidean distance between each first surface point and each second surface point, the minimum surface distance between the target chess piece and its corresponding adjacent chess piece is determined, and this minimum surface distance is used as the contour gap in the corresponding direction. For example, for an adjacent chess piece located to the right of the target chess piece, a first surface point can be selected from the outer surface of the target chess piece facing that adjacent chess piece, and a second surface point can be selected from the outer surface of the adjacent chess piece facing the target chess piece. The minimum distance between the corresponding surface points is used as the contour gap in that direction. This method avoids the error caused by describing the chess piece spacing solely using the distance between the 3D centroids of two chess pieces, allowing the obtained contour gap to more accurately reflect the actual space width that the end effector can enter.

[0029] Based on the relative distances, relative orientations, spatial height differences, and outline gaps between adjacent pieces, a spatial relationship is formed between the target piece and its adjacent pieces. The space surrounding the target piece can be divided into multiple directional regions centered on it, based on the relative orientations of its adjacent pieces. The adjacent pieces, relative distances, spatial height differences, and outline gaps corresponding to each directional region are then associated and stored. This spatial relationship is used to subsequently determine the accessibility of the end effector when approaching the target piece from different directions.

[0030] Based on the spatial relationship and the structural characteristics of the end effector, the grasping posture and entry path of the end effector are determined. Using the reference axis of the target piece's local three-dimensional coordinate system as a benchmark, the circumference of the target piece is divided into multiple angle intervals, and the contour gap and spatial height difference within each angle interval are statistically analyzed to obtain the accessible space in different directions. The candidate entry directions are screened based on the width, length, thickness, maximum opening distance, movement range of the gripping part, and the allowable posture change range of the end effector. If the available space corresponding to a candidate entry direction cannot accommodate the gripping part, or if the gripping part interferes with the three-dimensional model of adjacent pieces when entering along that direction, then that candidate entry direction is eliminated. Candidate entry directions that meet the requirements for the gripping part's entry space and can maintain a safe distance from adjacent pieces are retained as feasible entry directions. In this embodiment, determining the grasping posture and entry path of the end effector based on the spatial relationship and the structural characteristics of the end effector includes: Based on the relative positions and outline gaps between the target piece and its adjacent pieces, a first candidate entry direction around the target piece is determined. Using the horizontal plane of the target piece's local three-dimensional coordinate system as a reference, the target piece's circumference is divided into multiple directional regions, each corresponding to a candidate entry direction for the end effector. Based on the aforementioned relative positions of the adjacent pieces, each adjacent piece is mapped to its corresponding directional region, and the outline gap in that direction is used to determine whether the direction has initial space for the end effector to enter. For example, if there are no adjacent pieces to the left of the target piece or if there is a large outline gap to the left of the target piece, the left side of the target piece can be considered a first candidate entry direction. If there are closely spaced adjacent pieces to the right of the target piece with a small outline gap, the priority of that direction as a candidate entry direction is reduced. If the outline gap in a certain direction is less than a preset minimum entry gap, that direction is directly eliminated from the first candidate entry directions. This yields multiple directions around the target piece with initial entry potential.

[0031] Based on the dimensions and movement range of the clamping part of the end effector, specifically the maximum width, thickness, length, maximum opening distance, effective clamping depth, and achievable posture change range of the clamping part, a three-dimensional structural model of the clamping part is established. Then, based on multiple positional postures of the clamping part moving along each first candidate entry direction, the occupied space range of the clamping part in the corresponding candidate entry direction is determined. The occupied space range is spatially compared with the contour gap and the three-dimensional models of adjacent pieces in the corresponding candidate entry direction. When the occupied space range does not overlap with the three-dimensional models of adjacent pieces, and the minimum distance between them is greater than a preset safety distance, the corresponding candidate entry direction is determined as the second candidate entry direction. The second candidate entry direction represents the first candidate entry direction that can accommodate the clamping part without interfering with adjacent pieces. Based on the contour gap and spatial height difference corresponding to each of the second candidate entry directions, the available gripping space corresponding to each second candidate entry direction is determined. Taking the second candidate entry direction as the spatial entry direction, the available width of the gripping part in the horizontal direction is determined based on the contour gap in that direction, and the available height of the gripping part in the vertical direction is determined based on the spatial height difference between the target piece and adjacent pieces, thereby constructing a local available gripping space corresponding to the second candidate entry direction. The available gripping space can be represented as a three-dimensional spatial region surrounding the target gripping area. The horizontal dimension of this three-dimensional spatial region is determined by the contour gap in the corresponding direction, the vertical dimension is determined by the spatial height difference and the height relationship between the chessboard surface and the top of adjacent pieces, and the longitudinal dimension is determined based on the effective distance that the end effector can move along the target entry direction. Thus, the three-dimensional available gripping space corresponding to different second candidate entry directions can be obtained.

[0032] Based on the matching degree between the available gripping space and the structural dimensions of the end effector, the accessibility of each second candidate entry direction is evaluated to obtain accessibility evaluation results. The three-dimensional structural model corresponding to the gripping part is placed in the corresponding available gripping space, and it is determined whether the gripping part can complete opening, entering, closing, and gripping actions while maintaining a preset safety margin. If the gripping part can complete the above actions within the available gripping space, the direction is considered to meet the basic structural accessibility conditions; if the gripping part can enter but cannot complete the predetermined opening amount or closing action, the direction is considered not to meet the gripping requirements and is eliminated. Furthermore, the accessibility evaluation results of each second candidate entry direction can be determined based on the remaining space between the available gripping space and the structural dimensions of the end effector. A larger remaining space results in a higher accessibility evaluation result.

[0033] Based on the reachability evaluation results, the grasping posture and entry path of the end effector are determined, including: The target entry direction is determined based on the accessibility evaluation results, and the posture of the end effector's gripping part facing the target entry direction is defined as the grasping posture. The direction that satisfies the preset accessibility conditions and has the highest evaluation result is selected from the second candidate entry directions as the target entry direction. The target entry direction represents the direction of motion of the end effector as it enters from the space surrounding the target piece. The main motion axis of the gripping part is determined based on the target entry direction, so that the gripping opening and closing direction of the gripping part matches the predetermined gripping direction of the target piece. Simultaneously, the posture angle of the gripping part in the vertical direction is determined based on the three-dimensional model of the target piece.

[0034] Using the target entry direction as a constraint, the target grasping position corresponding to the grasping posture is determined based on the three-dimensional model of the target piece. Multiple candidate clamping areas can be determined on the outer surface of the target piece's three-dimensional model, and then filtered based on the distance between the candidate clamping area and adjacent pieces, the effective clamping range of the clamping part, and the center of gravity of the target piece. When a candidate clamping area allows the clamping part to enter from the target entry direction, and the center of gravity of the target piece is within the effective support range of the clamping part after clamping, while maintaining a preset safe distance between the clamping part and adjacent pieces, the candidate clamping area is retained. Further, from the retained candidate clamping areas, a position with a small deviation from the center of gravity of the target piece and a large clamping space is selected as the target grasping position.

[0035] An entry path consisting of an approach segment and a grasp segment is generated based on the target grasping position, the target entry direction, and the current position of the end effector. The approach segment moves the end effector from its current position to a preset approach position around the target piece. During the approach segment, the end effector can maintain a relatively stable posture and gradually approach the target piece in a direction avoiding adjacent pieces. The approach position is assumed to be located at a preset distance extending in the opposite direction from the target grasping position along the target entry direction, allowing the end effector sufficient space for posture adjustment before entering the grasping area. The grasp segment moves the end effector further from the preset approach position along the target entry direction to the target grasping position.

[0036] The end effector is controlled to approach the target piece along the entry path, and its motion state is dynamically adjusted based on real-time sensing results. Since the piece may be affected by factors such as board vibration, human movement, collisions with other pieces, or visual positioning errors, the actual position of the target piece and adjacent pieces may deviate from the initial modeling result. Therefore, during the approach segment of the end effector's movement along the entry path, the spatial position of the target piece and adjacent pieces is continuously sensed, and the motion state of the end effector is dynamically adjusted based on the real-time sensing results. In this embodiment, this includes: During the approach segment of the entry path, the end effector acquires real-time image data and real-time depth data of the target piece and its adjacent pieces at a preset sampling period. The preset sampling period can be set according to the sampling frequency of the vision sensor, the movement speed of the robotic arm, and the spacing between the pieces, so that the movement distance of the end effector between two adjacent perceptions is less than the preset update distance, thereby ensuring that the change in the position of the piece can be detected in time during the process of the end effector rapidly approaching the piece.

[0037] The real-time spatial relationships of the target piece and its adjacent pieces are determined based on the real-time image data and real-time depth data; the real-time spatial relationships can be obtained using the spatial relationship acquisition method described above.

[0038] The real-time spatial relationship is compared with the spatial relationship to obtain the comparison result; the real-time relative distance, real-time relative orientation, real-time spatial height difference and real-time contour gap in the real-time spatial relationship are compared with the corresponding initial relative distance, initial relative orientation, initial spatial height difference and initial contour gap, respectively, so as to determine the spatial state changes of the target piece and each adjacent piece.

[0039] Based on the comparison results, the real-time distance between the end effector and each adjacent piece is calculated. When the comparison results show that the change between the real-time spatial relationship of the target piece and each adjacent piece and the initial spatial relationship is within a preset allowable error range, the original entry path can continue to be used for approach movement. When the real-time contour gap of an adjacent piece decreases or the real-time relative orientation changes significantly, the spatial constraints corresponding to that adjacent piece are re-determined based on the updated real-time spatial relationship. Further, specifically, the three-dimensional spatial position of the clamping part at the current moment is determined based on the real-time pose of the end effector, and combined with the real-time three-dimensional models of each adjacent piece, the minimum three-dimensional distance between the outer surface of the clamping part and the outer surface of the corresponding adjacent piece is calculated. This minimum three-dimensional distance is used as the real-time distance between the end effector and the corresponding adjacent piece. When calculating the real-time distance, it can be calculated separately for each adjacent piece, thereby obtaining multiple real-time distances, and the minimum real-time distance among them is used as the main safety evaluation parameter of the current end effector movement state.

[0040] Based on the relationship between the real-time distance and the preset safety distance, the motion state of the end effector is dynamically adjusted, including: The real-time distance between the end effector and each adjacent chess piece is compared with a first safety distance threshold and a second safety distance threshold, wherein the first safety distance threshold is greater than the second safety distance threshold; the two safety distance thresholds can be determined based on the size of the end effector gripping part, the positioning error of the robotic arm, the visual measurement error, and the preset safety margin.

[0041] When the real-time distance is greater than or equal to the first safe distance threshold, the end effector is controlled to move at a preset approach speed along the entry path. This indicates that there is sufficient space for movement between the end effector and adjacent pieces. In this state, the end effector maintains its current direction of movement and continues to update the real-time spatial relationship between the target piece and adjacent pieces according to a preset sampling period.

[0042] When the real-time distance is less than the first safe distance threshold and greater than or equal to the second safe distance threshold, the approach speed of the end effector is reduced. This indicates that the end effector has entered the vicinity of an adjacent piece, but has not yet reached a high collision risk state; in this case, the approach speed of the end effector is reduced. The approach speed can be divided into a preset approach speed, a medium approach speed, and a low approach speed. When the real-time distance enters the area between the first and second safe distance thresholds, the end effector is reduced from the preset approach speed to the medium approach speed; when the real-time distance further approaches the second safe distance threshold, it is further reduced to the low approach speed.

[0043] When the real-time distance is less than the second safe distance threshold, the end effector is controlled to pause its movement, and the local obstacle avoidance direction is re-determined based on the real-time spatial relationship between the target piece and its adjacent pieces. This indicates that continuing to move along the current entry path carries a high risk of collision. Therefore, the end effector is controlled to pause its movement, and the local obstacle avoidance direction is re-determined based on the real-time spatial relationship between the target piece and its adjacent pieces. The re-determination method involves determining multiple candidate obstacle avoidance directions on both sides of the current movement direction of the end effector based on the real-time spatial relationship; obtaining the real-time contour gap corresponding to each candidate obstacle avoidance direction; determining the candidate obstacle avoidance direction with the largest real-time contour gap as the local obstacle avoidance direction; controlling the end effector to perform position correction along the local obstacle avoidance direction, moving towards the target piece; controlling the end effector to move a preset correction distance along the local obstacle avoidance direction to avoid the corresponding adjacent pieces; and then moving towards the target piece again.

[0044] When the end effector reaches the grasping phase, the grasping parameters are adjusted based on real-time sensing results, including: Based on the outer contour dimensions of the target chess piece and the structural dimensions of the clamping part of the end effector, the target clamping position and target opening amount of the clamping part relative to the target chess piece are determined. These target clamping position and target opening amount are gripping parameters. After the end effector moves along the gripping segment of the entry path to the target gripping area, the outer contour dimensions of the target chess piece are determined based on the real-time 3D model of the target chess piece. The outer contour dimensions may include the maximum outer width, maximum outer height, contour width in the clamping direction, and the local contour dimensions corresponding to the clamping area. Simultaneously, based on the structural parameters of the clamping part of the end effector, the maximum opening amount, minimum opening amount, clamping part length, clamping part thickness, and the effective clamping range between the two clamping fingers are determined. Based on the outer contour dimensions of the target chess piece and the structural dimensions of the clamping part of the end effector, the target clamping position and target opening amount of the clamping part relative to the target chess piece are determined. The target clamping position represents the position of the two clamping parts relative to the target chess piece, and the target opening amount represents the opening distance maintained by the two clamping parts before performing the clamping action. For example, for a chess piece with an approximately cylindrical or cylindrical outer contour, the initial clamping center of the clamping part can be determined based on the real-time center position of the target chess piece, and the clamping center can be set in the radial middle region of the target chess piece, allowing the two clamping parts to move inward from opposite sides of the target chess piece. For a chess piece with an irregular outer contour, a suitable local contour region for clamping can be determined based on the three-dimensional model of the target chess piece, and the center of the local contour region can be determined as the target clamping position. The target opening amount can be determined based on the maximum contour size of the target chess piece in the clamping direction, allowing the clamping part to pass through both sides of the target chess piece when approaching it, while reserving an opening margin to compensate for positioning errors and chess piece contour measurement errors. The target clamping position and target opening amount serve as the gripping parameters for the current gripping stage.

[0045] Based on the target clamping position and target opening amount, determine the real-time grabbing distance between each adjacent piece; map the outer contours of the two clamping parts under the current target clamping position and target opening amount onto the three-dimensional space around the target piece, and calculate the minimum distance between the outer contour of the clamping part and the outer surface of the real-time three-dimensional model of each adjacent piece, and use the minimum distance as the real-time grabbing distance of the corresponding adjacent piece.

[0046] Based on the real-time grasping distance, the target clamping position and target opening amount are adjusted. Each real-time grasping distance is compared with a preset safe grasping distance to determine whether the current target clamping position and target opening amount are suitable for actual grasping. When each real-time grasping distance is greater than or equal to the preset safe grasping distance, it indicates that clamping according to the current target clamping position and target opening amount will not cause significant interference between the clamping part and adjacent pieces, and the current grasping parameters can be maintained. When at least one real-time grasping distance is less than the preset safe grasping distance, it indicates that the current target clamping position or target opening amount may cause the clamping part to approach or collide with adjacent pieces during the closing process. In this case, the target clamping position and target opening amount are adjusted. Preferably, the target clamping position can be adjusted first in a direction away from the corresponding adjacent piece to increase the spatial distance between the clamping part and the adjacent piece. When adjusting the target clamping position still cannot meet the preset safe grasping distance, the target opening amount is reduced to reduce the space occupied by the clamping part around the target piece.

[0047] Based on the adjusted target clamping position and target opening amount, the target chess piece is clamped; based on the adjusted target clamping position and target opening amount, the clamping part of the end effector is controlled to move to the target clamping position, and the clamping part is kept in an open state according to the adjusted target opening amount. Then, the two clamping parts are controlled to move towards each other, so that the clamping parts gradually contact the outer surface of the target chess piece and form a clamp.

[0048] The contact force and relative displacement of the clamping part are obtained, and the contact state of the target piece is determined based on the contact force and the relative displacement. The contact force can be obtained by force sensor, pressure sensor or torque estimation result of drive motor set on the clamping part; the relative displacement indicates whether the clamping part slides with the target piece.

[0049] The target clamping force is adjusted according to the contact state, where the target clamping force is a grasping parameter. When the contact force reaches a preset stable contact force range and the relative displacement remains stable within a preset time period, the target piece is determined to be in a stable clamping state, and the current target clamping force is maintained. When the contact force reaches a preset contact force threshold, but the relative displacement continues to increase, it is determined that there is relative sliding between the target piece and the clamping part, and the target piece is determined to be in a sliding state. The target clamping force is increased to prevent it from falling off.

[0050] Grabbing is performed according to the aforementioned grabbing parameters, and the target piece is removed, including: The target piece is gripped according to the gripping parameters; the spatial relationships around the gripping part are updated to generate a safe movement path; the target piece is moved out according to the safe movement path. After gripping the target piece, the real-time spatial relationships of the gripping part and its surrounding target pieces and adjacent pieces are reacquired, and it is determined whether there is spatial interference when the gripping part moves upward based on the real-time spatial relationships; if there is spatial interference, the rotation attitude of the end effector is adjusted according to the real-time spatial relationships so that the target piece avoids adjacent pieces, and the end effector is controlled to move vertically to lift and move the target piece out.

[0051] This invention establishes a three-dimensional spatial relationship between the target piece and its neighboring pieces by fusing two-dimensional image data and three-dimensional point cloud data. This allows for accurate acquisition of relative distance, relative orientation, spatial height difference, and contour gap information, providing more accurate spatial constraints for the end effector's grasping posture and entry path planning. Furthermore, by combining the end effector's gripping part dimensions and range of motion, candidate entry directions are screened and their accessibility evaluated. Collision detection is used to determine a suitable entry path, reducing the risk of interference between the end effector and neighboring pieces in dense chessboard environments. Simultaneously, real-time images and depth information are continuously acquired as the end effector approaches the target piece. The movement speed and direction are dynamically adjusted based on the real-time distance between the end effector and neighboring pieces, improving dynamic obstacle avoidance capabilities in complex chessboard environments. During the grasping phase, grasping parameters are adaptively determined, enabling the grasping action to adapt to different piece states, improving gripping stability and grasping success rate, reducing collision and misgrabbing risks, and enhancing the grasping success rate, stability, and safety in dense chessboard environments.

[0052] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

[0053] The present invention has been described above with reference to embodiments thereof. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

[0054] Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions, and modifications can be made to the embodiments of the present invention without departing from the spirit and scope of the invention.

[0055] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An adaptive grasping control method for a chess-like interactive device based on multimodal perception, characterized in that, include: Acquire multimodal information of the target piece and its adjacent pieces, and establish spatial relationships between the target piece and its adjacent pieces. The spatial relationships include relative distance, relative orientation, spatial height difference, and outline gap. Based on the spatial relationship and the structural features of the end effector, the grasping posture and entry path of the end effector are determined. The structural features of the end effector include the size of the clamping part and the range of motion of the clamping part. The end effector is controlled to approach the target piece along the entry path, and its movement state is dynamically adjusted according to real-time perception results, including the real-time distance between the end effector and each adjacent piece. When the end effector reaches the grasping stage, the grasping parameters are adjusted based on the real-time sensing results. The target piece is moved out by grabbing according to the grabbing parameters. The step of determining the grasping posture and entry path of the end effector based on the spatial relationship and the structural characteristics of the end effector includes: Based on the relative position and outline gap between the target piece and each adjacent piece, determine the first candidate entry direction of the target piece in the circumferential direction; Based on the size of the clamping part and the range of motion of the clamping part of the end effector, a second candidate entry direction is obtained. The second candidate entry direction represents the first candidate entry direction that can accommodate the clamping part without interfering with adjacent pieces. Based on the contour gap and spatial height difference corresponding to each of the second candidate entry directions, determine the available grabbing space corresponding to each of the second candidate entry directions; Based on the degree of matching between the available grasping space and the structural dimensions of the end effector, the accessibility of each second candidate entry direction is evaluated to obtain the accessibility evaluation result; Based on the reachability evaluation results, the grasping posture and entry path of the end effector are determined.

2. The adaptive grasping control method for a chess-like interactive device based on multimodal perception according to claim 1, characterized in that, The multimodal information includes two-dimensional image data and three-dimensional point cloud data; the acquisition of multimodal information of the target piece and adjacent pieces, and the establishment of spatial relationships between the target piece and adjacent pieces, includes: Acquire two-dimensional image data and three-dimensional point cloud data of the target piece and its adjacent pieces; The two-dimensional image data is identified to determine the outline regions of the target chess piece and each adjacent chess piece; Based on the coordinate transformation relationship between the two-dimensional image data and the three-dimensional point cloud data, the contour region is mapped to the three-dimensional point cloud region corresponding to the three-dimensional point cloud data to obtain the three-dimensional point cloud corresponding to the target chess piece and each adjacent chess piece. Spatial reconstruction is performed on the three-dimensional point cloud corresponding to the target chess piece and each adjacent chess piece to obtain the three-dimensional model of the target chess piece and the three-dimensional model of each adjacent chess piece. Based on the 3D model of the target piece and the 3D models of each adjacent piece, establish the spatial relationship between the target piece and each adjacent piece.

3. The adaptive grasping control method for a chess-like interactive device based on multimodal perception according to claim 2, characterized in that, The step of establishing the spatial relationship between the target piece and its neighboring pieces based on the target piece's 3D model and the 3D models of each adjacent piece includes: The three-dimensional centroid of the target chess piece's three-dimensional model and the three-dimensional centroids of each adjacent chess piece's three-dimensional model are determined respectively, and a local three-dimensional coordinate system of the target chess piece is established with the three-dimensional centroid of the target chess piece as the reference point; The relative distance between the target piece and each of its adjacent pieces is calculated based on the coordinate difference between the three-dimensional centroid of the target piece and the three-dimensional centroid of each of its adjacent pieces in the local three-dimensional coordinate system of the target piece. Based on the projection direction of the coordinate difference onto the horizontal plane of the local three-dimensional coordinate system of the target piece, determine the relative orientation of each adjacent piece relative to the target piece; Based on the vertical coordinate difference between the target piece's three-dimensional centroid and the three-dimensional centroids of each adjacent piece, calculate the spatial height difference between the target piece and each adjacent piece; Multiple first surface points are obtained by sampling from the outer surface of the target chess piece's 3D model, and multiple second surface points are obtained by sampling from the outer surface of the corresponding adjacent chess piece's 3D model. The minimum surface distance between the target chess piece and the corresponding adjacent chess piece is determined based on the 3D Euclidean distance between each first surface point and each second surface point, and the minimum surface distance is used as the contour gap in the corresponding direction. Based on the relative distance, relative orientation, spatial height difference, and outline gap of each adjacent piece, the spatial relationship between the target piece and each adjacent piece is formed.

4. The adaptive grasping control method for a chess-like interactive device based on multimodal perception according to claim 3, characterized in that, The step of determining the grasping posture and entry path of the end effector based on the reachability evaluation results includes: The target entry direction is determined based on the reachability evaluation result, and the attitude of the gripping part of the end effector toward the target entry direction is determined as the grasping attitude; Using the target's direction of entry as a constraint, the target grabbing position corresponding to the grabbing posture is determined based on the three-dimensional model of the target piece; An entry path consisting of an approach segment and a grasping segment is generated based on the target grasping position, the target entry direction, and the current position of the end effector.

5. The adaptive grasping control method for a chess-like interactive device based on multimodal perception according to claim 4, characterized in that, The control end effector approaches the target chess piece along the entry path and dynamically adjusts its movement state based on real-time sensing results, including: During the approach segment movement of the end effector along the entry path, real-time image data and real-time depth data of the target piece and each adjacent piece are acquired at a preset sampling period. The real-time spatial relationships between the target chess piece and its adjacent chess pieces are determined based on the real-time image data and real-time depth data. The real-time spatial relationship is compared with the spatial relationship to obtain the comparison result; Based on the comparison results, the real-time distance between the end effector and each adjacent chess piece is calculated; The motion state of the end effector is dynamically adjusted based on the relationship between the real-time distance and the preset safe distance.

6. The adaptive grasping control method for a chess-like interactive device based on multimodal perception according to claim 5, characterized in that, The dynamic adjustment of the motion state of the end effector based on the relationship between the real-time distance and the preset safety distance includes: The real-time distance between the end effector and each adjacent chess piece is compared with a first safe distance threshold and a second safe distance threshold, wherein the first safe distance threshold is greater than the second safe distance threshold; When the real-time distance is greater than or equal to the first safe distance threshold, the end effector is controlled to move at a preset approach speed according to the entry path; When the real-time distance is less than the first safe distance threshold and greater than or equal to the second safe distance threshold, reduce the approach speed of the end effector; When the real-time distance is less than the second safe distance threshold, the end effector is controlled to pause its movement, and the local obstacle avoidance direction is re-determined according to the real-time spatial relationship between the target piece and adjacent pieces. The end effector is controlled to correct its position along the local obstacle avoidance direction and move toward the target piece.

7. The adaptive grasping control method for a chess-like interactive device based on multimodal perception according to claim 6, characterized in that, The step of adjusting the grasping parameters based on real-time sensing results when the end effector reaches the grasping stage includes: Based on the outer contour dimensions of the target chess piece and the structural dimensions of the clamping part of the end effector, the target clamping position and target opening amount of the clamping part relative to the target chess piece are determined, and the target clamping position and target opening amount are grasping parameters; Based on the target clamping position and the target opening amount, determine the real-time grabbing distance between each adjacent piece; Adjust the target clamping position and target opening amount based on the real-time grasping distance.

8. The adaptive grasping control method for a chess-like interactive device based on multimodal perception according to claim 7, characterized in that, The step of adjusting the grasping parameters based on real-time sensing results when the end effector reaches the grasping stage also includes: Based on the adjusted target clamping position and target opening amount, the target chess piece is clamped; The contact force and relative displacement of the clamping part are obtained, and the contact state of the target chess piece is determined based on the contact force and the relative displacement. The target clamping force is adjusted according to the contact state, and the target clamping force is a gripping parameter.

9. The adaptive grasping control method for a chess-like interactive device based on multimodal perception according to claim 8, characterized in that, The step of grasping according to the grasping parameters and moving the target piece out includes: The target chess piece is held according to the grasping parameters; Update the spatial relationships around the clamping part to generate a safe movement path; Move the target piece out according to the safe movement path.

Images