Chess assistant referee method and system

Abstract

The application provides a chess auxiliary judging method and system, and relates to the technical field of auxiliary judging. The application synchronously collects image sequences through deployment of multi-view RGB-D cameras, and generates a three-dimensional point cloud model of a hand and a chess piece based on three-dimensional reconstruction and image detection technology, thereby realizing real-time, automatic and accurate detection of chess violation events. The accuracy of the judgment is ensured by calculating the minimum spatial distance between point clouds to determine contact and verifying the compliance of the behavior in combination with the change of the chessboard state. The three-dimensional spatial geometric information is used to overcome the visual limitations of a single view and the shortcomings of two-dimensional image analysis, effectively solve the missed judgment and misjudgment problem in the chess piece dense and shielding scene, improve the judging efficiency and the fairness of the game, and realize automatic judging.

Description

Technical Field

[0001] This invention relates to the field of auxiliary refereeing technology, specifically to a method and system for auxiliary refereeing in chess. Background Technology

[0002] In chess, the touch-move rule (that a piece must be moved upon touching it) is a fundamental and crucial rule. Currently, the adjudication of this rule primarily relies on the chief umpire's visual observation or video replays after a dispute arises. However, this manual approach has significant limitations: firstly, the umpire's field of vision may have blind spots, especially in areas with densely packed pieces or when a player's hand obscures the view, making it easy to miss subtle, momentary touches; secondly, relying on post-incident video replays disrupts the rhythm of the game, affecting players' continuous thinking and overall gaming experience.

[0003] To improve the objectivity of judgments, some existing technical solutions attempt to use cameras for auxiliary analysis. However, most of these solutions are based on two-dimensional image analysis from a single perspective. Their shortcomings are that they cannot effectively solve the problem of visual occlusion in three-dimensional space, and for judging touch, which is essentially a three-dimensional contact behavior, they lack precise geometric basis. They can only rely on features in the image, such as overlap and color, which are easily affected by lighting and angle, to make inferences. In other words, they cannot accurately quantify spatial contact behavior through three-dimensional image detection, resulting in a high rate of false positives and false negatives.

[0004] Therefore, there is an urgent need in this field for a method that can overcome the limitations of two-dimensional vision and achieve real-time, automatic, and accurate three-dimensional detection and judgment of illegal touches of chess pieces in chess competition scenarios. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the present invention provides a method and system for assisting referees in chess.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for assisting referees in chess includes the following steps:

[0008] S1. Simultaneously acquire a multi-view image sequence containing the chessboard and the player's operating area using at least two RGB-D cameras set at different spatial locations. The image sequence includes a synchronized RGB image sequence and a depth image sequence.

[0009] S2. Perform three-dimensional reconstruction processing on the RGB image sequence and the depth image sequence from different perspectives at the same time to generate a three-dimensional point cloud model of the hand and a three-dimensional point cloud model corresponding to each chess piece at that time, which are respectively denoted as hand point cloud data and chess piece point cloud dataset, wherein each chess piece has a unique and continuous identity identifier.

[0010] S3. Based on the hand point cloud data and the chess piece point cloud dataset, a violation determination is made, specifically including the following sub-steps:

[0011] S31. Contact Target Recognition: Calculate the minimum spatial distance between the hand point cloud data and each chess piece point cloud data in the chess piece point cloud dataset; if the minimum spatial distance of a certain chess piece is less than the preset contact distance threshold, it is determined that contact has occurred, and the identity identifier of the chess piece is recorded as the ID of the touched chess piece.

[0012] S32. Related Behavior Capture and Judgment: After determining that contact has occurred, continuously monitor the chess piece point cloud dataset. When a chessboard state change event is detected, immediately perform a compliance judgment. The chessboard state change event includes chess piece movement events and chess piece disappearance events, wherein:

[0013] If a piece movement event is detected, record the identity of all pieces that have moved to form a set of moving piece IDs; determine whether the touched piece ID is included in the set of moving piece IDs; if not, generate a response target inconsistency violation event.

[0014] If a piece disappearance event is detected, the identity identifiers of all disappeared pieces are recorded to form a set of disappeared piece IDs; it is determined whether the piece identified by the touched piece ID is an opponent's piece, and at the same time, it is determined whether the touched piece ID is included in the set of disappeared piece IDs; if the piece identified by the touched piece ID is an opponent's piece and the touched piece ID is not included in the set of disappeared piece IDs, then a target non-elimination violation event is generated.

[0015] S4. Output the alarm information for the generated violation events.

[0016] Further, in step S31, the minimum spatial distance between the hand point cloud data and the point cloud data of a certain chess piece is calculated, specifically by: calculating the Euclidean distance between each point in the hand point cloud data and each point in the chess piece point cloud data, and selecting the minimum value from all calculated Euclidean distances as the minimum spatial distance.

[0017] Further, in step S321, the detection of movement events and recording of the identity identifiers of all moving pieces specifically involves: for each piece in the piece point cloud dataset, continuously comparing its point cloud centroid coordinates at the moment of contact determination with subsequent moments; if the magnitude of a piece's centroid displacement vector is greater than or equal to a preset displacement threshold, then it is determined that the piece has moved, and its identity identifier is added to the moving piece ID set.

[0018] Further, in step S2, generating the 3D point cloud model of the hand and the 3D point cloud models of each chess piece at that moment specifically includes:

[0019] S21. Perform hand segmentation and chess piece instance segmentation on the RGB image of each viewpoint to obtain a two-dimensional hand region mask and an independent region mask for each chess piece.

[0020] S22. Register the hand region mask and the independent region mask of the chess piece in each viewpoint with the depth image of the corresponding viewpoint to obtain the three-dimensional point cloud fragment of the hand and the three-dimensional point cloud fragment of each chess piece in that viewpoint.

[0021] S23. Merge the three-dimensional point cloud fragments of the hand from all perspectives to generate unified hand point cloud data; and for each piece, merge the three-dimensional point cloud fragments from all perspectives to generate unified point cloud data with an identity identifier for that piece. The point cloud data of all pieces constitute the piece point cloud dataset.

[0022] Furthermore, in step S21, the hand segmentation and chess piece segmentation are implemented using a deep learning-based semantic segmentation model.

[0023] Furthermore, in step S4, the alarm information includes the type of violation event, the timestamp of the occurrence, the ID of the touched piece, and a 3D visualization scene screenshot generated based on the hand point cloud data and the piece point cloud data.

[0024] Furthermore, in step S32, if a piece movement event is detected, and the ID of the touched piece is a king, and the set of moving piece IDs includes both a king and a rook, then a compliance determination of the castling process is further performed:

[0025] S81. Timing compliance judgment: Determine whether the order of the king and rook piece movement events conforms to the preset rule of moving the king first and then the rook.

[0026] S82. Interaction compliance judgment: Determine whether the king's piece movement event and the rook's piece movement event are triggered by hand operations associated with the same hand point cloud data;

[0027] S83. If step S81 determines that the timing is non-compliant, or step S82 determines that the operation is not performed by the same hand, then based on the non-compliance event of the response target, a sub-type violation event marked as transposition process violation is generated, and the timing compliance judgment result and the interaction compliance judgment result are incorporated into the alarm information.

[0028] Furthermore, prior to step S21, a step S20 is included to perform anti-interference repair on the depth image sequence:

[0029] S201. Simultaneously acquire a polarization image sequence that is temporally and spatially aligned with the RGB image sequence and the depth image sequence using the RGB-D camera;

[0030] S202. Based on the polarization image sequence, calculate and generate a polarization feature map characterizing the reflection properties of the object surface;

[0031] S203. For the same acquisition time, extract one frame of RGB image and one frame of depth image from the RGB image sequence and the depth image sequence respectively, fuse the RGB image, the depth image and the polarization feature map, and input them into the pre-trained deep inpainting neural network.

[0032] S204. The deep repair neural network outputs a repaired depth image, which is used to replace the original depth image and participate in the registration process in step S22.

[0033] A chess referee assistance system, characterized in that it includes:

[0034] The data acquisition module uses at least two RGB-D cameras positioned at different spatial locations to simultaneously acquire a multi-view image sequence containing the chessboard and the player's operating area. The image sequence includes a synchronized RGB image sequence and a depth image sequence.

[0035] The 3D reconstruction and identification module is used to perform 3D reconstruction processing on the RGB image sequence and the depth image sequence from different perspectives at the same time, and generate a 3D point cloud model of the hand and a 3D point cloud model corresponding to each chess piece at that time, which are respectively denoted as hand point cloud data and chess piece point cloud dataset, wherein each chess piece has a unique and persistent identity identifier.

[0036] The violation determination module is used to determine violations based on the hand point cloud data and the chess piece point cloud dataset, specifically including:

[0037] The contact target recognition unit is used to calculate the minimum spatial distance between the hand point cloud data and each chess piece point cloud data in the chess piece point cloud dataset; if the minimum spatial distance of a certain chess piece is less than the preset contact distance threshold, it is determined that contact has occurred, and the identity identifier of the chess piece is recorded as the ID of the touched chess piece.

[0038] The associated behavior capture and judgment unit is used to continuously monitor the chess piece point cloud dataset after determining that contact has occurred. When a chessboard state change event is detected, a compliance judgment is immediately performed. The chessboard state change event includes chess piece movement events and chess piece disappearance events, wherein:

[0039] If a piece movement event is detected, record the identity of all pieces that have moved to form a set of moving piece IDs; determine whether the touched piece ID is included in the set of moving piece IDs; if not, generate a response target inconsistency violation event.

[0040] If a piece disappearance event is detected, the identity identifiers of all disappeared pieces are recorded to form a set of disappeared piece IDs; it is determined whether the piece identified by the touched piece ID is an opponent's piece, and at the same time, it is determined whether the touched piece ID is included in the set of disappeared piece IDs; if the piece identified by the touched piece ID is an opponent's piece and the touched piece ID is not included in the set of disappeared piece IDs, then a target non-elimination violation event is generated.

[0041] The output module is used to output alarm information for the generated violation events.

[0042] The beneficial effects of this invention are:

[0043] This invention utilizes multi-view RGB-D cameras to synchronously acquire image sequences and generates 3D point cloud models of hands and chess pieces based on 3D reconstruction and image detection technologies. This enables real-time, automatic, and accurate detection of unauthorized touches on chess pieces. Contact is determined by calculating the minimum spatial distance between point clouds, and the piece displacement is verified within a preset time window to ensure accuracy. By leveraging 3D spatial geometric information, this invention overcomes the limitations of single-view vision and the shortcomings of 2D image analysis, effectively solving the problems of missed and incorrect judgments in scenarios with dense chess pieces and occlusion. This improves referee efficiency and fairness, achieving automated refereeing. Attached Figure Description

[0044] Figure 1 This is a flowchart of a chess referee assistance method according to an embodiment of the present invention;

[0045] Figure 2 This is a flowchart illustrating the determination process of an embodiment of the present invention;

[0046] Figure 3 This is a block diagram of a chess referee assistance system according to an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] like Figures 1-2As shown in the figure, an embodiment of the present invention provides a chess referee assistance method, which includes the following steps:

[0049] S1. Simultaneously acquire a multi-view image sequence containing the chessboard and the player's operating area using at least two RGB-D cameras set in different spatial locations. The image sequence includes a synchronized RGB image sequence and a depth image sequence.

[0050] In chess matches, the interaction between a player's hand and the pieces is instantaneous, dynamic, and easily obstructed. For example, a player's hand may obscure parts of a piece, and dense arrangement of pieces may cause visual overlap. Step S1, through the synchronous acquisition of data from a multi-view RGB-D camera, can cover these blind spots while taking into account both color texture information (RGB image) and spatial distance information (depth image), laying the foundation for the subsequent conversion of two-dimensional data into a three-dimensional spatial model.

[0051] In this embodiment of the invention, to ensure the integrity of the viewpoint, the entire chessboard and the area where the player's hands operate can be covered by cameras at at least two different positions. This ensures that each chessboard square is simultaneously covered by the field of view of at least two cameras, avoiding partial omissions due to the player's arms or body obstructing the view. At the same time, the acquisition time error of different cameras is controlled within milliseconds (usually ≤10ms) to prevent the same action from being captured as different states at different times by different cameras. For example, the hand may have left the chess piece, but one camera may still delay capturing the contact image, leading to misalignment in subsequent 3D reconstruction. In addition, the RGB image must clearly present the texture of the chess piece and the details of the hand to support semantic segmentation, while the depth image must provide spatial distance data with millimeter-level accuracy. The camera acquisition frame rate must be ≥30fps to capture the instantaneous touch action of the hand in 0.1-0.5s, and the depth measurement accuracy must be ≤2mm at a distance of 1m to avoid misjudgment of subsequent contact determination due to insufficient data accuracy.

[0052] In terms of resolution, RGB images must be ≥1920×1080 (1080P) to ensure clear distinction of details such as the top markings of the pieces and finger joints; depth image resolution must be ≥640×480 to ensure that a single piece area can cover more than 10 depth pixels, meeting the accuracy requirements for centroid calculation. The depth measurement range is set at 0.5m-3m, which is suitable for a typical deployment distance of 1-2m (too close and it's easy to be obstructed, too far and the accuracy decreases), and can also cover the complete movement trajectory of the player's hand from the resting position (the sides of the chessboard) to touching the piece. A frame rate of ≥30fps is to avoid frame skipping and ensure that each touch action is captured by at least 3 frames of images, while the USB 3.0 or Ethernet interface can meet the data transfer requirements of about 180MB / s, preventing latency caused by data caching. In a specific embodiment of the present invention, the structured light camera is suitable for high-precision acquisition at close range (0.5m-2m), while the TOF (Time of Flight) camera has stronger resistance to ambient light interference and can be flexibly selected according to the complexity of the lighting at the competition venue. For example, the TOF camera is preferred for competition venues with strong lighting to avoid reflections causing flying spots (incorrect depth values) in the depth data.

[0053] In a specific embodiment of the present invention, taking a dual-camera diagonal deployment scheme as an example, camera 1 can be deployed at a 45° position to the left of the white side of the chessboard, and camera 2 can be deployed at a 45° position to the right of the black side. Both cameras are 1.2-1.5m from the edge of the chessboard, and the lens tilt angle is controlled between 30° and 45°. The projection distance (baseline) between the two cameras on the horizontal plane needs to be maintained at 1.5-2m. If the baseline is too short, it will reduce the depth resolution of the 3D reconstruction (the depth value differentiation between different points will decrease), and if it is too long, it will increase the difficulty of synchronization control. At the same time, the deployment environment should avoid direct sunlight. A polarizing filter can be installed in front of the camera lens to prevent overexposure of RGB images. The cameras need to be stabilized by a tripod or fixed support to avoid shaking that causes depth data offset (shaking may cause the depth value error of the same pixel to be >5mm, which directly affects subsequent contact determination).

[0054] Simultaneously, this invention can send periodic trigger pulses (frequency consistent with the frame rate, 30fps corresponds to a 33.3ms period) to all cameras via an external synchronizer (such as an Intel RealSense Sync Module). The cameras only start acquiring data upon receiving the trigger signal, and the synchronization error can be controlled within 1ms, far below the allowable threshold of 10ms. If the cameras lack a hardware trigger interface, software synchronization is used for auxiliary calibration. All cameras can connect to the same host and achieve system time synchronization via the NTP protocol (error ≤ 1ms). A microsecond-level timestamp is recorded during each image acquisition, and the host subsequently matches synchronized image pairs from different cameras based on a timestamp difference < 1ms. To verify synchronization effectiveness, an LED flashing light (10Hz frequency) can be placed in the center of the chessboard, and the timestamps of the flashing frames captured by each camera can be extracted. If the time difference for 100 consecutive frames is < 1ms, the synchronization is considered successful; otherwise, recalibration is required.

[0055] In this invention, the image sequence acquired in step S1 is a structured set with spatiotemporal and viewpoint correlations. For any time t (corresponding to a timestamp), the image sequence contains RGB and depth images from K cameras (corresponding to K viewpoints), and the corresponding data structure is as follows (K is the total number of cameras, k∈{1,2,...,K} is the camera number):

[0056] Single-view data unit: ;

[0057] In the formula, Let be the RGB image (24-bit color depth, BMP / PNG format) of the kth camera at time t, where k∈{1,2,...,K} is the camera number (corresponding to a unique viewpoint); This is the depth image of the k-th camera at time t (16-bit unsigned integer, in mm, in RAW / PNG format). The timestamp (accurate to microseconds) for the kth camera is captured for this frame. The intrinsic parameters (focal length) of the k-th camera , Main point , ) and extrinsic parameters (rotation matrix) Translation vector This is used for subsequent conversion of pixel coordinates to 3D coordinates.

[0058] Multi-view synchronized data set: That is, the set of synchronized data units of all perspectives at time t.

[0059] S2. Perform 3D reconstruction processing on the RGB image sequence and the depth image sequence from different perspectives at the same time to generate a 3D point cloud model of the hand and a 3D point cloud model corresponding to each chess piece at that time, which are denoted as hand point cloud data. Chess Piece Point Cloud Dataset (N is the total number of pieces on the board), where each piece has a unique and persistent identity.

[0060] This step transforms the multi-view synchronized RGB and depth images acquired in step S1 into unobstructed, high-precision 3D point cloud models of the hand and chess piece in a unified world coordinate system. This solves the problem that single-view 2D images cannot describe 3D spatial contact relationships. For example, in a 2D image, the hand and chess piece might be misjudged as being in contact due to overlapping projections, while 3D point clouds can distinguish between suspension and contact using real spatial coordinates. Simultaneously, multi-view fusion can fill in occlusion areas from a single viewpoint. For instance, when a hand occludes a part of a chess piece, the point cloud from another viewpoint can completely capture that area, providing a reliable spatial geometric basis for determining unauthorized touches.

[0061] Specifically, in step S2, generating the 3D point cloud model of the hand and the 3D point cloud models of each chess piece at that moment includes:

[0062] S21. Perform hand segmentation and chess piece instance segmentation on the RGB image of each viewpoint to obtain a two-dimensional hand region mask and an independent region mask for each chess piece. The hand segmentation and chess piece segmentation are implemented using a deep learning-based semantic segmentation model.

[0063] This invention uses semantic segmentation to accurately extract pixel-level regions of the hand and chess pieces from RGB images from each viewpoint, generating hand region masks and chess piece region masks to distinguish the target (hand / chess piece) from the background.

[0064] Due to the characteristics of chess scenes, such as densely packed pieces, dynamic hand movements, and complex backgrounds (tabletop, clothing), deep learning semantic segmentation models adapted to small targets and highly overlapping scenes are required. Mask R-CNN or U-Net++ are commonly used. Mask R-CNN can simultaneously achieve object detection and pixel-level segmentation, accurately locating each piece (small target) and hand (dynamic target), with an output mask edge accuracy of ±1 pixel, and can distinguish adjacent pieces, such as the pieces on the center d4 and e4 squares of the chessboard. U-Net++, on the other hand, has stronger generalization ability on small sample data, chess competition scene data is readily available, and the segmentation speed is fast (single frame processing time ≤30ms, meeting the 30fps real-time requirement), making it suitable for real-time segmentation of dynamic hands.

[0065] To ensure segmentation accuracy, model training requires constructing a dataset (≥5000 images) containing different lighting conditions, hand skin color, and chess piece wear states. Cross-entropy loss and Dice loss can be used to address class imbalance (where background pixels account for a high proportion of images, while hand / chess piece pixels account for a low proportion). Furthermore, the original mask output by the model needs to be optimized through morphological operations, specifically:

[0066] First, dilation (kernel size 3×3) is performed to fill the tiny holes inside the mask; then, erosion (same kernel size) is performed to eliminate edge burrs and ensure smooth mask edges; finally, a minimum target area threshold (hand mask area ≥ 500 pixels, chess piece mask area ≥ 100 pixels) is used to filter out false target masks caused by noise, such as missegmentation of small areas due to light reflection; ultimately, two binary masks (single channel, pixel value 0 = background, 1 = target) are output for each viewpoint's RGB image, specifically for the hand region. and chess piece area mask The hand region mask covers all pixels of the hand (including fingers and palm), and the deviation between the edge and the actual hand contour is ≤2 pixels. In the chess piece region mask, each chess piece corresponds to an independent connected region, with no adjacent chess piece masks sticking together and no chess piece pixels missing.

[0067] S22. Register the hand region mask and the independent region mask of the chess piece from each viewpoint with the depth image of the corresponding viewpoint to obtain the three-dimensional point cloud fragment of the hand and the three-dimensional point cloud fragment of each chess piece under that viewpoint.

[0068] This step involves pixel-level registration of the 2D target mask with the depth image acquired in step S1 at the same viewpoint and time, extracting the depth data of the target region and converting it into a single-view 3D point cloud fragment. Registration, in this context, achieves spatial location correspondence. The specific process includes the following steps:

[0069] Using the pre-calibrated camera intrinsic parameters from step S1 (obtainable via Zhang Zhengyou's checkerboard calibration method), a mapping relationship between pixel coordinates and the three-dimensional coordinates of the camera coordinate system is established. The intrinsic parameter matrix... The definition is as follows:

[0070] First, the pixel coordinates (x, y) are obtained through the intrinsic parameter matrix. The inverse operation converts the coordinates to normalized coordinates; then it is multiplied by the effective depth value corresponding to the pixel (filtering out invalid values ​​that are outside the 0.5-3m measurement range or are 0) to obtain the three-dimensional coordinates in the camera coordinate system.

[0071] Intrinsic parameter matrix The expression is:

[0072] ;

[0073] In the formula, , This refers to the focal length (in pixels) of the k-th camera in the x / y directions, determined by the camera hardware, such as that of the Intel RealSense D455. ≈600 pixels; , Here are the coordinates (in pixels) of the principal point of the k-th camera, which is the intersection of the camera's optical axis and the imaging plane, usually the image center, such as in a 1920×1080 image. ≈960, ≈540.

[0074] For each camera k and time t, calculate a 3D point cloud segment of the target; specifically, first convert the 2D pixel coordinates... The formula for converting to normalized (dimensionless) coordinates in the camera coordinate system is:

[0075] ;

[0076] In the formula, For normalized coordinates, It is the inverse of the intrinsic parameter matrix.

[0077] Then only the depth values ​​corresponding to pixels with a mask of 1 are retained. It also filters out invalid depths, such as depth values ​​of 0 or those exceeding the camera's measurement range by 0.5-3m.

[0078] The pixel is obtained by combining the normalized coordinates with the effective depth value. 3D coordinates in the camera coordinate system (Unit: mm):

[0079] ;

[0080] In the formula, , The horizontal / vertical coordinates in the camera coordinate system (x-axis to the right, y-axis down); The depth coordinates are in the camera coordinate system (the z-axis is forward along the camera's optical axis, i.e., the distance from the camera).

[0081] After the 3D coordinates are calculated, the 3D point cloud fragments of the hand are obtained by classifying them according to the mask type. And 3D point cloud fragments of each chess piece ( Let p be the chess piece number, p∈{1,2,...,N}, where N is the total number of chess pieces on the board. By all satisfying of Composition, that is ; By all satisfying of Composition, that is , This is the independent region mask for the p-th piece under the k-th camera.

[0082] S23, The hand 3D point cloud fragment fused from all perspectives. The process generates unified hand point cloud data; and for each piece, it integrates 3D point cloud fragments from all perspectives to generate unified point cloud data with an identity identifier for that piece. The point cloud data of all pieces constitute the piece point cloud dataset.

[0083] This step involves capturing hand point cloud fragments from all perspectives. Chess Piece Dot Cloud Fragment The data is converted to a unified world coordinate system and deduplicated to generate complete, unobstructed, and non-redundant hand point cloud data. Chess Piece Point Cloud Data ,in Let p be the unified 3D point cloud of the p-th piece, where p∈{1,2,...,N} is the piece number.

[0084] Each piece's identity employs a dual binding mechanism of initial position and feature matching to ensure its uniqueness throughout the movement process. Specifically, at the initial moment of the game (before the piece moves), a unique ID is assigned to each piece, such as White Pawn-d4 and Black Queen-e8, and its initial point cloud shape features, such as volume and surface texture point distribution, are recorded. In subsequent frames, the point cloud data of the same piece is continuously associated through inter-frame point cloud feature matching and centroid trajectory tracking, avoiding identity confusion caused by piece movement, rotation, or partial occlusion. This identity identifier is used throughout the entire acquisition and judgment process, providing a core index for locating touched pieces and associating moved / disappeared pieces in S3.

[0085] It should be noted that, to ensure consistency across different game scenarios, the world coordinate system must be fixed on the chessboard plane. The origin of the world coordinate system is set as the geometric center of the chessboard, i.e., the intersection of d4 and e4, and d5 and e5. The X-axis is along the horizontal direction of the chessboard (from the White player's perspective, the rightward direction is positive); the Y-axis is along the vertical direction of the chessboard (from the White player's perspective, the forward direction is positive); and the Z-axis is perpendicular to the chessboard plane and upward (perpendicular to the tabletop, with the height direction being positive).

[0086] When performing coordinate system transformation based on camera extrinsic parameters, the single-view point cloud fragment is first transformed from the camera coordinate system to the world coordinate system using the camera extrinsic parameter matrix; the transformation formula is:

[0087] ;

[0088] In the formula, These are three-dimensional coordinates in the world coordinate system, corresponding to the X / Y / Z components of the world coordinate system, respectively.

[0089] Let be the extrinsic parameter matrix of the camera, and its formula is:

[0090] ;

[0091] In the formula, It is a 3×3 rotation matrix that describes the orientation of the camera coordinate system relative to the world coordinate system. For example, a camera tilt angle of 30° corresponds to the Z-axis component of the rotation matrix. It is a 3×1 translation vector describing the position of the camera's optical center in the world coordinate system, such as X=1200mm, Y=800mm, Z=1500mm in the world coordinate system.

[0092] After converting the multi-view point clouds to the world coordinate system, point cloud fusion and optimization are then performed, with distance thresholds set. For all viewpoints of the hand (or multiple viewpoints of the same piece), if the Euclidean distance between two points is < If a point is repeatedly measured at the same spatial point, only one of the points is retained, typically the point with the smallest depth variance, to avoid redundant data. A statistical filtering algorithm is used to calculate the average distance of each point's neighborhood (e.g., the 20 nearest neighbors). If the deviation of a point from the average distance is greater than twice the standard deviation, it is considered a noise point and removed, such as a flying point with a depth measurement error. If a region only exists in some views, such as the bottom of a chess piece obscured by a hand, missing points are filled in using neighborhood interpolation, such as fitting points in the missing region based on the shape of adjacent chess pieces, ensuring the integrity of the chess piece point cloud.

[0093] After point cloud fusion and optimization, the final hand point cloud data at time t is generated. Chess Piece Point Cloud Dataset Among them, hand point cloud data Covering all areas of the hand (fingers, palm, wrist) with no obvious gaps, such as fingertip dot cloud density ≥ 5 points / mm. 2 Point cloud coordinate error ≤ 2mm, no noise points; each chess piece point cloud in the chess piece point cloud dataset It is an independent connected region with no adjacent pieces attached (attachment distance ≥ 3mm), the centroid calculation error of the piece point cloud is ≤ 1mm, and the identity identifier corresponds one-to-one with the piece entity.

[0094] S3. Based on the hand point cloud data and the chess piece point cloud dataset, a violation determination is made, specifically including the following sub-steps:

[0095] S31. Contact target identification: Calculate the minimum spatial distance between the hand point cloud data and each chess piece point cloud data in the chess piece point cloud dataset; if the minimum spatial distance of a certain chess piece is less than the preset contact distance threshold, then it is determined that contact has occurred, and the identity identifier of the chess piece is recorded as the ID of the touched chess piece.

[0096] The core objective of this step is to accurately locate the piece touched by the player, providing a clear target for subsequent action determination. By traversing the spatial distance between the hand point cloud and the point clouds of all pieces, the problem of misjudging the contact object caused by the dense arrangement of multiple pieces is avoided.

[0097] In step S31, the minimum spatial distance between the hand point cloud data and the point cloud data of a certain chess piece is calculated. Specifically, the Euclidean distance between each point in the hand point cloud data and each point in the chess piece point cloud data is calculated, and the minimum value is selected from all the calculated Euclidean distances as the minimum spatial distance.

[0098] Hand dot cloud (m is the number of hand points), where the world coordinates of the i-th hand point are: The p-th piece's dot cloud ( (where p is the number of point cloud data points for the p-th piece), where p is the number of point cloud data points for the p-th piece. The world coordinates of each chess piece are: ( (where p is the point index of the p-th piece), then the Euclidean distance between the two points is calculated as follows:

[0099] ;

[0100] is the hand point number, and m is the total number of hand point clouds; The unit is millimeters (mm), which is consistent with the unit of point cloud coordinates.

[0101] Iterate through the distances between all points on the hand and all points on the piece, and take the minimum value as the minimum spatial distance between the hand and the piece. The corresponding formula is:

[0102] ;

[0103] Repeat the above calculation for all pieces in the chess piece point cloud dataset to obtain the minimum distance between each piece and the hand. .

[0104] This invention introduces a contact distance threshold. (The value range can be set to 1-5mm, the default is 3mm), if there exists a certain piece p ( )satisfy If a hand makes contact with a piece, it is determined that the hand has made contact with that piece, and its identity is recorded as the ID of the piece that was touched, such as White Pawn-d4; if all pieces If no contact is found, the subsequent steps will not proceed.

[0105] Among them, the contact distance threshold The setting is based on the fact that chess pieces have tiny bumps and depressions on their surface (such as the texture of wooden chess pieces), and the actual physical distance of touch is usually ≤3mm. This threshold covers both the slight touch scenario and is compatible with point cloud measurement errors (≤2mm).

[0106] S32. Related Behavior Capture and Judgment: After determining that contact has occurred, continuously monitor the chess piece point cloud dataset. When a chessboard state change event is detected, immediately perform a compliance judgment. The chessboard state change event includes chess piece movement events and chess piece disappearance events.

[0107] The core of this step is the verification of the correlation between the touch action and subsequent actions, that is, judging whether the player's behavior after touching a piece is compliant according to the chess "touch-move" and "capture" rules. Chessboard state change events mainly include "piece movement events" (piece position changes) and "piece disappearance events" (pieces are removed from the chessboard, usually in capture scenarios). The monitoring and judgment of these two types of events are as follows:

[0108] If a piece movement event is detected, the identity identifiers of all pieces that have moved are recorded to form a set of moving piece IDs; it is then determined whether the touched piece ID is included in the set of moving piece IDs. If not, a response target inconsistency violation event is generated.

[0109] The core of monitoring chess piece movement events is the tracking of the chess piece's centroid displacement. Specifically, for each chess piece in the chess piece point cloud dataset, the centroid coordinates of its point cloud are continuously compared at the contact determination time (t0) and the subsequent frame time (t≥t0). If the magnitude of a chess piece's centroid displacement vector is greater than or equal to a preset displacement threshold, it is determined that the chess piece has moved, and its identity is added to the set of moving chess piece IDs.

[0110] The specific calculation formula is as follows:

[0111] The centroid coordinates of the piece at contact time t0 are:

[0112] ;

[0113] The centroid coordinates of the piece at time t in subsequent frames:

[0114] ;

[0115] Magnitude of the centroid displacement vector:

[0116] ;

[0117] Movement determination rule: If If the piece moves, then the move is determined. This is the displacement threshold, with a value range of 0.5-2mm and a default value of 1mm.

[0118] Displacement threshold The setting is based on the following: slight vibrations of the tabletop (≤0.8mm) and point cloud measurement errors (≤0.5mm) need to be filtered out, while ensuring that the effective movement of the chess piece away from the original square is accurately captured (the side length of the chess square is about 57mm, and a 1mm displacement is far from meeting the "leaving the square" standard, and is only used as a quantitative basis for "whether it moves").

[0119] Compliance judgment logic: Set of moving chess piece IDs ( (where r is the number of pieces to move, r≤N), and the ID of the piece being touched is... This indicates that the player touched the ID. The corresponding piece was moved, but instead of moving that piece, another piece was moved, violating the "touch-move" rule and generating a response target inconsistency violation event; if If the behavior is deemed compliant, no violation event is generated.

[0120] If a piece disappearance event is detected, the identity identifiers of all disappeared pieces are recorded to form a set of disappeared piece IDs; it is determined whether the piece identified by the touched piece ID is an opponent's piece, and at the same time, it is determined whether the touched piece ID is included in the set of disappeared piece IDs; if the piece identified by the touched piece ID is an opponent's piece and the touched piece ID is not included in the set of disappeared piece IDs, then a target non-elimination violation event is generated.

[0121] The monitoring logic for chess piece disappearance events is as follows: For each chess piece in the chess piece point cloud dataset, if no valid point cloud data of the piece is detected within 3 consecutive frames (approximately 100ms) (or the number of point clouds is ≤50, far lower than the normal number of chess piece point clouds), then the piece is determined to have disappeared, and its identity is added to the disappeared chess piece ID set. ( (Number of pieces that disappeared). The disappearance event usually corresponds to the "capture" scenario, that is, after one's own piece captures an opponent's piece, the opponent's piece is removed from the board.

[0122] Compliance assessment requires meeting two conditions: ① The ID of the touched piece The corresponding piece is the opponent's piece, which can be determined by the "ownership" information in the initial identity identifier, such as "Black Pawn-e7" being the opponent's piece for the White player; ② The ID of the touched piece If both conditions are met, it means the player touched the opponent's piece (which should be captured according to the rules), but the piece did not disappear (i.e., it was not captured), violating the capture requirement in the "touch-move" rule, resulting in a target not being eliminated violation event; if (The opponent's piece is captured) or If a piece is on your side (without the obligation to capture other pieces), then the action is considered compliant, and no violation event is generated.

[0123] S4. Output the alarm information for the generated violation event. The alarm information includes the type of violation event, the timestamp of the occurrence, the ID of the touched piece, and a 3D visualization screenshot of the scene generated based on the hand point cloud data and the piece point cloud data.

[0124] In this embodiment of the invention, the timestamp is the core index for tracing violations, synchronizing the competition process, and associating original data. Its design must meet the requirements of high precision, associativity, and ease of reading. Specifically, the timestamp is based on the trigger time of the violation judgment to ensure precise matching with the event's timing; the precision must reach the millisecond level, such as 2024-05-20 14:35:22.789, ensuring accurate association with the original image sequence collected in step S1, facilitating referees' subsequent retrieval of multi-view RGB / depth images for verification at the corresponding time; simultaneously, it must be synchronized with the official chess clock time (error ≤ 10ms) to avoid timing chaos caused by the disconnect between the alarm time and the chess clock time. Furthermore, the timestamp can also support chronological sorting (arranging multiple violations in a short period in chronological order) and historical backtracking, storing violation records for nearly 24 hours for easy review and retrieval.

[0125] The 3D visualization scene screenshot allows referees to quickly understand the violation scene. It is based on the generation of 3D point clouds of the hand and the violating piece in step S2. Differentiated rendering ensures visual differentiation. The hand point cloud can be semi-transparent red (without obscuring the piece), the touched piece is solid blue, and the surface is marked with identity marks, such as "Black Pawn-e7". Moved / disappeared pieces are highlighted in yellow, and the background is weakened with light gray (to avoid interference). The default output is a "dual-view composite image". The top view shows the planar positional relationship between the hand and the piece on the chessboard (such as the positional difference between the touched piece and the moving piece), and the side view shows the contact state between the hand and the touched piece (such as the contact distance marking), which solves the limitations of a single view.

[0126] In addition, the screenshots are overlaid with quantitative annotations: the minimum contact point between the hand and the touched piece is marked with a yellow cross, with the annotation "Contact distance: 2.3mm"; if it is a piece movement event, the centroid displacement of the touched piece and the moving piece is marked, such as the displacement of the touched piece: 0.4mm, and the displacement of the moving piece: 15.2mm; if it is a piece disappearance event, the point cloud state of the touched piece is marked, such as the number of point clouds of the touched piece: 32 (disappearance judgment), allowing the referee to obtain the core judgment basis without understanding the technical principles.

[0127] In a specific embodiment of the present invention, the screenshot can support interactive operations: it can be rotated around the axis to view hidden perspectives, zoomed in 1-10 times to focus on the contact point, and the quality of the original point cloud map can be switched to meet the verification needs in different scenarios.

[0128] To improve the basis for judgment, the alarm information is supplemented with three types of auxiliary data: ① Details of the object associated with the violation: indicating the owner, type, and initial position of the touched piece, such as "Opponent's piece: Black pawn, initial position e7"); ② Judgment rule prompt: directly indicating the corresponding chess clause, such as "According to FIDE rule 4.3: Touching an opponent's piece should result in capturing that piece; failure to capture it is a violation"; ③ Confidence score: calculated based on the point cloud quality (density, noise ratio) and the deviation of the judgment data (difference between contact distance and threshold, stability of displacement / disappearance state) (e.g., 99.2%). High confidence (≥95%) directly prompts a penalty, medium confidence (80%-95%) suggests review, and low confidence (<80%) is marked as a suspected false alarm and requires manual confirmation.

[0129] It should be noted that the output alert information does not replace the referee's judgment, but rather assists the referee in making more accurate decisions by providing objective evidence and reducing the cost of information acquisition. For example, if a player touches the opponent's black pawn -e7 but does not capture it, the system detects that "the piece disappearance event did not occur" and generates a "target not eliminated violation event." After reviewing the 3D screenshot and contact distance data, the referee can quickly make a judgment according to the rules. If the player denies the touch, the referee can retrieve the multi-view RGB image and depth data at the corresponding time using the timestamp to present objective evidence to the player and resolve the dispute.

[0130] Meanwhile, after the game, the referee team can retrieve all the alert information of violations through the cloud system, review the accuracy of the judgment by combining 3D screenshots with the original data, and use typical violation cases as training materials to improve their officiating ability.

[0131] In one embodiment of the present invention, in step S32, if a piece movement event is detected, and the ID of the touched piece is a king, and the set of moving piece IDs includes both kings and rooks, then a compliance determination of the castling process is further performed:

[0132] S81. Timing compliance judgment: Determine whether the order of the king and rook piece movement events conforms to the preset rule of moving the king first and then the rook.

[0133] The core objective of this step is to verify whether the movement order of the King and Rook conforms to the "King before Rook" rule. The criterion is the trigger time of the centroid displacement of the piece's point cloud; the trigger time of the King's movement must be recorded. (That is, the magnitude of the king's centroid displacement vector is first greater than or equal to the displacement threshold) (the moment) and the moment the car moves (triggered by the movement) Set a time difference threshold. This threshold is set based on the normal operating rhythm of chess players, covering the reasonable time for moving the queen first, transferring the hand to the rook and moving it, while also distinguishing the order of deliberate violations. The timing rules must meet the following requirements. and ,like or If so, the timing is deemed non-compliant.

[0134] S82. Interaction Compliance Judgment: Determine whether the King's piece movement event and the Rook's piece movement event are triggered by hand operations associated with the same hand point cloud data.

[0135] The interaction compliance check aims to verify whether the movement of the King and Rook was triggered by the same hand action of the same player, preventing collusion between two players to violate the rules or accidentally touch other pieces. The determination is based on the identity association of hand point clouds; by analyzing hand shape features and the location of the operation area, a unique temporary ID is assigned to each hand. Record the hand ID that triggered the king's movement. (That is, at the moment Wang moves, the minimum distance to Wang's point cloud is less than the contact threshold) (Hand ID), and the hand ID that triggered the car to move. ,like If so, it is determined that the operation was performed by different hands and the interaction is non-compliant.

[0136] S83. If step S81 determines that the timing is non-compliant, or step S82 determines that the operation is not performed by the same hand, then based on the non-compliance event of the response target, a sub-type violation event marked as transposition process violation is generated, and the timing compliance judgment result and the interaction compliance judgment result are incorporated into the alarm information.

[0137] If S81 or S82 determines non-compliance, a new subclass label "Transposition Process Violation" should be added to the existing response target inconsistency violation event to clarify the violation type. The alert information needs to be supplemented with key data such as the timing judgment results (e.g., "The rook moved before the king, timing violation"), the interaction judgment results (e.g., "The king and rook's movements were triggered by different hand operations, interaction violation"), the movement times of the king and rook, and hand IDs. This provides the referee with accurate evidence of the violation and helps them quickly understand the violation scenario and judgment logic.

[0138] In one embodiment of the present invention, before step S21, a step S20 is further included to perform anti-interference repair on the depth image sequence:

[0139] S201. Simultaneously acquire a polarization image sequence that is temporally and spatially aligned with the RGB image sequence and the depth image sequence using the RGB-D camera.

[0140] Depth images are susceptible to interference from ambient light (such as reflections from stadium lights or strong sunlight from outside windows), the surface material of the chess pieces (such as the specular reflection of smooth plastic chess pieces), and local occlusion, resulting in "flying points" (incorrect depth values) or "holes" (missing depth values). This leads to a decrease in the accuracy of 3D point cloud registration, which in turn affects the accuracy of contact distance calculation and violation judgment. By using polarized images to assist in depth restoration and utilizing polarization features to characterize the surface reflection properties of objects, effective depth information can be accurately distinguished from interfering noise, significantly improving the robustness of depth data.

[0141] Specifically, RGB-D cameras must be equipped with polarization sensors (such as single-chip polarization cameras) to support the synchronous acquisition of polarization images (including four polarization directions: 0°, 45°, 90°, and 135°). The polarization image sequence must be spatiotemporally synchronized with the RGB image sequence and the depth image sequence, with a temporal synchronization error ≤10ms. Spatial synchronization is achieved through camera calibration (external parameter calibration of the polarization sensor and the RGB / depth sensor, with a reprojection error ≤0.5 pixels). In terms of image parameters, the resolution of the polarization image must be consistent with that of the depth image (≥640×480) to ensure feature matching accuracy and lay the data foundation for subsequent polarization feature map generation.

[0142] S202. Based on the polarization image sequence, calculate and generate a polarization feature map characterizing the reflection properties of the object surface.

[0143] Polarization feature map generation is based on polarization images in four directions. Core features characterizing the reflectivity of an object's surface are calculated to generate a single-channel polarization feature map. The feature calculation method is as follows:

[0144] polarization degree ,in, , These represent the maximum and minimum grayscale values ​​for the four polarization directions of the pixel, respectively. A higher degree of polarization indicates stronger surface reflection.

[0145] Azimuth It represents the polarization direction of the reflected light.

[0146] Polarization degree P and azimuth angle Normalized to [0, 255], a polarization feature map is generated by weighted summation (weight ratio 3:2). Highlighting the distinctive differences in areas with high reflectivity interference.

[0147] S203. For the same acquisition time, extract one frame of RGB image and one frame of depth image from the RGB image sequence and the depth image sequence respectively, fuse the RGB image, the depth image and the polarization feature map, and input them into the pre-trained deep repair neural network.

[0148] Extract RGB images at the same time t. (3-channel) Original depth image (1 channel) Polarization feature map (1 channel), concatenated into a 5-channel input tensor according to the channel dimension. The model chosen is the pre-trained U-Net++ deep inpainting neural network, which excels at inpainting small missing / noise areas and boasts fast inference speed (single frame processing time ≤20ms), meeting real-time requirements. During training, the dataset must contain RGB-D+ polarization image pairs under different lighting and reflection conditions, labeled with true depth values, and employ the MSE loss function (minimizing the error between the inpainted depth and the true depth). After training, the inpainting error must be ≤0.8mm to ensure the inpainting effect meets the accuracy requirements of subsequent registration.

[0149] S204. The deep repair neural network outputs a repaired depth image, which is used to replace the original depth image and participate in the registration process in step S22.

[0150] The deep inpainting neural network outputs the inpainted depth image. The original depth image is automatically replaced. The repaired depth image must meet the quality standards of flying point ratio ≤ 0.3%, hole filling rate ≥ 95%, and depth error ≤ 1mm (at a distance of 1m). If it does not meet the standards, a secondary repair is triggered (re-input the model and adjust the fusion weights).

[0151] The repaired depth image is used for mask registration in step S22 to generate a high-precision 3D point cloud fragment, ensuring the accuracy of contact distance calculation and violation judgment.

[0152] According to the chess refereeing method of this invention, the invention deploys multi-view RGB-D cameras to synchronously acquire image sequences and generates three-dimensional point cloud models (including unique identifiers) of hands and chess pieces based on three-dimensional reconstruction and image detection technology. This enables real-time, automatic, and accurate detection of chess violations. Contact is determined by calculating the minimum spatial distance between point clouds, and compliance of behavior is verified by combining chessboard state change events (movement / disappearance), ensuring the accuracy of the judgment. By utilizing three-dimensional spatial geometric information, the invention overcomes the visual limitations of a single viewpoint and the shortcomings of two-dimensional image analysis, effectively solving the problems of missed and misjudged judgments in scenes with dense chess pieces and occlusion, improving refereeing efficiency and fairness, and realizing automated refereeing.

[0153] Corresponding to the chess referee assistance method in the above embodiments, the present invention also proposes a chess referee assistance system.

[0154] like Figure 3 As shown in the figure, an embodiment of the present invention provides a chess referee assistance system, which includes: a data acquisition module, a three-dimensional reconstruction and identification module, a violation determination module, and an output module.

[0155] The data acquisition module is used to simultaneously acquire a multi-view image sequence containing the chessboard and the player's operating area using at least two RGB-D cameras set in different spatial locations. The image sequence includes a synchronized RGB image sequence and a depth image sequence.

[0156] The 3D reconstruction and identification module is used to perform 3D reconstruction processing on the RGB image sequence and the depth image sequence from different perspectives at the same time, generating a 3D point cloud model of the hand and a 3D point cloud model corresponding to each chess piece at that time, which are respectively denoted as hand point cloud data and chess piece point cloud dataset, wherein each chess piece has a unique and persistent identity identifier.

[0157] The violation determination module is used to determine violations based on the hand point cloud data and the chess piece point cloud dataset, and specifically includes a contact target recognition unit and an associated behavior capture and determination unit.

[0158] The contact target recognition unit is used to calculate the minimum spatial distance between the hand point cloud data and each chess piece point cloud data in the chess piece point cloud dataset, and to determine whether there is a chess piece whose minimum spatial distance is less than a preset contact distance threshold. If there is, the identity of the chess piece is recorded as the ID of the touched chess piece.

[0159] The associated behavior capture and judgment unit is used to continuously monitor the chess piece point cloud dataset after a contact is detected, and immediately perform compliance judgment when a chessboard state change event is detected. Specifically: if a chess piece movement event is detected, the identity identifiers of all moving chess pieces are recorded to form a set of moving chess piece IDs; it is determined whether the ID of the touched chess piece is included in the set of moving chess piece IDs. If it is not included, a response target inconsistency violation event is generated; if a chess piece disappearance event is detected, the identity identifiers of all disappearing chess pieces are recorded to form a set of disappearing chess piece IDs; it is determined whether the chess piece identified by the touched chess piece ID is an opponent's chess piece, and simultaneously, it is determined whether the touched chess piece ID is included in the set of disappearing chess piece IDs; if the chess piece identified by the touched chess piece ID is an opponent's chess piece and the touched chess piece ID is not included in the set of disappearing chess piece IDs, a target not eliminated violation event is generated.

[0160] The output module is used to output alarm information for the generated violation events.

[0161] According to an embodiment of the chess refereeing system of the present invention, the present invention deploys multi-view RGB-D cameras to synchronously acquire image sequences, and generates three-dimensional point cloud models (including unique identifiers) of hands and chess pieces based on three-dimensional reconstruction and image detection technology, thereby realizing real-time, automatic, and accurate detection of three types of chess violations; it determines contact by calculating the minimum spatial distance between point clouds, and verifies the compliance of behavior by combining chessboard state change events (movement / disappearance), ensuring the accuracy of the judgment; by utilizing three-dimensional spatial geometric information, it overcomes the visual limitations of a single perspective and the shortcomings of two-dimensional image analysis, effectively solving the problems of missed and misjudged judgments in scenes with dense chess pieces and occlusion, improving refereeing efficiency and fairness of the game, and realizing automated refereeing.

[0162] Although embodiments of the present invention have been described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for assisting referees in chess, characterized in that, Includes the following steps: S1. Simultaneously acquire a multi-view image sequence containing the chessboard and the player's operating area using at least two RGB-D cameras set at different spatial locations. The image sequence includes a synchronized RGB image sequence and a depth image sequence. S2. Perform three-dimensional reconstruction processing on the RGB image sequence and the depth image sequence from different perspectives at the same time to generate a three-dimensional point cloud model of the hand and a three-dimensional point cloud model corresponding to each chess piece at that time, which are respectively denoted as hand point cloud data and chess piece point cloud dataset, wherein each chess piece has a unique and continuous identity identifier. S3. Based on the hand point cloud data and the chess piece point cloud dataset, a violation determination is made, specifically including the following sub-steps: S31. Contact Target Recognition: Calculate the minimum spatial distance between the hand point cloud data and each chess piece point cloud data in the chess piece point cloud dataset; if the minimum spatial distance of a certain chess piece is less than the preset contact distance threshold, it is determined that contact has occurred, and the identity identifier of the chess piece is recorded as the ID of the touched chess piece. S32. Related Behavior Capture and Judgment: After determining that contact has occurred, continuously monitor the chess piece point cloud dataset. When a chessboard state change event is detected, immediately perform a compliance judgment. The chessboard state change event includes chess piece movement events and chess piece disappearance events, wherein: If a piece movement event is detected, record the identity of all pieces that have moved to form a set of moving piece IDs; determine whether the touched piece ID is included in the set of moving piece IDs; if not, generate a response target inconsistency violation event. If a piece disappearance event is detected, the identity identifiers of all disappeared pieces are recorded to form a set of disappeared piece IDs; it is determined whether the piece identified by the touched piece ID is an opponent's piece, and at the same time, it is determined whether the touched piece ID is included in the set of disappeared piece IDs; if the piece identified by the touched piece ID is an opponent's piece and the touched piece ID is not included in the set of disappeared piece IDs, then a target non-elimination violation event is generated. S4. Output the alarm information for the generated violation events.

2. The chess auxiliary referee method according to claim 1, characterized in that, In step S31, the minimum spatial distance between the hand point cloud data and the point cloud data of a certain chess piece is calculated. Specifically, the Euclidean distance between each point in the hand point cloud data and each point in the chess piece point cloud data is calculated, and the minimum value is selected from all the calculated Euclidean distances as the minimum spatial distance.

3. The chess auxiliary referee method according to claim 2, characterized in that, In step S321, the detection of movement events and recording of the identity identifiers of all pieces that have moved specifically includes: For each piece in the chess piece point cloud dataset, the centroid coordinates of its point cloud are continuously compared at the moment of contact determination and at subsequent moments; if the magnitude of the centroid displacement vector of a piece is greater than or equal to a preset displacement threshold, it is determined that the piece has moved, and its identity is added to the set of moving piece IDs.

4. The chess auxiliary referee method according to claim 3, characterized in that, In step S2, generating the 3D point cloud model of the hand and the 3D point cloud models of each chess piece at that moment specifically includes: S21. Perform hand segmentation and chess piece instance segmentation on the RGB image of each viewpoint to obtain a two-dimensional hand region mask and an independent region mask for each chess piece. S22. Register the hand region mask and the independent region mask of the chess piece in each viewpoint with the depth image of the corresponding viewpoint to obtain the three-dimensional point cloud fragment of the hand and the three-dimensional point cloud fragment of each chess piece in that viewpoint. S23. Merge the three-dimensional point cloud fragments of the hand from all perspectives to generate unified hand point cloud data; and for each piece, merge the three-dimensional point cloud fragments from all perspectives to generate unified point cloud data with an identity identifier for that piece. The point cloud data of all pieces constitute the piece point cloud dataset.

5. The chess auxiliary referee method according to claim 4, characterized in that, In step S21, the hand segmentation and chess piece segmentation are implemented using a deep learning-based semantic segmentation model.

6. The chess auxiliary referee method according to claim 5, characterized in that, In step S4, the alarm information includes the type of violation event, the timestamp of the occurrence, the ID of the touched piece, and a 3D visualization scene screenshot generated based on the hand point cloud data and the piece point cloud data.

7. The chess auxiliary referee method according to claim 3, characterized in that, In step S32, if a piece movement event is detected, and the ID of the touched piece is a king, and the set of moving piece IDs includes both a king and a rook, then a compliance determination of the castling process is further performed: S81. Timing compliance judgment: Determine whether the order of the king and rook piece movement events conforms to the preset rule of moving the king first and then the rook. S82. Interaction compliance judgment: Determine whether the king's piece movement event and the rook's piece movement event are triggered by hand operations associated with the same hand point cloud data; S83. If step S81 determines that the timing is non-compliant, or step S82 determines that the operation is not performed by the same hand, then based on the non-compliance event of the response target, a sub-type violation event marked as transposition process violation is generated, and the timing compliance judgment result and the interaction compliance judgment result are incorporated into the alarm information.

8. The chess auxiliary referee method according to claim 4, characterized in that, Before step S21, step S20 is included to perform anti-interference repair on the depth image sequence: S201. Simultaneously acquire a polarization image sequence that is temporally and spatially aligned with the RGB image sequence and the depth image sequence using the RGB-D camera; S202. Based on the polarization image sequence, calculate and generate a polarization feature map characterizing the reflection properties of the object surface; S203. For the same acquisition time, extract one frame of RGB image and one frame of depth image from the RGB image sequence and the depth image sequence respectively, fuse the RGB image, the depth image and the polarization feature map, and input them into the pre-trained deep inpainting neural network. S204. The deep repair neural network outputs a repaired depth image, which is used to replace the original depth image and participate in the registration process in step S22.

9. A chess referee assistance system, characterized in that, include: The data acquisition module uses at least two RGB-D cameras positioned at different spatial locations to simultaneously acquire a multi-view image sequence containing the chessboard and the player's operating area. The image sequence includes a synchronized RGB image sequence and a depth image sequence. The 3D reconstruction and identification module is used to perform 3D reconstruction processing on the RGB image sequence and the depth image sequence from different perspectives at the same time, and generate a 3D point cloud model of the hand and a 3D point cloud model corresponding to each chess piece at that time, which are respectively denoted as hand point cloud data and chess piece point cloud dataset, wherein each chess piece has a unique and persistent identity identifier. The violation determination module is used to determine violations based on the hand point cloud data and the chess piece point cloud dataset, specifically including: The contact target recognition unit is used to calculate the minimum spatial distance between the hand point cloud data and each chess piece point cloud data in the chess piece point cloud dataset; if the minimum spatial distance of a certain chess piece is less than the preset contact distance threshold, it is determined that contact has occurred, and the identity identifier of the chess piece is recorded as the ID of the touched chess piece. The associated behavior capture and judgment unit is used to continuously monitor the chess piece point cloud dataset after determining that contact has occurred. When a chessboard state change event is detected, a compliance judgment is immediately performed. The chessboard state change event includes chess piece movement events and chess piece disappearance events, wherein: If a piece movement event is detected, record the identity of all pieces that have moved to form a set of moving piece IDs; determine whether the touched piece ID is included in the set of moving piece IDs; if not, generate a response target inconsistency violation event. If a piece disappearance event is detected, the identity identifiers of all disappeared pieces are recorded to form a set of disappeared piece IDs; it is determined whether the piece identified by the touched piece ID is an opponent's piece, and at the same time, it is determined whether the touched piece ID is included in the set of disappeared piece IDs; if the piece identified by the touched piece ID is an opponent's piece and the touched piece ID is not included in the set of disappeared piece IDs, then a target non-elimination violation event is generated. The output module is used to output alarm information for the generated violation events.

Images