A move guidance system based on three-dimensional visual feedback

Abstract

The present invention discloses a chess piece placement guidance system based on three-dimensional visual feedback, which relates to the field of image recognition technology, and includes a sensor module, an image processing and recognition module, an abnormality recognition module, a three-dimensional reconstruction and tracking module, and a chess piece placement guidance module; the sensor module is used to acquire image data, and at the same time, adjust the image recognition area according to the image detection data fed back by the image processing and recognition module, and transmit the acquired image data to the image processing and recognition module; the image processing and recognition module is used to analyze and process the acquired image data, including grid line positioning, image detection data and chess piece recognition, and output the analyzed and processed image data to the abnormality recognition module and the three-dimensional reconstruction and tracking module; the abnormality recognition module is used to analyze abnormal data in the image data, and transmit the analyzed abnormal data to the three-dimensional reconstruction and tracking module.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a move guidance system based on three-dimensional visual feedback. Background Art

[0002] With the development of computer science, people have begun to explore the use of computers to assist in chess and card games. In particular, the introduction of machine learning and deep learning technologies has made computer-assisted systems for chess and card games more intelligent. Utilizing image processing and artificial intelligence technologies, more and more entertainment robots have entered our lives, including Go robots. By effectively calibrating the Go robot's visual system, accurate positioning of movement and targets can be achieved.

[0003] However, traditional visual feedback systems are usually based on two-dimensional image processing technology. They rely on flat images captured by cameras, which are easily affected by perspective distortion, lighting changes, and shadows, resulting in reduced positioning accuracy. In addition, traditional two-dimensional systems cannot directly obtain the height information of objects and can only estimate the height through image processing algorithms. This may lead to errors in height measurement. When dealing with occlusion problems, it is difficult to accurately identify the positions of all chess pieces and obtain the positions of the chess pieces. At the same time, the camera angle needs to be manually adjusted to obtain a complete chessboard image.

[0004] In view of the above technical defects, a solution is proposed. Summary of the Invention

[0005] The purpose of the present invention is to provide a chessboard placement guidance system based on three-dimensional visual feedback to solve the technical defects mentioned above. The present invention obtains real-time image data of the chessboard and its surrounding environment through a binocular camera installed diagonally above the chessboard. At the same time, the image angle of the binocular camera is adjusted according to the image detection of the number of corner points, so that the image collected by the binocular camera can completely cover the entire chessboard, and the edges of the external grid lines are detected through the four corner points. Based on the external grid lines, a bird's-eye view of the chessboard is obtained through perspective transformation.

[0006] The object of the present invention can be achieved by the following technical solutions: a move-guiding system based on three-dimensional visual feedback, comprising a sensor module, an image processing and recognition module, an abnormality recognition module, a three-dimensional reconstruction and tracking module, and a move-guiding module;

[0007] The sensor module is used to acquire image data, adjust the image recognition area according to the image detection data fed back by the image processing and recognition module, and transmit the acquired image data to the image processing and recognition module;

[0008] The image processing and recognition module is used to analyze and process the acquired image data, including grid line positioning, image detection data and chess piece recognition, and output the analyzed and processed image data to the anomaly recognition module and the 3D reconstruction and tracking module;

[0009] The abnormality recognition module is used to analyze the abnormal data in the image data and transmit the analyzed abnormal data to the 3D reconstruction and tracking module;

[0010] The 3D reconstruction and tracking module is used to reconstruct the 3D model of the chessboard and chess pieces, continuously track the position and number changes of the chess pieces, and transmit the 3D model data to the placement guidance module;

[0011] The chess piece placement guidance module is used to obtain and control the placement of chess pieces.

[0012] Preferably, the real-time acquisition process of image data is as follows:

[0013] The binocular camera fixed diagonally above the chessboard acquires image data of the chessboard and its surroundings in real time. The image detection data fed back by the image processing and recognition module is used to obtain the four corner points of the chessboard and the camera field of view coordinates. The camera field of view coordinates are established using the left camera as a reference.

[0014] By identifying the four corner points of the chessboard, obtaining the coordinates of the center point of the chessboard and the position coordinates of the four corner points, calculating the horizontal difference between the coordinates of the center point of the chessboard and the center point of the camera's field of view coordinates, and the vertical difference between the coordinates of the center point of the chessboard and the center point of the camera's field of view coordinates, the position of the binocular camera is adjusted according to the calculated differences.

[0015] Preferably, the process of determining the number of corner points in the image detection data is as follows:

[0016] When any of the four corner points of the chessboard is not detected from the image, the sensor module is negatively feedback-adjusted, and the binocular camera is controlled to rotate 5° clockwise to detect and recognize the image data in real time until any of the four corner points of the chessboard is detected in the image, and then the rotation stops;

[0017] After the binocular camera rotates 360° clockwise, if any of the four corner points of the chessboard is not detected in the image, a prompt signal is issued;

[0018] When any of the four corner points of the chessboard is detected from the image, the sensor module is negatively feedback adjusted to control the binocular camera to rotate 1° clockwise until the center of the chessboard coincides with the center of the camera's field of view coordinates. The binocular camera stops rotating and image data is acquired in real time.

[0019] Preferably, the pre-processing process before image data processing is as follows:

[0020] Convert the acquired image data into a grayscale image and reduce the noise part of the grayscale image by using Gaussian blur filtering technology;

[0021] Find the area with a sharp change in brightness in the grayscale image, identify multiple parallel and equally spaced straight lines, and combine the four corner points to obtain the four boundaries of the chessboard:

[0022] The Canny algorithm is used to detect the edges of the chessboard grid lines. The non-maximum suppression is used to refine the edges of the chessboard grid lines, and only the local maximum points are retained. For each pixel, the gradient direction of the pixel point is checked and compared with the two adjacent pixels in the gradient direction. If the gradient amplitude of the current pixel is not the maximum value between the two adjacent pixels, it is suppressed and the points of the true edge are retained.

[0023] Preferably, the external grid line edge detection screening process is as follows:

[0024] In the grayscale image, the high threshold is set to 30% of the maximum value of the image gradient amplitude, and the low threshold is set to 15% of the maximum value of the image gradient amplitude. The high and low thresholds are used to filter the checkerboard pixels:

[0025] When the number of chessboard pixels is greater than or equal to 30% of the maximum value, it is marked as a strong edge;

[0026] When the chessboard pixels are less than or equal to 15% of the maximum value of the image gradient amplitude, they are excluded;

[0027] When the chessboard pixel is greater than 15% and less than 30% of the maximum value of the image gradient amplitude and is not connected to any strong edge, it is marked as a weak edge and excluded; when it is connected to a strong edge, it is marked as a strong edge; connect the strong edge fragments to form a complete edge.

[0028] Preferably, the grid line positioning process is as follows:

[0029] Get the detected external grid line edges, locate them, and use perspective transformation to locate the internal grid lines of the chessboard:

[0030] Get the four corner points and external grid line edge information of the chessboard, correct the chessboard to a top-down view through perspective transformation, and calculate the internal grid lines of the chessboard between adjacent straight lines based on the size and shape of the corrected small squares.

[0031] Preferably, the chess piece identification process is as follows:

[0032] By converting the grayscale image into a binary image, the connectivity between pixels is defined, where white pixels represent chess pieces and black pixels represent background. The black and white parts of the binary image are defined separately, where the chess pieces are white and the background is black. Connected regions are marked by connecting the four directions: up, down, left, and right. The entire binary image is traversed from the upper left corner. When an unmarked white pixel is encountered, it is marked as a new region, and all its connected white pixels are recursively marked with the same region number. The above process is repeated until all white pixels are marked, and candidate regions are extracted. Each marked connected region is an independent candidate region for chess pieces. The geometric features of each candidate region are calculated to verify whether it is indeed a chess piece.

[0033] The process of identifying the black and white colors of chess pieces:

[0034] Noise points are removed through morphological operations while maintaining the integrity of the chess pieces. The brightness threshold of black chess pieces is set to 50, and the brightness threshold of white chess pieces is set to 200. The areas with brightness lower than the brightness threshold of black chess pieces are extracted as black chess pieces; the areas with brightness higher than the brightness threshold of white chess pieces are extracted as white chess pieces.

[0035] Preferably, the abnormal data in the analyzed image data is obtained and the abnormal data is analyzed as follows:

[0036] Determine the number of corner points of the acquired image data:

[0037] When four corner points are detected on the chessboard, the template is aligned with the actual image using a matching algorithm based on the known line positions, spacing, and the distance and angle relationship between the lines, and the rule that the horizontal and vertical lines on the chessboard are equally spaced and parallel, thus completing the obscured part of the chessboard grid lines.

[0038] When it is detected that the number of corner points in the chessboard is less than 4, a prompt signal is issued and sent to the sensor module for identification, and the position of the Go chessboard is continuously detected.

[0039] Preferably, the process of reconstructing the three-dimensional model of the chessboard and chess pieces is as follows:

[0040] By obtaining a rectified overhead image of the chessboard, a stereo matching algorithm is used to calculate the disparity map of the left and right images. Based on the disparity map and camera parameters, the depth information of each pixel is calculated. Based on the depth map and camera parameters, the two-dimensional image coordinates are converted into three-dimensional world coordinates. The three-dimensional modeling software Blender is used to create a three-dimensional model of the chessboard. The coordinates (X, Y, Z) of the center point of each chess piece in three-dimensional space are calculated, and the coordinates of the chess piece center point are imported into the three-dimensional model. The corresponding chess piece models are placed according to the three-dimensional coordinates of the chess piece, the position of the chess piece is displayed in the three-dimensional model, and the three-dimensional coordinates of the chess piece are stored in a structured data format.

[0041] Preferably, the process of obtaining and controlling the placement of chess pieces is as follows:

[0042] The binocular camera scans the surrounding environment to generate a world coordinate system, and maps the chessboard position and the real-time position of the robotic arm to the world coordinate system. The coordinate information in the world coordinate system is used to control the robotic arm to place the chess pieces.

[0043] The surrounding environment is scanned by a binocular camera to generate a world coordinate system. Stereo matching is performed on the image data to generate a disparity map. Based on the disparity map and camera parameters, a depth map is calculated, and the two-dimensional image coordinates are converted into three-dimensional world coordinates. A fixed reference point is selected as the origin of the world coordinate system, and the boundary line of the chessboard is used as the coordinate axis direction. In the generated world coordinate system, the position of the chessboard is identified and mapped to the world coordinate system. The robotic arm is calibrated to obtain the position and posture of its end effector. According to the three-dimensional coordinates of the chess piece, the motion path of the robotic arm is planned.

[0044] The beneficial effects of the present invention are as follows:

[0045] 1) The present invention uses a binocular camera to acquire image data of the chessboard and its surroundings in real time. Simultaneously, based on image detection of the number of corner points, negative feedback is applied to the binocular camera's image acquisition angle. By acquiring the binocular camera's rotation angle and the number of corner points in the image data, the binocular camera's rotation angle is adjusted in real time. When the center point of the chessboard coincides with the center point of the camera's field of view coordinates, the binocular camera's position is adjusted more quickly, ensuring that the image captured by the binocular camera fully covers the entire chessboard.

[0046] 2) The present invention detects the edges of the outer grid lines from the four corner points of the chessboard. Based on the outer grid lines, the chessboard is corrected using perspective transformation to give the effect of being photographed from directly above. The standard grid size is estimated by calculating the average spacing between adjacent straight lines. By scaling the known Go grid template, the internal grid line edge structure is accurately analyzed to infer the entire grid layout. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be further described below with reference to the accompanying drawings;

[0048] Figure 1 It is a flow chart of the system of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] Example: See Figure 1 As shown, this embodiment provides a move guidance system based on three-dimensional visual feedback, including the following modules:

[0051] The sensor module is used to obtain image data of the chessboard and the surrounding environment, adjust the image recognition area according to the image detection data fed back by the image processing and recognition module, and transmit the obtained image data to the image processing and recognition module;

[0052] The binocular camera fixed diagonally above the chessboard is used to obtain image data of the chessboard and its surroundings in real time;

[0053] Mount the binocular camera diagonally above the chessboard, about 30 cm vertically above it. Initially ensure that all grid lines on the chessboard are within the binocular camera's field of view. Use a tripod to mount the camera, ensuring that the bracket does not move during use to avoid affecting image stability. Use the binocular camera to obtain real-time image data of the chessboard and its surroundings.

[0054] The image detection data fed back by the image processing and recognition module is used to obtain the four corner points of the chessboard and the camera field of view coordinates. Usually, the left camera is used as the main camera and the left camera is used as a reference to establish the camera field of view coordinates. The right camera or the virtual center point is used as an auxiliary reference. If the left camera is blocked, the camera field of view coordinates are adjusted accordingly.

[0055] The difference between the center of the board and the center of the camera's field of view is calculated using the four corner points of the board, i.e., the inner corner points of the four corners of the Go board. The coordinates of the four corner points are A (x1, y1), B (x2, y2), C (x3, y3), and D (x4, y4). The coordinates of the center of the board (Cx, Cy) are calculated using the following formula:

[0056] ;

[0057] ;

[0058] The center of the camera's field of view is usually half the width and height of the image. The image size is W×H, so the coordinates of the center of the field of view (Vx, Vy) are:

[0059] ;

[0060] ;

[0061] Determine the horizontal and vertical offsets of the chessboard. The horizontal offset Δx can be expressed as the difference in X coordinates between the center of the chessboard and the center of the field of view:

[0062] ;

[0063] The vertical offset Δy is expressed as the Y coordinate difference:

[0064] ;

[0065] The calculated Δx and Δy values ​​are converted into instructions that can be understood by the binocular camera control system, taking into account the influence of factors such as the camera's focal length on the actual movement distance;

[0066] By identifying the four corner points of the chessboard, the coordinates of the center point of the chessboard are obtained based on the position coordinates of the four corner points on the chessboard. The horizontal difference between the coordinates of the center point of the chessboard and the center point of the camera's field of view coordinates, as well as the vertical difference between the coordinates of the center point of the chessboard and the center point of the camera's field of view coordinates, are calculated. Based on the calculated offset, a command is sent to the binocular camera control system to adjust the position of the binocular camera so that the chessboard is located in the center of the camera image.

[0067] The image processing and recognition module is used to analyze the acquired image data, including grid line positioning, image detection data and chess piece recognition, and output the analyzed image data to the anomaly recognition module and the 3D reconstruction and tracking module;

[0068] By acquiring image data from the sensor module, the Harris corner algorithm is used to detect the four corners of a standard 19x19 grid Go board:

[0069] Use the cv2.findChessboardCorners function to detect the inner corners of the chessboard and generate the following code example:

[0070] import cv2

[0071] import numpy as np

[0072] # Read the image

[0073] image = cv2.imread('path_to_image.jpg') # Get from binocular camera

[0074] # Convert to grayscale image

[0075] gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

[0076] # Detect chessboard corners

[0077] pattern_size = (19, 19) # For a standard Go board

[0078] found, corners = cv2.findChessboardCorners(gray, pattern_size, None,

[0079] flags=cv2.CALIB_CB_ADAPTIVE_THRESH + cv2.CALIB_CB_NORMALIZE_IMAGE)

[0080] if found:

[0081] print("Found the corner points of the chessboard")

[0082] Sub-pixel accuracy

[0083] criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)

[0084] corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)

[0085] # Draw corner points

[0086] cv2.drawChessboardCorners(image, pattern_size, corners, found)

[0087] # Display the image

[0088] cv2.imshow('Chessboard Corners', image)

[0089] cv2.waitKey(0)

[0090] cv2.destroyAllWindows()

[0091] else:

[0092] print("No checkerboard corners found")

[0093] Determine the number of corner points in the image detection data:

[0094] When any of the four corner points of the chessboard is not detected in the image, the sensor module is negatively feedback-adjusted to control the binocular camera to rotate 5° clockwise, and the image data is detected and recognized in real time until any of the four corner points of the chessboard is detected in the image, and then the rotation stops;

[0095] After the binocular camera rotates 360° clockwise, if any of the four corner points of the chessboard is not detected in the image, a prompt signal is issued and transmitted to the abnormal recognition module to confirm whether the chessboard is placed within the recognition range of the binocular camera;

[0096] When any of the four corner points of the chessboard is detected from the image, the binocular camera is rotated 1° clockwise to make the center of the chessboard coincide with the center of the camera's field of view coordinates. The binocular camera stops rotating and image data is acquired in real time.

[0097] The preprocessing process before image data processing is as follows:

[0098] The acquired image data is converted into a grayscale image to simplify subsequent processing, and Gaussian blur filtering technology is applied to reduce the noise part of the grayscale image to make the chessboard edge in the grayscale image clearer;

[0099] By using a convolution kernel generated by a Gaussian function to smooth the image, the noise in the image can be effectively reduced while retaining more edge information. By defining a Gaussian kernel, a Gaussian kernel is a two-dimensional array whose element values ​​are determined by a Gaussian distribution, and the distribution is usually symmetrical. The center position has the largest weight, and its standard deviation σ determines the size of the kernel and the degree of blur. Each element of the kernel represents the degree of influence of the surrounding pixels on the central pixel. A larger σ will result in a wider kernel, resulting in a stronger blurring effect. The Gaussian kernel is convolved with each pixel in the grayscale image, and the kernel is covered on the image. The sum of the products of the pixel values ​​in the kernel coverage area and the weights of the corresponding kernel positions is calculated, and this sum is used as the new value of the center pixel.

[0100] Image processing is performed through Gaussian blur, and the convolution kernel generated by the Gaussian function is used to smooth the image, effectively reducing the noise in the image while retaining more edge information. The specific steps for Gaussian blurring to reduce grayscale image noise are as follows:

[0101] S301: Define Gaussian kernel:

[0102] The Gaussian kernel is a two-dimensional array whose element values ​​are determined by a Gaussian distribution. The distribution is usually symmetrical, and the center position has the largest weight. The standard deviation σ determines the size of the kernel and the degree of blur.

[0103] S302: Apply Gaussian kernel:

[0104] Convolve the Gaussian kernel with each pixel in the grayscale image, cover the kernel on the image, calculate the sum of the products of the pixel values ​​in the kernel coverage area and the weights of the kernel corresponding positions, and then use this sum as the new value of the center pixel. The process can be expressed by the following formula:

[0105] ;

[0106] in is the original image, is the processed image, are the elements of the Gaussian kernel It is the new pixel value at position (x, y) after Gaussian blur processing, which is obtained by The weighted average of the local area centered at (x, y) is obtained. It is a double summation symbol, which means that All pixels in the neighborhood of are operated, k is half of the Gaussian kernel radius, For the original image In the figure, the pixel value at the neighborhood of position (x, y) with an offset of (i, j) is located. i and j represent the offset in the horizontal and vertical directions, respectively. G(i, j) is the element in the Gaussian kernel, which corresponds to the weight at position (i, j) relative to the center point (0, 0). Each element of the Gaussian kernel is calculated from a Gaussian distribution. Generally, the weight at the center of the kernel is the largest, and the weight gradually decreases as the distance from the center increases.

[0107] S303: Noise reduction:

[0108] Since the Gaussian kernel assigns weights to surrounding pixels, but the weights decrease with increasing distance, the local averaging effect causes the random noise on a single pixel to be replaced by the average of neighboring pixels. The local averaging effect helps to eliminate or weaken those isolated noise points, and the local averaging effect will not have a significant impact on the average value of the neighborhood. At the same time, the Gaussian kernel is a weighted average, so the pixel changes near the edge will be more moderate than those in the flat area, maintaining the clarity of the edge to a certain extent.

[0109] S304 Select appropriate parameters:

[0110] In general, the kernel size is (6σ + 1) × (6σ + 1) to ensure sufficient coverage. Gaussian blur is implemented using a library such as OpenCV. The operation code for Gaussian blur using OpenCV in Python is as follows:

[0111] ```Python

[0112] import cv2

[0113] # Read the image and convert it to grayscale

[0114] image = cv2.imread('chessboard.jpg')

[0115] gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

[0116] # Apply Gaussian blur

[0117] blurred_image = cv2.GaussianBlur(gray_image,(5, 5), 0) # Here (5, 5) is the kernel size, 0 means σ is automatically calculated based on the kernel size

[0118] # Display the results

[0119] cv2.imshow('Blurred Image', blurred_image)

[0120] cv2.waitKey(0)

[0121] cv2.destroyAllWindows()

[0122] ```

[0123] Apply Gaussian blur using `cv2.GaussianBlur` function, adjusting it according to the kernel size and setting the σ value;

[0124] Find the area with a sharp change in brightness in the grayscale image, identify multiple parallel and equally spaced straight lines, and combine the four corner points to determine the four boundaries of the chessboard:

[0125] The Canny algorithm is used to detect the edges of the chessboard's external grid lines. Non-maximum suppression is used to refine the edges of the chessboard grid lines, retaining only the local maximum points. For each pixel, the gradient direction of the pixel is checked and compared with the two adjacent pixels in that gradient direction. If the gradient amplitude of the current pixel is not the maximum of the two adjacent pixels, it is set to 0 (i.e., suppressed), and points that may be true edges are retained.

[0126] According to experience, in grayscale images, the high threshold is usually set to 30% of the maximum value of the image gradient amplitude, and the low threshold is set to about half of the high threshold, that is, 15%. The high and low thresholds are used to filter the checkerboard pixels:

[0127] When the chessboard pixels are greater than or equal to 30% of the pixels, it is marked as a strong edge;

[0128] When the chessboard pixels are less than or equal to 15% of the pixels, they are excluded;

[0129] When the chessboard pixels are greater than 15% and less than 30% of the pixels and are not connected to any strong edge, they are marked as weak edges and excluded; when they are connected to a strong edge, they are marked as strong edges;

[0130] Connect strong edge segments to form complete edges, starting from all strong edges until there are no more strong edges to connect. Canny edge detection can effectively extract the chessboard edge information in the grayscale image while minimizing the influence of noise;

[0131] Grid line positioning, obtain the detected external grid line edges, perform positioning, and locate the internal grid lines of the chessboard through perspective transformation:

[0132] The four corner points and external grid line edge information of the chessboard are obtained, the internal structure is accurately analyzed, and the chessboard is corrected using perspective transformation to make it look like it was photographed from directly above, performing perspective correction. Each small square presents a regular size and shape, and the standard grid size is estimated by calculating the average spacing between adjacent straight lines, thereby inferring the entire grid layout.

[0133] Use the connected component labeling algorithm to find all independent candidate regions of chess pieces:

[0134] By converting the grayscale image into a binary image (i.e., a black and white image), all pixels are divided into two categories (foreground and background). The adaptive threshold method automatically calculates the threshold according to different areas of the image. The quality of the binary image is ensured through visual inspection and parameter adjustment. The connectivity between pixels is defined through the binary chessboard image, where white pixels represent chess pieces and black pixels represent background. The black and white parts of the binary image are defined separately, where the chess pieces are white and the background is black. The connected areas are marked by connecting the four directions of up, down, left, and right. The entire binary image is traversed from the upper left corner. When an unmarked white pixel is encountered, it is marked as a new area, and all its connected white pixels are recursively marked with the same area number. The above process is repeated until all white pixels are marked, and candidate areas are extracted. Each marked connected area is an independent chess piece candidate area. The geometric features of each candidate area are calculated to verify whether it is indeed a chess piece.

[0135] The process of identifying the color of black and white chess pieces:

[0136] Morphological operations were used to remove small noise points while maintaining the integrity of the chess pieces. The threshold was manually set based on experience and the specific conditions of the image. After multiple trials and adjustments to the threshold, the optimal segmentation effect was found. The brightness threshold for black chess pieces was set to 50, while the brightness threshold for white chess pieces was set to 200. Areas with brightness below the brightness threshold for black chess pieces were extracted as black chess pieces, and areas with brightness above the brightness threshold for white chess pieces were extracted as white chess pieces.

[0137] Anomaly recognition module, used to obtain abnormal data from the analyzed image data, analyze the abnormal data, and transmit the analyzed abnormal data to the 3D reconstruction and tracking module;

[0138] When the binocular camera acquires image data, the number of diagonal points is detected to determine whether the binocular camera's detection angle can completely cover the chessboard:

[0139] When four corner points are detected on the chessboard, based on the known line positions, spacing, and the distance and angle relationship between the lines, and taking advantage of the rule that the horizontal and vertical lines on the chessboard are equally spaced and parallel, for example, the distance between two adjacent horizontal or vertical lines should be fixed, a standard chessboard template is used. The template is aligned with the actual image through a matching algorithm to complete the occluded part of the chessboard grid lines.

[0140] When it is detected that the number of corner points on the chessboard is less than 4, a prompt signal is issued and sent to the sensor module for identification, and the position of the Go chessboard is continuously detected;

[0141] The 3D reconstruction and tracking module is used to reconstruct the chessboard grid line model and reconstruct the 3D model of the chessboard and chess pieces. It obtains abnormal data from the abnormality recognition module, optimizes the recognition, and continuously tracks the position and number changes of chess pieces. It also transmits the 3D model data to the placement guidance module.

[0142] Obtain the corrected top-down perspective of the chessboard, use a stereo matching algorithm to calculate the disparity map of the left and right images, which represents the horizontal offset of each pixel in the two images. Calculate the depth information of each pixel based on the disparity map and camera parameters. The depth map represents the distance from each pixel to the camera. Convert the two-dimensional image coordinates to three-dimensional world coordinates based on the depth map and camera parameters. Create a three-dimensional model of the chessboard using the three-dimensional modeling software Blender. Calculate the (X, Y, Z) coordinates of each chess piece center point in three-dimensional space. Import the three-dimensional positions of the chess pieces into the three-dimensional model for further analysis and visualization. Place the corresponding chess piece models in the three-dimensional model according to their three-dimensional coordinates. Display the positions of the chess pieces in the three-dimensional model for intuitive viewing of the results. Store the three-dimensional coordinates of the chess pieces in a structured data format.

[0143] The chess piece placement guidance module is used to obtain and control the placement of chess pieces;

[0144] By scanning the surrounding environment with a fixed binocular camera, a world coordinate system is generated. The chessboard position and the real-time position of the robotic arm are mapped to the world coordinate system. The coordinate information in the world coordinate system is used to control the robotic arm to accurately place the chess pieces.

[0145] A fixed binocular camera is used to scan the surrounding environment to generate a world coordinate system. The image data is stereo matched to generate a disparity map. Based on the disparity map and camera parameters, a depth map is calculated, and the two-dimensional image coordinates are converted into three-dimensional world coordinates. A fixed reference point is selected as the origin of the world coordinate system, such as a corner point of a chessboard, and the directions of the X-axis, Y-axis, and Z-axis are defined. The boundary line of the chessboard is usually selected as the coordinate axis direction. In the generated world coordinate system, the position of the chessboard is identified and mapped to the world coordinate system. The robotic arm is calibrated to obtain the position and posture of its end effector to ensure that the working range of the robotic arm can cover the entire chessboard. According to the three-dimensional coordinates of the chess pieces, the motion path of the robotic arm is planned.

[0146] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0147] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment as needed.

[0148] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A move guidance system based on three-dimensional visual feedback, characterized in that: It includes sensor module, image processing and recognition module, anomaly recognition module, 3D reconstruction and tracking module and move guidance module; The sensor module is used to acquire image data, adjust the image recognition area according to the image detection data fed back by the image processing and recognition module, and transmit the acquired image data to the image processing and recognition module; The image processing and recognition module is used to analyze and process the acquired image data, including grid line positioning, image detection data and chess piece recognition, and output the analyzed and processed image data to the anomaly recognition module and the 3D reconstruction and tracking module; The abnormality recognition module is used to analyze the abnormal data in the image data and transmit the analyzed abnormal data to the 3D reconstruction and tracking module; The 3D reconstruction and tracking module is used to reconstruct the 3D model of the chessboard and chess pieces, continuously track the position and number changes of the chess pieces, and transmit the 3D model data to the placement guidance module; The chess piece placement guidance module is used to obtain and control the placement of chess pieces; The real-time acquisition process of image data is as follows: Acquire image data of the chessboard and its surroundings in real time. Adjust the position of the binocular camera based on the image detection data fed back by the image processing and recognition module, combined with the four corner points of the chessboard and the camera's field of view coordinates. The process of judging the number of corner points in image detection data is as follows: When any of the four corner points of the chessboard is not detected from the image, the sensor module is negatively feedback-adjusted to control the binocular camera to rotate 5° clockwise until any of the four corner points of the chessboard is detected in the image, and then the rotation stops; After the binocular camera rotates 360° clockwise, if any of the four corner points of the chessboard is still not detected in the image, a prompt signal is issued; When any of the four corner points of the chessboard is detected from the image, the sensor module is negatively feedback-adjusted to control the binocular camera to rotate 1° clockwise until the center of the chessboard coincides with the center of the camera's field of view coordinates. The binocular camera stops rotating and image data is acquired in real time. The abnormal data in the analyzed image data is obtained and the analysis process of the abnormal data is as follows: Determine the number of corner points of the acquired image data: When the number of corner points detected in the chessboard is 4, the obscured part of the chessboard grid line is completed by using the rule that the horizontal and vertical lines of the chessboard are equally spaced and parallel; When it is detected that the number of corner points on the chessboard is less than 4, a prompt signal is issued and sent to the sensor module for identification, and the position of the Go chessboard is continuously detected; The grid line positioning process is as follows: Get the detected external grid line edges, locate them, and use perspective transformation to locate the internal grid lines of the chessboard: Get the four corner points and external grid line edge information of the chessboard, and calculate the internal grid lines of the chessboard between adjacent straight lines through the top view after perspective transformation.

2. The three-dimensional visual feedback-based move guidance system according to claim 1, characterized in that: The preprocessing process before image data processing is as follows: The acquired image data is converted into a grayscale image. The grayscale image is denoised using Gaussian blur filtering. Multiple parallel and equally spaced straight lines are identified through areas with sharp brightness changes in the grayscale image. Combined with the four corner points, the boundaries of the chessboard are obtained: The Canny algorithm is used to detect the edges of the chessboard grid lines, perform non-maximum suppression, refine the edges of the chessboard grid lines, and retain the points of the true edges.

3. The three-dimensional visual feedback-based move guidance system according to claim 2, characterized in that: The external grid line edge detection screening process is as follows: In the grayscale image, the high threshold is set to 30% of the maximum value of the image gradient amplitude, and the low threshold is set to 15% of the maximum value of the image gradient amplitude. The high threshold and low threshold are used to filter the checkerboard pixels: When the number of chessboard pixels is greater than or equal to 30% of the maximum value, it is marked as a strong edge; When the checkerboard pixel is less than or equal to 15% of the maximum value of the image gradient amplitude, it is marked as a weak edge; When the checkerboard pixel is greater than 15% and less than 30% of the maximum value of the image gradient amplitude and is not connected to any strong edge, it is marked as a weak edge; when it is connected to a strong edge, it is marked as a strong edge, and the strong edge fragments are connected to form a complete edge.

4. The three-dimensional visual feedback-based move guidance system according to claim 1, characterized in that: The chess piece recognition process is as follows: By converting the grayscale image into a binary image, defining the connectivity between pixels, defining the black and white parts of the binary image separately, marking all connected white pixels with the same region number, and extracting candidate regions; The process of identifying the black and white colors of chess pieces: Noise points are removed through morphological operations while maintaining the integrity of the chess pieces. The brightness threshold of black chess pieces is set to 50, and the brightness threshold of white chess pieces is set to 200. The areas with brightness lower than the brightness threshold of black chess pieces are extracted as black chess pieces; the areas with brightness higher than the brightness threshold of white chess pieces are extracted as white chess pieces.

5. The three-dimensional visual feedback-based move guidance system according to claim 1, characterized in that: The process of reconstructing the 3D model of the chessboard and chess pieces is as follows: By obtaining a rectified overhead image of the chessboard, the disparity map of the left and right images is calculated using a stereo matching algorithm. Based on the disparity map and camera parameters, the two-dimensional image coordinates are converted into three-dimensional world coordinates. The corresponding chess piece models are placed according to the three-dimensional coordinates of the chess pieces, and the positions of the chess pieces are displayed in the three-dimensional model.

6. The three-dimensional visual feedback-based move guidance system according to claim 1, characterized in that: The process of obtaining and controlling the placement of chess pieces is as follows: Use a binocular camera to scan the surrounding environment, generate a world coordinate system, and map the chessboard position and the real-time position of the robotic arm to the world coordinate system. Use the coordinate information in the world coordinate system to control the robotic arm to place the chess pieces.

Images