Tablet drop guiding system based on three-dimensional visual feedback

Abstract

The invention, which relates to the technical field of image identification, discloses a three-dimensional visual feedback-based child drop guiding system comprising a sensor module, an image processing and identification module, an abnormality identification module, a three-dimensional reconstruction and tracking module and a child drop guiding module. The sensor module is used for acquiring image data, adjusting an image recognition area according to the image detection data fed back by the image processing and recognition module, and transmitting the acquired image data to the image processing and recognition module; the image processing and recognition module is used for analyzing and processing the acquired image data, including grid line positioning, image data detection and chess piece recognition, and outputting the analyzed and processed image data to the abnormity recognition module and the three-dimensional reconstruction and tracking module; and the abnormity identification module is used for analyzing abnormal data in the image data and transmitting the analyzed abnormal data to the three-dimensional reconstruction and tracking module.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a move guidance system based on three-dimensional visual feedback. Background Art

[0002] With the development of computer science, people began to explore the use of computers to assist chess games. In particular, the introduction of machine learning and deep learning technologies has made the computer-assisted systems of chess games more intelligent. Using image processing technology and artificial intelligence technology, more and more entertainment robots have entered our lives, including Go robots. By effectively calibrating the Go robot's visual system, accurate positioning of movement and targets can be achieved. However, traditional visual feedback systems are usually based on two-dimensional image processing technology, and the plane images taken by the camera are easily affected by perspective deformation, lighting changes and shadows, resulting in reduced positioning accuracy. In addition, traditional two-dimensional systems cannot directly obtain the height information of objects, and can only estimate the height through image processing algorithms, which may lead to errors in height measurement. When dealing with occlusion problems, it is difficult to accurately identify the positions of all chess pieces and obtain the positions of chess pieces. At the same time, the camera angle needs to be manually adjusted to obtain a complete chessboard image. In view of the above technical defects, a solution is now proposed. Summary of the invention

[0003] The purpose of the present invention is to provide a chessboard placement guidance system based on three-dimensional visual feedback to solve the above-mentioned technical defects. The present invention obtains the image data of the chessboard and the surrounding environment in real time by installing a binocular camera obliquely above the chessboard. At the same time, according to the image detection of the number of corner points, the image angle of the binocular camera is adjusted so that the image collected by the binocular camera can completely cover the entire chessboard, and the edge of the external grid line is detected through the four corner points. Based on the external grid line, the bird's-eye view of the chessboard is obtained through perspective transformation.

[0004] The object of the present invention can be achieved by the following technical solutions: A move-guiding system based on three-dimensional visual feedback, comprising a sensor module, an image processing and recognition module, an abnormality recognition module, a three-dimensional reconstruction and tracking module and a move-guiding module; The sensor module is used to acquire image data, adjust the image recognition area according to the image detection data fed back by the image processing and recognition module, and transmit the acquired image data to the image processing and recognition module; The image processing and recognition module is used to analyze and process the acquired image data, including grid line positioning, image detection data and chess piece recognition, and output the analyzed and processed image data to the anomaly recognition module and the three-dimensional reconstruction and tracking module; The abnormality recognition module is used to analyze the abnormal data in the image data and transmit the analyzed abnormal data to the three-dimensional reconstruction and tracking module; The 3D reconstruction and tracking module is used to reconstruct the 3D model of the chessboard and chess pieces, continuously track the position and number changes of the chess pieces, and transmit the 3D model data to the chess piece placement guidance module; The chess piece placement guidance module is used to obtain and control the placement of chess pieces.

[0005] Preferably, the real-time acquisition process of image data is as follows: The binocular camera fixed diagonally above the chessboard is used to obtain image data of the chessboard and its surroundings in real time. The image detection data fed back by the image processing and recognition module is used to obtain the four corner points of the chessboard and the camera field of view coordinates. The left camera is used as a reference to establish the camera field of view coordinates. By identifying the four corner points of the chessboard, obtaining the coordinates of the center point of the chessboard and the position coordinates of the four corner points, calculating the difference between the coordinates of the center point of the chessboard and the center point of the camera's field of view in the horizontal direction, and the difference between the coordinates of the center point of the chessboard and the center point of the camera's field of view in the vertical direction, the position of the binocular camera is adjusted according to the calculated difference.

[0006] Preferably, the process of determining the number of corner points in the image detection data is as follows: When any of the four corner points of the chessboard is not detected from the image, the sensor module is negatively feedback adjusted, and the binocular camera is controlled to rotate 5° clockwise to detect and identify the image data in real time until any of the four corner points of the chessboard is detected in the image, and then the rotation stops; After the binocular camera rotates 360° clockwise, if any of the four corner points of the chessboard is not detected in the image, a prompt signal is issued; When any of the four corner points of the chessboard is detected from the image, the sensor module is negatively feedback adjusted to control the binocular camera to rotate 1° clockwise until the center point of the chessboard coincides with the center point of the camera field of view coordinates, the binocular camera stops rotating, and image data is acquired in real time.

[0007] Preferably, the preprocessing process before image data processing is as follows: The acquired image data is converted into a grayscale image, and the noise part of the grayscale image is reduced by using Gaussian blur filtering technology; Find the area where the brightness changes sharply in the grayscale image, identify multiple parallel and equally spaced straight lines, and combine the four corner points to obtain the four boundaries of the chessboard: The Canny algorithm is used to detect the edge of the chessboard grid line. The chessboard grid line edge is refined through non-maximum suppression, and only the local maximum point is retained. For each pixel point, the gradient direction of the pixel point is checked and compared with the two adjacent pixels in the gradient direction. If the gradient amplitude of the current pixel is not the maximum value between the two adjacent pixels, it is suppressed and the real edge point is retained.

[0008] Preferably, the external grid line edge detection screening process is as follows: In the grayscale image, the high threshold is set to 30% of the maximum value of the image gradient amplitude, and the low threshold is set to 15% of the maximum value of the image gradient amplitude. The chessboard pixels are screened by the high threshold and the low threshold: When the chessboard pixels are greater than or equal to 30% of the maximum value, it is marked as a strong edge; When the chessboard pixels are less than or equal to 15% of the maximum value of the image gradient amplitude, they are excluded; When the chessboard pixel is greater than 15% and less than 30% of the maximum value of the image gradient amplitude, and is not connected to any strong edge, it is marked as a weak edge and excluded; when it is connected to a strong edge, it is marked as a strong edge; connect the strong edge fragments to form a complete edge.

[0009] Preferably, the grid line positioning process is as follows: Get the detected external grid line edges, locate them, and locate the internal grid lines of the chessboard through perspective transformation: Get the four corner points and external grid line edge information of the chessboard, correct the chessboard to a top view through perspective transformation, and calculate the internal grid lines of the chessboard between adjacent straight lines based on the size and shape of the corrected small squares.

[0010] Preferably, the chess piece identification process is as follows: By converting the grayscale image into a binary image, the connectivity between pixels is defined, where white pixels represent chess pieces and black pixels represent background. The black and white parts of the binary image are defined separately, and the chess pieces in the binary image are white and the background is black. By connecting the four directions of up, down, left, and right, the connected areas are marked, and the entire binary image is traversed from the upper left corner. When an unmarked white pixel is encountered, it is marked as a new area, and all its connected white pixels are recursively marked with the same area number. The above process is repeated until all white pixels are marked, and candidate areas are extracted. Each marked connected area is an independent candidate area for chess pieces. The geometric features of each candidate area are calculated to verify whether it is indeed a chess piece. The process of identifying the black and white colors of chess pieces: Noise points are removed through morphological operations while maintaining the integrity of the chess pieces. The brightness threshold of black chess pieces is set to 50, while the brightness threshold of white chess pieces is set to 200. Areas with brightness lower than the brightness threshold of black chess pieces are extracted as black chess pieces; areas with brightness higher than the brightness threshold of white chess pieces are extracted as white chess pieces.

[0011] Preferably, the abnormal data in the analyzed image data is obtained and the process of analyzing the abnormal data is as follows: Determine the number of corner points of the acquired image data: When the number of corner points detected in the chessboard is 4, based on the known line positions, spacing, distance and angle relationship between the lines, and using the rule that the horizontal and vertical lines of the chessboard are equally spaced and parallel, the template is aligned with the actual image through a matching algorithm to complete the blocked part of the chessboard grid line; When it is detected that the number of corner points in the chessboard is less than 4, a prompt signal is issued and sent to the sensor module for identification, and the position detection of the Go chessboard is continuously performed.

[0012] Preferably, the three-dimensional model reconstruction process of the chessboard and chess pieces is as follows: By obtaining the corrected overhead image of the chessboard, the stereo matching algorithm is used to calculate the disparity map of the left and right images. According to the disparity map and camera parameters, the depth information of each pixel is calculated. According to the depth map and camera parameters, the two-dimensional image coordinates are converted into three-dimensional world coordinates. The three-dimensional modeling software Blender is used to create a three-dimensional model of the chessboard, and the coordinates (X, Y, Z) of the center point of each chess piece in the three-dimensional space are calculated. The coordinates of the center point of the chess piece are imported into the three-dimensional model, and the corresponding chess piece model is placed according to the three-dimensional coordinates of the chess piece. The position of the chess piece is displayed in the three-dimensional model, and the three-dimensional coordinates of the chess piece are stored in a structured data format.

[0013] Preferably, the process of obtaining and controlling the placement of chess pieces is as follows: The binocular camera scans the surrounding environment to generate a world coordinate system, and maps the chessboard position and the real-time position of the robotic arm to the world coordinate system. The coordinate information in the world coordinate system is used to control the robotic arm to place the chess pieces. The surrounding environment is scanned by a binocular camera to generate a world coordinate system, and the image data is stereo matched to generate a disparity map. According to the disparity map and camera parameters, the depth map is calculated, and the two-dimensional image coordinates are converted into three-dimensional world coordinates. A fixed reference point is selected as the origin of the world coordinate system, and the boundary line of the chessboard is used as the coordinate axis direction. In the generated world coordinate system, the position of the chessboard is identified and mapped to the world coordinate system, the robotic arm is calibrated, the position and posture of its end effector are obtained, and the motion path of the robotic arm is planned according to the three-dimensional coordinates of the chess piece.

[0014] The beneficial effects of the present invention are as follows: 1) The present invention uses a binocular camera to obtain image data of the chessboard and the surrounding environment in real time, and at the same time, according to the image detection of the number of corner points, performs negative feedback adjustment on the image acquisition angle of the binocular camera. By obtaining the rotation angle of the binocular camera and the number of corner points in the image data, the rotation angle of the binocular camera is adjusted in real time. When the center point of the chessboard coincides with the center point of the camera field of view coordinate, the position of the binocular camera is adjusted more quickly, so that the image collected by the binocular camera can completely cover the entire chessboard; 2) The present invention detects the edges of the external grid lines through the four corner points of the chessboard. Based on the external grid lines, the chessboard is corrected to the effect of being photographed from directly above using perspective transformation. The standard grid size is estimated by calculating the average interval between adjacent straight lines. The known Go grid template is scaled proportionally to accurately analyze the internal grid line edge structure, and then the entire grid layout is inferred. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention will be further described below in conjunction with the accompanying drawings; Figure 1 It is a system flow chart of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Example: See Figure 1 As shown, this embodiment provides a move guidance system based on three-dimensional visual feedback, including the following modules: The sensor module is used to obtain image data of the chessboard and the surrounding environment, adjust the image recognition area according to the image detection data fed back by the image processing and recognition module, and transmit the obtained image data to the image processing and recognition module; The binocular camera fixed diagonally above the chessboard is used to obtain image data of the chessboard and surrounding environment in real time; Install the binocular camera diagonally above the chessboard, about 30 cm vertically from the chessboard, and preliminarily ensure that all grid lines on the chessboard are within the field of view of the binocular camera. Use a tripod to mount the camera and ensure that the bracket will not move during use to avoid affecting the stability of the image. Use the binocular camera to obtain real-time image data of the chessboard and the surrounding environment. The four corner points of the chessboard and the camera field of view coordinates are obtained through the image detection data fed back by the image processing and recognition module. Usually, the left camera is used as the main camera and the left camera is used as a reference to establish the camera field of view coordinates. The right camera or the virtual center point is used as an auxiliary reference. If the left camera is blocked, the camera field of view coordinates are moved and adjusted accordingly. Through the four corner points of the chessboard, that is, the inner corner points of the four corners of the Go chessboard, the difference between the center point of the chessboard and the center point of the camera field of view coordinates is calculated. The coordinates of the four corner points are A (x1, y1), B (x2, y2), C (x3, y3), and D (x4, y4). The coordinates of the center point (Cx, Cy) of the chessboard are calculated using the following formula: ; ;

[0018] The center point of the camera's field of view is usually half of the image width and height. The image size is W×H, so the coordinates of the center point of the field of view (Vx, Vy) are: ; ;

[0019] Determine the horizontal and vertical offsets of the chessboard. The horizontal offset Δx can be expressed by the X-coordinate difference between the center of the chessboard and the center of the field of view: ;

[0020] The vertical offset Δy is expressed as the Y coordinate difference: ;

[0021] The calculated Δx and Δy values ​​are converted into commands that can be understood by the binocular camera control system, taking into account the influence of factors such as the focal length of the camera on the actual moving distance; By identifying the four corner points of the chessboard, the coordinates of the center point of the chessboard are obtained according to the position coordinates of the four corner points on the chessboard, the difference between the coordinates of the center point of the chessboard and the center point of the camera's field of view in the horizontal direction, and the difference between the coordinates of the center point of the chessboard and the center point of the camera's field of view in the vertical direction are calculated, and a command is sent to the binocular camera control system according to the calculated offset to adjust the position of the binocular camera so that the chessboard is located in the center of the camera screen; The image processing and recognition module is used to analyze the acquired image data, including grid line positioning, image detection data and chess piece recognition, and output the analyzed image data to the anomaly recognition module and the three-dimensional reconstruction and tracking module; By acquiring the image data from the sensor module, the Harris corner algorithm is used to detect the four corners of a standard 19x19 grid Go board: Use the cv2.findChessboardCorners function to detect the inner corners of the chessboard and generate the code example: import cv2 import numpy as np # Read the image image = cv2.imread('path_to_image.jpg') # Get from the binocular camera # Convert to grayscale image gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) # Detect chessboard corners pattern_size = (19, 19) # For a standard Go board found, corners = cv2.findChessboardCorners(gray, pattern_size, None, flags=cv2.CALIB_CB_ADAPTIVE_THRESH + cv2.CALIB_CB_NORMALIZE_IMAGE) if found: print("Found the corner points of the chessboard") Sub-pixel precision criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001) corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria) # Draw corner points cv2.drawChessboardCorners(image, pattern_size, corners, found) # Display the image cv2.imshow('Chessboard Corners', image) cv2.waitKey(0) cv2.destroyAllWindows() else: print("No checkerboard corners found") Determine the image detection data of the number of corner points: When any of the four corner points of the chessboard is not detected from the image, the sensor module is negatively feedback adjusted, and the binocular camera is controlled to rotate 5° clockwise to detect and identify the image data in real time until any of the four corner points of the chessboard is detected in the image, and then the rotation stops; After the binocular camera rotates 360° clockwise, if any of the four corner points of the chessboard is not detected in the image, a prompt signal is issued and transmitted to the abnormal recognition module to confirm whether the chessboard is placed within the recognition range of the binocular camera; When any of the four corner points of the chessboard is detected from the image, the binocular camera is rotated clockwise by 1° to make the center point of the chessboard coincide with the center point of the camera field of view coordinates, the binocular camera is stopped from rotating, and image data is acquired in real time; The preprocessing process before image data processing is as follows: The acquired image data is converted into a grayscale image to simplify subsequent processing, and Gaussian blur filtering technology is applied to reduce the noise part of the grayscale image to make the chessboard edge in the grayscale image clearer; By using a convolution kernel generated by a Gaussian function to smooth the image, the noise in the image can be effectively reduced while retaining more edge information. By defining a Gaussian kernel, a Gaussian kernel is a two-dimensional array whose element values ​​are determined by a Gaussian distribution, and the distribution is usually symmetrical. The center position has the largest weight, and its standard deviation σ determines the size of the kernel and the degree of blur. Each element of the kernel represents the degree of influence of the surrounding pixels on the central pixel. A larger σ will result in a wider kernel, resulting in a stronger blurring effect. The Gaussian kernel is convolved with each pixel in the grayscale image, and the kernel is covered on the image. The sum of the products of the pixel values ​​in the kernel coverage area and the weights of the corresponding positions of the kernel are calculated, and then this sum is used as the new value of the center pixel. Image processing is performed through Gaussian blur, and the convolution kernel generated by the Gaussian function is used to smooth the image, effectively reducing the noise in the image while retaining more edge information. The specific steps for Gaussian blur to reduce grayscale image noise are as follows: S301: Define Gaussian kernel: The Gaussian kernel is a two-dimensional array whose element values ​​are determined by the Gaussian distribution. The distribution is usually symmetrical, and the center position has the largest weight. The standard deviation σ determines the size of the kernel and the degree of blur. S302: Apply Gaussian kernel: Convolve the Gaussian kernel with each pixel in the grayscale image, cover the kernel on the image, calculate the sum of the products of the pixel values ​​in the kernel coverage area and the weights of the corresponding positions of the kernel, and then use this sum as the new value of the center pixel. The process can be expressed by the following formula: ;

[0022] in is the original image, is the processed image, is the element of the Gaussian kernel is the new pixel value at position (x, y) after Gaussian blur processing. The weighted average of the local area centered at (x, y) is obtained. is a double summation symbol, which means that for a All pixels in the neighborhood of are operated, k is half of the Gaussian kernel radius, For the original image In the figure, the pixel value at the neighborhood of position (x, y) with an offset of (i, j), i and j represent the offset in the horizontal and vertical directions respectively. G(i, j) is the element in the Gaussian kernel, which corresponds to the weight at position (i, j) relative to the center point (0, 0). Each element of the Gaussian kernel is calculated from a Gaussian distribution. Usually, the weight at the center of the kernel is the largest, and the weight gradually decreases as the distance from the center point increases. S303: Reduce Noise: Since the Gaussian kernel gives weights to the surrounding pixels, but the weights decrease as the distance increases, the local averaging effect makes the random noise on a single pixel replaced by the average of neighboring pixels; the local averaging effect helps to eliminate or weaken those isolated noise points, and the local averaging effect will not have a significant impact on the average value of the neighborhood. At the same time, the Gaussian kernel is a weighted average, so the pixel changes near the edge will be more moderate than those in the flat area, maintaining the clarity of the edge to a certain extent; S304 Select appropriate parameters: Generally, the kernel size is (6σ + 1) × (6σ + 1) to ensure sufficient coverage. Gaussian blur is implemented using a library such as OpenCV. The code for Gaussian blur using OpenCV in Python is as follows: ```Python import cv2 # Read the image and convert it to grayscale image = cv2.imread('chessboard.jpg') gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) # Apply Gaussian blur blurred_image = cv2.GaussianBlur(gray_image,(5, 5), 0) # Here (5, 5) is the kernel size, 0 means automatically calculating σ based on the kernel size # Display results cv2.imshow('Blurred Image', blurred_image) cv2.waitKey(0) cv2.destroyAllWindows() ``` Use the cv2.GaussianBlur function to apply Gaussian blur, adjust it according to the kernel size, and set the σ value; Find the area where the brightness changes sharply in the grayscale image, identify multiple parallel and equally spaced straight lines, and combine the four corner points to determine the four boundaries of the chessboard: The Canny algorithm is used to detect the edges of the chessboard's external grid lines. The non-maximum suppression is used to refine the edges of the chessboard grid lines, and only the local maximum points are retained. For each pixel, the gradient direction of the pixel is checked and compared with the two adjacent pixels in the gradient direction. If the gradient amplitude of the current pixel is not the maximum value of the two adjacent pixels, it is set to 0 (i.e., suppressed), and points that may be real edges are retained; According to experience, in grayscale images, the high threshold is usually set to 30% of the maximum value of the image gradient amplitude, and the low threshold is set to about half of the high threshold, that is, 15%. The high threshold and the low threshold are used to screen the chessboard pixels: When the chessboard pixels are greater than or equal to 30% of the pixels, it is marked as a strong edge; When the chessboard pixels are less than or equal to 15% of the pixels, they are excluded; When the chessboard pixel is greater than 15% and less than 30% of the pixels and is not connected to any strong edge, it is marked as a weak edge and excluded; when it is connected to a strong edge, it is marked as a strong edge; Connect strong edge fragments to form complete edges, starting from all strong edges until there are no more strong edges to connect. Canny edge detection can effectively extract the chessboard edge information in the grayscale image while minimizing the influence of noise; Grid line positioning, get the detected external grid line edge, locate it, and locate the internal grid lines of the chessboard through perspective transformation: The four corner points of the chessboard and the edge information of the external grid lines are obtained, the internal structure is accurately analyzed, and the chessboard is corrected to the effect of shooting from directly above using perspective transformation, and perspective correction is performed; each small square presents a regular size and shape, and the standard grid size is estimated by calculating the average interval between adjacent straight lines, and then the entire grid layout is inferred; Use the connected region labeling algorithm to find all independent chess piece candidate regions: By converting the grayscale image into a binary image (i.e., a black and white image), all pixels are divided into two categories (foreground and background). According to different areas of the image, the adaptive threshold method automatically calculates the threshold. Through visual inspection and parameter adjustment, the quality of the binary image is ensured. Through the binary chessboard image, the connectivity between pixels is defined, where white pixels represent chess pieces and black pixels represent background. The black and white parts of the binary image are defined separately. The chess pieces in the binary image are white and the background is black. By connecting the four directions of up, down, left, and right, the connected areas are marked. The entire binary image is traversed from the upper left corner. When an unmarked white pixel is encountered, it is marked as a new area, and all its connected white pixels are recursively marked with the same area number. The above process is repeated until all white pixels are marked, and candidate areas are extracted. Each marked connected area is an independent chess piece candidate area. The geometric features of each candidate area are calculated to verify whether it is indeed a chess piece. The process of identifying the color of black and white chess pieces: Morphological operations are used to remove small noise points while maintaining the integrity of the chess pieces. The threshold is manually set based on experience and the specific conditions of the image. The best segmentation effect is found through multiple trials and threshold adjustments. The brightness threshold of black chess pieces is set to 50, while the brightness threshold of white chess pieces is set to 200. The areas with brightness lower than the brightness threshold of black chess pieces are extracted as black chess pieces; the areas with brightness higher than the brightness threshold of white chess pieces are extracted as white chess pieces. An abnormality recognition module is used to obtain abnormal data in the analyzed image data, analyze the abnormal data, and transmit the analyzed abnormal data to the three-dimensional reconstruction and tracking module; When the binocular camera acquires image data, the number of diagonal points is detected to determine whether the detection angle of the binocular camera can completely cover the chessboard: When the number of corner points in the chessboard is detected to be 4, based on the known line positions, spacings, distances between lines, and angle relationships, the rule that the horizontal and vertical lines of the chessboard are equally spaced and parallel is used. For example, the distance between two adjacent horizontal or vertical lines should be fixed. A standard chessboard template is used, and the template is aligned with the actual image through a matching algorithm to complete the blocked chessboard grid line part. The template is aligned with the actual image through a matching algorithm to complete the blocked part. When it is detected that the number of corner points on the chessboard is less than 4, a prompt signal is issued and sent to the sensor module for identification, and the position of the Go chessboard is continuously detected; The 3D reconstruction and tracking module is used to reconstruct the chessboard grid line model and the 3D model of the chessboard and chess pieces, obtain the abnormal data of the abnormal recognition module, optimize the recognition, and continuously track the position and quantity changes of the chess pieces, and transmit the 3D model data to the placement guidance module; Get the corrected top-down perspective of the chessboard, use the stereo matching algorithm to calculate the disparity map of the left and right images, the disparity map represents the horizontal offset of each pixel in the two images, calculate the depth information of each pixel based on the disparity map and camera parameters, the depth map represents the distance from each pixel to the camera, convert the two-dimensional image coordinates into three-dimensional world coordinates based on the depth map and camera parameters, create a three-dimensional model of the chessboard using the three-dimensional modeling software Blender, calculate the (X, Y, Z) coordinates of each chess piece center point in three-dimensional space, import the three-dimensional position of the chess piece into the three-dimensional model for further analysis and visualization, place the corresponding chess piece model in the three-dimensional model according to the three-dimensional coordinates of the chess piece, display the position of the chess piece in the three-dimensional model so that the result can be viewed intuitively, and store the three-dimensional coordinates of the chess piece in a structured data format; The chess piece placement guidance module is used to obtain and control the placement of chess pieces; By scanning the surrounding environment with a fixed binocular camera, a world coordinate system is generated, and the chessboard position and the real-time position of the robotic arm are mapped to the world coordinate system. The coordinate information in the world coordinate system is used to control the robotic arm to accurately place the chess pieces. A world coordinate system is generated by scanning the surrounding environment with a fixed binocular camera. The image data is stereo matched to generate a disparity map. According to the disparity map and camera parameters, the depth map is calculated and the two-dimensional image coordinates are converted into three-dimensional world coordinates. A fixed reference point is selected as the origin of the world coordinate system, such as a corner point of a chessboard. The directions of the X-axis, Y-axis and Z-axis are defined. The boundary line of the chessboard is usually selected as the direction of the coordinate axis. In the generated world coordinate system, the position of the chessboard is identified and mapped to the world coordinate system. The robot arm is calibrated to obtain the position and posture of its end effector to ensure that the working range of the robot arm can cover the entire chessboard. According to the three-dimensional coordinates of the chess pieces, the motion path of the robot arm is planned.

[0023] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0024] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0025] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A move guidance system based on three-dimensional visual feedback, characterized in that: It includes sensor module, image processing and recognition module, abnormality recognition module, 3D reconstruction and tracking module and move guidance module; The sensor module is used to acquire image data, adjust the image recognition area according to the image detection data fed back by the image processing and recognition module, and transmit the acquired image data to the image processing and recognition module; The image processing and recognition module is used to analyze and process the acquired image data, including grid line positioning, image detection data and chess piece recognition, and output the analyzed and processed image data to the anomaly recognition module and the three-dimensional reconstruction and tracking module; The abnormality recognition module is used to analyze the abnormal data in the image data and transmit the analyzed abnormal data to the three-dimensional reconstruction and tracking module; The 3D reconstruction and tracking module is used to reconstruct the 3D model of the chessboard and chess pieces, continuously track the position and number changes of the chess pieces, and transmit the 3D model data to the chess piece placement guidance module; The chess piece placement guidance module is used to obtain and control the placement of chess pieces.

2. The move-guidance system based on three-dimensional visual feedback according to claim 1, characterized in that: The real-time acquisition process of image data is as follows: The image data of the chessboard and the surrounding environment are acquired in real time. The image detection data fed back by the image processing and recognition module is combined with the four corner points of the chessboard and the camera field of view coordinates to adjust the position of the binocular camera.

3. The move-guidance system based on three-dimensional visual feedback according to claim 2, characterized in that: The process of judging the number of corner points in the image detection data is as follows: When any of the four corner points of the chessboard is not detected from the image, the sensor module is negatively feedback adjusted to control the binocular camera to rotate 5° clockwise until any of the four corner points of the chessboard is detected in the image, and then the rotation stops; After the binocular camera rotates 360° clockwise, if any of the four corner points of the chessboard is not detected in the image, a prompt signal is issued; When any of the four corner points of the chessboard is detected from the image, the sensor module is negatively feedback adjusted to control the binocular camera to rotate 1° clockwise until the center point of the chessboard coincides with the center point of the camera field of view coordinates, the binocular camera stops rotating, and image data is acquired in real time.

4. The move-guiding system based on three-dimensional visual feedback according to claim 3, characterized in that: The preprocessing process before image data processing is as follows: The acquired image data is converted into a grayscale image, and the grayscale image is denoised using Gaussian blur filtering technology. Multiple parallel and equally spaced straight lines are identified through the areas where the brightness changes sharply in the grayscale image, and the four corner points are combined to obtain the boundaries of the chessboard: The Canny algorithm is used to detect the edges of the chessboard grid lines, perform non-maximum suppression, refine the edges of the chessboard grid lines, and retain the points of the real edges.

5. The move-guiding system based on three-dimensional visual feedback according to claim 4, characterized in that: The external grid line edge detection screening process is as follows: In the grayscale image, the high threshold is set to 30% of the maximum value of the image gradient amplitude, and the low threshold is set to 15% of the maximum value of the image gradient amplitude. The chessboard pixels are screened by the high threshold and the low threshold: When the chessboard pixels are greater than or equal to 30% of the maximum value, it is marked as a strong edge; When the chessboard pixel is less than or equal to 15% of the maximum value of the image gradient amplitude, it is marked as a weak edge; When the chessboard pixel is greater than 15% and less than 30% of the maximum value of the image gradient amplitude and is not connected to any strong edge, it is marked as a weak edge; when it is connected to a strong edge, it is marked as a strong edge, and the strong edge fragments are connected to form a complete edge.

6. The move-guiding system based on three-dimensional visual feedback according to claim 5, characterized in that: The grid line positioning process is as follows: Get the detected external grid line edges, locate them, and locate the internal grid lines of the chessboard through perspective transformation: Get the four corner points and external grid line edge information of the chessboard, and calculate the internal grid lines of the chessboard between adjacent straight lines through the top view after perspective transformation.

7. The move-guiding system based on three-dimensional visual feedback according to claim 6, characterized in that: The chess piece recognition process is as follows: By converting the grayscale image into a binary image, defining the connectivity between pixels, defining the black and white parts of the binary image separately, marking all its connected white pixels with the same region number, and extracting the candidate regions; The process of identifying the black and white colors of chess pieces: Noise points are removed through morphological operations while maintaining the integrity of the chess pieces. The brightness threshold of black chess pieces is set to 50, while the brightness threshold of white chess pieces is set to 200. Areas with brightness lower than the brightness threshold of black chess pieces are extracted as black chess pieces; areas with brightness higher than the brightness threshold of white chess pieces are extracted as white chess pieces.

8. The move-guiding system based on three-dimensional visual feedback according to claim 1, characterized in that: The abnormal data in the analyzed image data is obtained and the process of analyzing the abnormal data is as follows: Determine the number of corner points of the acquired image data: When the number of corner points detected in the chessboard is 4, the blocked part of the chessboard grid line is completed by using the rule that the horizontal and vertical lines of the chessboard are equally spaced and parallel; When it is detected that the number of corner points in the chessboard is less than 4, a prompt signal is issued and sent to the sensor module for identification, and the position detection of the Go chessboard is continuously performed.

9. The move-guiding system based on three-dimensional visual feedback according to claim 1, characterized in that: The process of reconstructing the 3D model of the chessboard and chess pieces is as follows: By obtaining the corrected overhead image of the chessboard, the disparity map of the left and right images is calculated using the stereo matching algorithm. According to the disparity map and camera parameters, the two-dimensional image coordinates are converted into three-dimensional world coordinates, and the corresponding chess piece model is placed according to the three-dimensional coordinates of the chess piece, and the position of the chess piece is displayed in the three-dimensional model.

10. The move-guiding system based on three-dimensional visual feedback according to claim 1, characterized in that: The process of obtaining and controlling the placement of chess pieces is as follows: Use a binocular camera to scan the surrounding environment, generate a world coordinate system, and map the chessboard position and the real-time position of the robotic arm to the world coordinate system. Use the coordinate information in the world coordinate system to control the robotic arm to place the chess pieces.

Images