An image recognition method for a Rubik's Cube solving robot

Abstract

The present invention relates to the technical field of image recognition algorithms, and specifically discloses an image recognition algorithm for a Rubik's Cube solving robot, which includes: obtaining a Rubik's Cube image, wherein the Rubik's Cube image is an image after the Rubik's Cube is scrambled; performing image segmentation processing on the Rubik's Cube image to obtain the surface boundary points between each two adjacent planes of the Rubik's Cube; performing perspective transformation according to the surface boundary points to obtain a transformed image; capturing each independent color block according to the transformed image; and performing image color recognition on each of the independent color blocks. The image recognition algorithm for a Rubik's Cube solving robot provided by the present invention can effectively improve recognition accuracy and robustness by performing perspective transformation on the Rubik's Cube image, processing the image after the perspective transformation, and performing color recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition algorithms, and in particular to an image recognition method used in a Rubik's Cube solving robot. Background Art

[0002] In the prior art, the Rubik's Cube is usually restored manually after being scrambled, which is a slow method. With the emergence of Rubik's Cube solving robots, the speed of solving Rubik's Cube has been greatly improved. Most of the existing Rubik's Cube solving robots are controlled by image recognition, but the existing image recognition has high requirements for the objects to be recognized, such as the Rubik's Cube to be recognized must be placed in a fixed position, which makes the application of image recognition technology very limited. Summary of the invention

[0003] The present invention provides an image recognition method for a Rubik's Cube solving robot, which solves the problem of limited application of image recognition technology in Rubik's Cube solving existing in the related art.

[0004] As one aspect of the present invention, there is provided an image recognition method for a Rubik's Cube solving robot, comprising:

[0005] Obtain a Rubik's Cube image, where the Rubik's Cube image is an image after the Rubik's Cube is scrambled;

[0006] Performing image segmentation processing on the Rubik's Cube image to obtain the surface boundary points between every two adjacent planes of the Rubik's Cube;

[0007] Perform perspective transformation according to the surface boundary points to obtain a transformed image;

[0008] capturing each independent color patch according to the transformed image;

[0009] Image color recognition is performed on each of the independent color blocks.

[0010] Furthermore, the Rubik's Cube image includes a first Rubik's Cube image and a second Rubik's Cube image, the first Rubik's Cube image is a Rubik's Cube image obtained by shooting from a first perspective, the first Rubik's Cube image includes three Rubik's Cube planes, the second Rubik's Cube image is a Rubik's Cube image obtained by shooting from a second perspective, the second Rubik's Cube image includes another three Rubik's Cube planes.

[0011] Furthermore, the image segmentation processing is performed on the Rubik's Cube image to obtain the surface boundary points between every two adjacent planes of the Rubik's Cube, including:

[0012] Performing image segmentation according to basic features of the first Rubik's Cube image and basic features of the second Rubik's Cube image, respectively, to obtain three Rubik's Cube planes corresponding to the first Rubik's Cube image and another three Rubik's Cube planes corresponding to the second Rubik's Cube image;

[0013] The surface boundary points of two adjacent Rubik's Cube planes are selected according to the positions of the color blocks in the Rubik's Cube planes.

[0014] Furthermore, the face boundary points include a plurality of intersection points of two adjacent Rubik's Cube planes.

[0015] Furthermore, performing perspective transformation according to the surface boundary points to obtain a transformed image includes:

[0016] Obtain two coordinate arrays according to the first Rubik's Cube image and the second Rubik's Cube image respectively;

[0017] Calculate the perspective transformation matrix based on the two coordinate arrays;

[0018] Perspective transformation is performed on the first magic cube image and the second magic cube image according to the perspective transformation matrix to obtain a transformed image, wherein the transformed image is a vertical viewing angle.

[0019] Further, capturing each independent color block according to the transformed image includes:

[0020] According to the principle of similar colors, adjacent points with similar colors are clustered to obtain each independent nearly rectangular color block.

[0021] Furthermore, the performing image color recognition on each of the independent color blocks includes:

[0022] The independent color blocks are automatically identified according to the training model.

[0023] Furthermore, the training model is obtained by:

[0024] The Rubik's Cube image is subjected to labeling pre-training processing, and each Rubik's Cube plane is calibrated according to the direction to obtain all initial training images;

[0025] Performing perspective transformation on the initial training image to obtain a training transformed image;

[0026] The training transformed image is labeled to mark the color of each independent color block, thereby obtaining a training model.

[0027] The image recognition method for a Rubik's Cube solving robot provided by the present invention can effectively improve recognition accuracy and robustness by performing perspective transformation on the Rubik's Cube image, processing the image after the perspective transformation and performing color recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0029] Figure 1 A flow chart of an image recognition method for a Rubik's Cube solving robot provided by the present invention. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] In this embodiment, an image recognition method for a Rubik's Cube solving robot is provided. Figure 1 is a flow chart of an image recognition method for a Rubik's Cube solving robot provided according to an embodiment of the present invention. Figure 1 As shown, including:

[0034] S110, obtaining a Rubik's Cube image, wherein the Rubik's Cube image is an image after the Rubik's Cube is scrambled;

[0035] S120, performing image segmentation processing on the Rubik's Cube image to obtain surface boundary points between every two adjacent planes of the Rubik's Cube;

[0036] S130, performing perspective transformation according to the surface boundary points to obtain a transformed image;

[0037] S140, capturing each independent color block according to the transformed image;

[0038] S150: Perform image color recognition on each of the independent color blocks.

[0039] The image recognition method for a Rubik's Cube solving robot provided in an embodiment of the present invention can effectively improve recognition accuracy and robustness by performing perspective transformation on the Rubik's Cube image, processing the image after perspective transformation, and performing color recognition.

[0040] Specifically, the Rubik's Cube image includes a first Rubik's Cube image and a second Rubik's Cube image. The first Rubik's Cube image is a Rubik's Cube image taken from a first perspective, and the first Rubik's Cube image includes three Rubik's Cube planes. The second Rubik's Cube image is a Rubik's Cube image taken from a second perspective, and the second Rubik's Cube image includes another three Rubik's Cube planes.

[0041] It can be understood that the two cameras shoot from one vertex of the Rubik's Cube respectively, and obtain two original images of the Rubik's Cube containing three faces.

[0042] Specifically, the image segmentation process is performed on the Rubik's Cube image to obtain the surface boundary points between every two adjacent planes of the Rubik's Cube, including:

[0043] Performing image segmentation according to basic features of the first Rubik's Cube image and basic features of the second Rubik's Cube image, respectively, to obtain three Rubik's Cube planes corresponding to the first Rubik's Cube image and another three Rubik's Cube planes corresponding to the second Rubik's Cube image;

[0044] The surface boundary points of two adjacent Rubik's Cube planes are selected according to the positions of the color blocks in the Rubik's Cube planes.

[0045] It should be understood that the original image is labeled for pre-training, each side is calibrated into the left side, the top side and the right side, and all the initial images are trained.

[0046] The type of reference box anchor is determined by the scale and ratio. If the scale contains (s1, s2, s3) and the ratio contains (r1, r2, r3), the number of reference box anchor types is the number of scale types multiplied by the ratio, and the number of reference box anchor types is 9. If the size of the feature map is Z, the number of reference box anchors is 9z. However, it is unknown which reference frame anchors include the real target. Therefore, the reference frame target layer anchor target layer completes the task of distinguishing which reference frame anchors are positive samples (including the real target) and which reference frame anchors are negative samples (including only the background). The specific method is to calculate the intersection-over-Union (IoU) between the reference frame anchor and the correctly labeled data ground truth. There are three evaluation criteria: for each correctly labeled data ground truth, select a reference frame anchor with the largest IoU as a positive sample; for each reference frame anchor, the reference frame anchor whose IoU with the correctly labeled data ground truth is greater than a certain threshold t1 is used as a positive sample; not all reference frame anchors except the above two are negative samples, but reference frame anchors whose IoU with the correctly labeled data ground truth is less than a certain threshold t2 are negative samples. The samples between [t2, t1] are "don't care" samples, which are neither positive nor negative samples and do not participate in model optimization, that is, they do not participate in the calculation of the subsequent loss function.

[0047] First, in the algorithm ROIAlign, xi*(r,j) is a floating point coordinate position (the sampling point calculated during forward propagation). In the feature map before pooling, each point whose horizontal and vertical coordinates are less than 1 with xi*(r,j) should receive the gradient returned by the corresponding point yrj, so the back propagation formula of ROI Align is as follows: ,

[0048] In the above formula, d(.) represents the distance between two points, Δh and Δw represent the difference between the horizontal and vertical coordinates of xi and xi*(r,j), which are used as the coefficients of bilinear interpolation multiplied by the original gradient.

[0049] Set the loss function to L=L cls +L box +L mas k, where L cls and L boxThey are classification and regression losses respectively. For each region of interest ROI, the mask branch defines a K*m*2-dimensional matrix representing K different classifications. For each m*m region, there is one for each class. For each pixel, the sigmoid function is used to calculate the relative entropy and obtain the average relative entropy error L mask For each region of interest ROI, if the detection determines which category the region of interest ROI belongs to, the relative entropy error of that branch is used as the error value for calculation. Finally, it is exported as a json file for model building, and finally a test model of the surface segmentation process is obtained;

[0050] Select a suitable face division and face limiting threshold, and divide the original image containing three faces into three parts, namely the left face, the right face and the top face according to the established model.

[0051] Specifically, the face boundary points include a plurality of intersection points of two adjacent Rubik's Cube planes.

[0052] The boundary points are selected according to the principle of selecting the intersection points of the boundary lines as boundary points. The boundary points are limited to four, and the area finally divided by the boundary points and the boundary lines is a trapezoidal area.

[0053] Specifically, performing perspective transformation according to the surface boundary points to obtain a transformed image includes:

[0054] Obtain two coordinate arrays according to the first Rubik's Cube image and the second Rubik's Cube image respectively;

[0055] Calculate the perspective transformation matrix based on the two coordinate arrays;

[0056] Perspective transformation is performed on the first magic cube image and the second magic cube image according to the perspective transformation matrix to obtain a transformed image, wherein the transformed image is a vertical viewing angle.

[0057] Specifically, two coordinate arrays are read from the original image, and the perspective transformation matrix is ​​calculated; then the original image is perspective transformed according to the perspective transformation matrix to obtain a new image and output it to a new given plane, that is, to obtain a new vertical perspective transformed image. The transformation matrix is ​​as follows: .

[0058] Specifically, capturing each independent color block according to the transformed image includes:

[0059] According to the principle of similar colors, adjacent points with similar colors are clustered to obtain each independent nearly rectangular color block.

[0060] It should be understood that similar color points are clustered to capture a nearly rectangular color block for subsequent identification.

[0061] Specifically, performing image color recognition on each of the independent color blocks includes:

[0062] The independent color blocks are automatically identified according to the training model.

[0063] Specifically, the training model is obtained in the following way:

[0064] The Rubik's Cube image is subjected to labeling pre-training processing, and each Rubik's Cube plane is calibrated according to the direction to obtain all initial training images;

[0065] Performing perspective transformation on the initial training image to obtain a training transformed image;

[0066] The training transformed image is labeled to mark the color of each independent color block, thereby obtaining a training model.

[0067] It should be understood that the pre-trained vertical viewing angle block images are labeled as red, green, blue, orange, white, and yellow to obtain a training model; the specific training step algorithm is the same as the process of training the initial image in the previous article.

[0068] Based on the training model, the computer determines the color of the Rubik's Cube on its own.

[0069] The trained model is directly compared with the captured color block, and the similarity is used to determine which category it belongs to, that is, which color it is, and the recognition accuracy is obtained.

[0070] In summary, the image recognition method for a Rubik's Cube solving robot provided by the embodiment of the present invention has the following beneficial effects:

[0071] (1) The principle of perspective transformation is to project the image onto a completely new viewing plane, also known as projection mapping. OpenCV provides a number to implement the perspective transformation of an image. You only need to input the coordinates of the four vertices of the trapezoid and the coordinates of the four corners of the target canvas to automatically complete the transformation. The core code of perspective transformation is only two lines: first, read the two coordinate arrays according to the image and calculate the perspective transformation matrix; then perform perspective transformation on the original image according to the perspective transformation matrix, obtain a completely new image and output it to a new predetermined plane. Perspective transformation can well maintain the linearity of the original image.

[0072] In addition, the image recognition method provided by the embodiment of the present invention for use in a Rubik's Cube solving robot selects four vertices of a magic square after segmentation of the original image and performs perspective transformation on it, and finally obtains a magic square image of a vertical perspective, thereby greatly improving the accuracy of Rubik's Cube color block recognition.

[0073] (2) There are two main frameworks in the network architecture: training and prediction. The bottom layer of the main framework is essentially a multi-layer perceptron. Its success is mainly due to the local connection and shared weight method it adopts. The shared weight method can not only effectively suppress the risk caused by overfitting, but also reduce the number of weights and thus make the neural network easier to optimize. Convolutional neural network is a type of neural network. Because its weight sharing network structure is very similar to the structure of biological neural networks, convolutional neural networks can reduce the complexity of traditional network models and reduce the number of weights retained. When the input of the neural network is a multi-dimensional image, this advantage will be extremely prominent. It avoids the complex feature extraction process and data reverse reconstruction process in the traditional neural network recognition algorithm, and can also make the image directly used as the input of the neural network.

[0074] (3) The training results of the images after perspective transformation are much better than those before transformation, and the final prediction accuracy is also significantly different. After labeling the Rubik's Cube images, the training effect can be significantly improved to obtain a better model, and the final color block recognition accuracy will also be greatly improved. The model can complete recognition in situations such as poor clarity, insufficient light brightness, and occlusions, which shows that this algorithm is highly robust.

[0075] (4) The method of capturing color blocks is no longer the traditional method of capturing 1:1 rectangles, but a more accurate method of capturing color blocks through clustering. This also makes it possible to achieve the purpose of color block recognition even if the color blocks cannot be presented as rectangles due to occluding objects.

[0076] (5) This intelligent algorithm based on neural network improves recognition accuracy and robustness by training the model. This also shows that this algorithm is not only suitable for Rubik's Cube color block recognition, but can be applied to many similar problems. Through continuous training, this intelligent algorithm can recognize color blocks that traditional algorithms cannot recognize at all.

[0077] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. An image recognition method for a Rubik's Cube solving robot, characterized in that: include: Obtain a Rubik's Cube image, where the Rubik's Cube image is an image after the Rubik's Cube is scrambled; Performing image segmentation processing on the Rubik's Cube image to obtain the surface boundary points between every two adjacent planes of the Rubik's Cube; Perform perspective transformation according to the surface boundary points to obtain a transformed image; capturing each independent color patch according to the transformed image; Performing image color recognition on each of the independent color blocks; The Rubik's Cube image includes a first Rubik's Cube image and a second Rubik's Cube image, wherein the first Rubik's Cube image is a Rubik's Cube image obtained by shooting from a first perspective, and the first Rubik's Cube image includes three Rubik's Cube planes, and the second Rubik's Cube image is a Rubik's Cube image obtained by shooting from a second perspective, and the second Rubik's Cube image includes another three Rubik's Cube planes; The step of performing perspective transformation according to the surface boundary points to obtain a transformed image includes: Obtain two coordinate arrays according to the first Rubik's Cube image and the second Rubik's Cube image respectively; Calculate the perspective transformation matrix based on the two coordinate arrays; Perform perspective transformation on the first magic cube image and the second magic cube image according to the perspective transformation matrix to obtain transformation images, wherein the transformation images are vertical viewing angles; The step of capturing each independent color block according to the transformed image comprises: According to the principle of similar colors, adjacent points with similar colors are clustered to obtain each independent nearly rectangular color block.

2. The image recognition method for a Rubik's Cube solving robot according to claim 1, characterized in that: The step of performing image segmentation processing on the Rubik's Cube image to obtain the surface boundary points between every two adjacent planes of the Rubik's Cube includes: Performing image segmentation according to basic features of the first Rubik's Cube image and basic features of the second Rubik's Cube image, respectively, to obtain three Rubik's Cube planes corresponding to the first Rubik's Cube image and another three Rubik's Cube planes corresponding to the second Rubik's Cube image; The surface boundary points of two adjacent Rubik's Cube planes are selected according to the positions of the color blocks in the Rubik's Cube planes.

3. The image recognition method for a Rubik's Cube solving robot according to claim 2, characterized in that: The surface boundary points include a plurality of intersection points of two adjacent Rubik's Cube planes.

4. The image recognition method for a Rubik's Cube solving robot according to claim 1, characterized in that: The performing image color recognition on each of the independent color blocks includes: The independent color blocks are automatically identified according to the training model.

5. The image recognition method for a Rubik's Cube solving robot according to claim 4, characterized in that: The training model is obtained by: The Rubik's Cube image is subjected to labeling pre-training processing, and each Rubik's Cube plane is calibrated according to the direction to obtain all initial training images; Performing perspective transformation on the initial training image to obtain a training transformed image; The training transformed image is labeled to mark the color of each independent color block, thereby obtaining a training model.

Images