A camera-based real-time method and system for recognizing a state of a magic cube

Abstract

The application discloses a Rubik's cube state real-time identification method and system based on a camera, and the method comprises the following steps: image segmentation processing is performed on a camera image of a target Rubik's cube object to obtain a series of segmented object regions, and each object region is configured with a corresponding independent number; a parallelogram is used to perform region fitting on each independent numbered object region to determine all parallelogram regions existing on the image; and the states of each face of the target Rubik's cube object are determined by aggregating each parallelogram region according to the parallelogram regions. The embodiment of the application can identify the state of a Rubik's cube color block, has high accuracy and high speed, and can be widely applied to the computer technology field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a Rubik's cube state real-time identification method and system based on camera. BACKGROUND

[0002] Most of the existing Rubik's cube identification methods are neural network methods, such as YOLO, which are trained through pictures containing Rubik's cubes, and finally can identify the Rubik's cube.

[0003] The above method can only identify the position of the Rubik's cube in the image, cannot further identify the state of the Rubik's cube color block, and requires a large amount of calculation, which is difficult to realize real-time identification on mobile terminals such as mobile phones. SUMMARY

[0004] The main purpose of the embodiment of the present application is to provide a Rubik's cube state real-time identification method and system based on camera, which can identify the state of the Rubik's cube color block, has high accuracy and fast speed.

[0005] To achieve the above purpose, one aspect of the embodiment of the present application provides a Rubik's cube state real-time identification method based on camera, comprising the following steps:

[0006] The camera image of the target Rubik's cube object is subjected to image segmentation processing to obtain a series of segmented object regions, and each object region is configured with a corresponding independent number;

[0007] Each independent numbered object region is subjected to region fitting through a parallelogram to determine all parallelogram regions existing on the image;

[0008] According to each parallelogram region, each parallelogram region is aggregated to determine the state of each face of the target Rubik's cube object.

[0009] In some embodiments, the camera image of the target Rubik's cube object is subjected to image segmentation processing to obtain a series of segmented object regions, and each object region is configured with a corresponding independent number, comprising the following steps:

[0010] The number of colors in the camera image of the target Rubik's cube object is reduced;

[0011] Regions with the same pixels after color reduction are merged to complete the initial segmentation of the camera image of the target Rubik's cube object to obtain an initial segmentation region;

[0012] According to the total sum of the original image color cumulative value and the pixel number in the initial segmentation region, the average RGB value of the region is obtained, and the average RGB value of each region and its adjacent region is compared, and when the difference between them in the CIE Luv color space is less than a preset value, the pixel value accumulators and the pixel number are merged, and the rectangular region is updated, and finally the segmented object region is obtained, and each object region is configured with a corresponding independent number.

[0013] In some embodiments, the color number in the camera image of the target Rubik's object is reduced, specifically:

[0014] The color reduction processing is performed by the following formula: The color of the entire camera image is reduced to 16 colors, where R, G, B are the values of the original color, ranging from 0 to 255, R ′ , G ′ , B ′ are new values, ranging from 0 to 255, and the value of each channel is determined as one of {0, 90, 180}, reducing the color of the entire picture to 27 kinds;

[0015] The regions with the same reduced color pixels are merged to complete the initial segmentation of the camera image of the target Rubik's object, and the initial segmentation region is obtained, specifically:

[0016] The regions with the same reduced color pixels are merged, and in the merging process, each pixel is marked and the pixel number and the cumulative original pixel point RGB value in the region are recorded, the average pixel of the region is obtained, and the rectangular range where the region is located is updated, wherein the method of marking the pixel is: each time a new region is encountered, an id is assigned to the region, and the pixel is marked with the id to find the corresponding region through the pixel.

[0017] In some embodiments, the region fitting is performed on each independently numbered object region by a parallelogram to determine all parallelogram regions existing on the image, including the following steps:

[0018] The edges are scanned around the circumscribed rectangle of the region obtained after image segmentation to obtain edge line segments;

[0019] According to the obtained edge line segments, the angles, lengths and coordinates of the parallelogram are obtained;

[0020] Excluding incorrect parallelograms, specifically: checking whether the center coordinates of the parallelogram fall within the circumscribed rectangle of the region, and if not, excluding the parallelogram; comparing the actual region area with the area obtained by the parallelogram area formula, and filtering out the parallelogram with a large difference in the ratio.

[0021] In some embodiments, the method further comprises the following steps of:

[0022] The method further comprises the following steps of:

[0023] The method further comprises the following steps of:

[0024] The method further comprises the following steps of:

[0025] The method further comprises the following steps of: p b p b p p b b

[0026] The method further comprises the following steps of:

[0027] The method further comprises the following steps of:

[0028] ​​​​​​​​Obtaining the length of the parallelogram: when the two angles α and β are found, one of the angles α is used to screen the line segment, so that the screened line segment corresponds to the angle equal to the angle, the vector n is (sinα, cosα), the distance of the screened line segment to the vector n is counted, the two distances existing so that the most line segments are classified in them are found, the difference between the two distances is determined as the distance between the edges of the parallelogram with angle α, the distance is approximated as the length d1 of the edge with angle β, and the length d2 of the α edge is obtained, and the lengths of the two edges are obtained by counting twice;

[0029] Obtaining the coordinates of the parallelogram: confirming the intersection point between the two intersecting straight lines, one of which is angle α and the other of which is angle β, the length of the line segment perpendicular to the straight line with angle α is d1, and the length of the line segment perpendicular to the straight line with angle β is d2, the intersection point of d1 and d2 is found, and the intersection point is confirmed as the upper left corner coordinate of the parallelogram, and the coordinate calculation formula is: Further, all information of the parallelogram is obtained.

[0030] In some embodiments, the state of each face of the target Rubik's object is determined according to the aggregation of each parallelogram region, including the following steps:

[0031] Traversing the quadrilateral, finding the adjacent quadrilaterals in two directions of the parallelogram around the current quadrilateral, and checking whether the adjacent quadrilaterals are similar quadrilaterals with the current quadrilateral after the adjacent quadrilaterals are found;

[0032] After the checking is successful, a parallelogram group is created, and the parallelogram group is expanded according to the added parallelogram to enable the parallelogram group to contain the added parallelogram;

[0033] The parallelogram added to the group has a group number, and when it expands to the surrounding adjacent parallelogram or is contained by other quadrilaterals, the merging of the group is automatically implemented to form a larger group, and the state of each face of the target Rubik's object is determined.

[0034] Another aspect of the embodiment of the application also provides a camera-based Rubik's state real-time identification system, which comprises:

[0035] The first module is configured to perform image segmentation processing on the camera image of the target Rubik's object to obtain a series of segmented object regions, and each object region is configured with a corresponding independent number;

[0036] The second module is configured to perform region fitting on each independent number of the object region by using a parallelogram to determine all parallelogram regions existing on the image;

[0037] The third module is configured to aggregate each parallelogram region according to each parallelogram region, and determine the state of each face of the target Rubik's object.

[0038] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0039] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0040] The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the above method.

[0041] The embodiment of the present application at least has the following beneficial effects: the present application provides a Rubik's state real-time identification method and system based on a camera, the scheme performs image segmentation processing on the camera image of a target Rubik's object to obtain a series of segmented object regions, each object region is configured with a corresponding independent number, performs region fitting on each independent number of the object region through a parallelogram, determines all parallelogram regions existing on the image, aggregates each parallelogram region according to each parallelogram region, and determines the state of each face of the target Rubik's object. The embodiment of the present application can identify the state of a Rubik's color block, is high in accuracy and fast in speed. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is an implementation environment schematic diagram provided by the embodiment of the present application;

[0043] Figure 2 is a flowchart of the overall steps provided by the embodiment of the present application;

[0044] Figure 3 is a schematic diagram of an original image provided by the embodiment of the present application;

[0045] Figure 4 is a schematic diagram of an image after color reduction provided by the embodiment of the present application;

[0046] Figure 5 is a schematic diagram of an image after reducing the colors of the entire picture to 27 kinds provided by the embodiment of the present application;

[0047] Figure 6 is a schematic diagram of a pixel merging process provided by an embodiment of the present application;

[0048] Figure 7 is a schematic diagram of a parallelogram obtaining process provided by an embodiment of the present application;

[0049] Figure 8 is a schematic diagram of identification and aggregation of each face of a magic cube provided by an embodiment of the present application;

[0050] Figure 9 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0052] It can be understood that the terms "first", "second", and the like used herein can be used to describe various concepts in this document, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0053] The terms "at least one", "multiple", "each", "any", and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0055] The camera-based Rubik's cube state real-time identification method and system provided by the embodiment of the present application relates to the technical field of computers. The camera-based Rubik's cube state real-time identification method provided by the embodiment of the present application can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; the software can be an application that implements the camera-based Rubik's cube state real-time identification method, and the like, but is not limited to the above forms.

[0056] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0057] As shown in Figure 1 , it is an implementation environment schematic diagram provided by the embodiment of the present application. Referring to Figure 1 , the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected by wireless or wired means for network connection to complete data transmission and exchange.

[0058] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0059] In addition, server 101 can also be a node server in a blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0060] Terminal 102 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc. Terminal 102 may also be a vehicle-mounted terminal of the various device types described above, but is not limited thereto. Terminal 102 and server 101 may be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0061] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a method for real-time recognition of the state of a Rubik's Cube based on a camera. The following is an example of applying the method for real-time recognition of the state of a Rubik's Cube based on a camera in the server 101. It can be understood that the method can also be applied to the terminal 102.

[0062] Reference Figure 2 , Figure 2 This is a flowchart of a method for real-time recognition of a Rubik's Cube state based on a camera applied to a server according to an embodiment of the present invention. The execution subject of this method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:

[0063] Performing image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number;

[0064] Performing region fitting on each independently numbered object region by means of a parallelogram to determine all parallelogram regions existing on the image;

[0065] According to each parallelogram area, each parallelogram area is aggregated to determine the state of each face of the target Rubik's Cube object.

[0066] In some embodiments, the image segmentation processing is performed on the photographed image of the target Rubik's object to obtain a series of segmented object regions, and each object region is configured with a corresponding independent number, including the following steps:

[0067] reducing the number of colors in the photographed image of the target Rubik's object;

[0068] merging regions with the same pixels after color reduction to complete the initial segmentation of the photographed image of the target Rubik's object and obtain initial segmentation regions;

[0069] obtaining the average RGB value of each region according to the total sum of the original image color cumulative value and the number of pixels in the initial segmentation region, comparing the average RGB value of each region with that of its adjacent region, merging them when the difference between them in the CIE Luv color space is less than a preset value, simultaneously merging the pixel value accumulator and the number of pixels, and updating the rectangular region, and finally obtaining the segmented object region, each of which is configured with a corresponding independent number.

[0070] In some embodiments, the number of colors in the photographed image of the target Rubik's object is reduced, specifically as follows:

[0071] color reduction is performed by the following formula: reducing the colors of the entire photographed image to 16 colors, wherein R, G, and B are the values of the original colors, ranging from 0 to 255, R ′ , G ′ , and B ′ are new values ranging from 0 to 255, the value of each channel is determined as one of {0, 90, 180}, and the colors of the entire image are reduced to 27 kinds;

[0072] The merging of regions with the same pixels after color reduction to complete the initial segmentation of the photographed image of the target Rubik's object and obtain initial segmentation regions is specifically as follows:

[0073] Merging regions with the same pixels after color reduction, marking each pixel and recording the number of pixels and the cumulative original pixel point RGB value in the region during the merging process, obtaining the average pixel of the region, and updating the rectangular range where the region is located, wherein the method of marking pixels is: each time a new region is encountered, an id is assigned to the region, and the pixel is marked with the id to find the corresponding region through the pixel.

[0074] In some embodiments, the region fitting is performed on each independent number of the object region by a parallelogram to determine all parallelogram regions existing on the image, including the following steps:

[0075] scanning the edges of the circumscribed rectangle of the region obtained after image segmentation to obtain edge segments;

[0076] According to the obtained edge line segment, the angle, length and coordinate of the parallelogram are obtained;

[0077] The incorrect parallelogram is excluded, specifically: checking whether the center coordinate of the parallelogram falls in the circumscribed rectangle of the region, if not, the parallelogram is excluded; comparing the actual area of the region with the area obtained by the area formula of the parallelogram, and filtering out the parallelogram with a large difference in the ratio.

[0078] In some embodiments, the circumscribed rectangle of the region obtained after image segmentation is scanned for edges, and edge line segments are obtained, including the following steps:

[0079] The circumscribed rectangle of the region obtained after image segmentation is scanned for edges; the scanning method is: along one side of the circumscribed rectangle of the region, every 10 pixels are scanned along the perpendicular to the side until the pixel in the region is encountered or the rectangle boundary is exceeded, and a line segment can be determined between the two pixels belonging to the region that are scanned;

[0080] The appropriateness of the line segment is verified, and the verification method is: scanning the region boundary pixels between the two points to determine whether these pixels are on the line segment, if they are all on the line segment, it means that the line segment is part of the region edge, the line segment is retained, otherwise it is discarded;

[0081] Each side of the circumscribed rectangle is traversed, and a line segment chain is used to represent the boundary of the target region;

[0082] Wherein, the vector method is used to verify whether the pixel is on the corresponding straight line, specifically: according to the known line segment end point, the angle of the line segment is calculated, and the distance from the end point to the line segment is calculated by the formula d = (x p -x b )sinα-(y p -y b )cosα, wherein the point to be verified is point p(x p ,y p ), and points b(x b ,y b ) are any points at both ends, when the distance d from the end point to the line segment is less than or equal to 1, it is judged that the point is on the straight line.

[0083] In some embodiments, the angle, length and coordinate of the parallelogram are obtained according to the obtained edge line segment, including the following steps:

[0084] Obtain the angle of a parallelogram: Convert all angles to a range from 0 to π, then divide them into 100 parts. Count all line segments by interval, and determine the first and second largest angles in the interval as the angles corresponding to the two sides of the parallelogram. The following method is used to find the first and second largest peaks: first sort the angles in descending order, with the angle corresponding to the first value being the angle of one side of the parallelogram. Then, limit the minimum angle between the two angles of the parallelogram. When the angle corresponding to the next value is greater than the minimum angle of the first angle, the second angle has been found, and the angles of the two sides of the parallelogram are obtained.

[0085] Get the length of the parallelogram: When two angles α and β are found, use one of the angles α to filter the line segments so that the angle corresponding to the filtered line segments is equal to the angle. Let vector n be (sinα, cosα), count the distances between the filtered line segments and this vector n, and find the two distances that have the most line segments classified in them. Infer that the difference between these two distances is the distance between the sides of the parallelogram with angle α. Approximate this distance as the length d1 of the side with angle β, and obtain the length d2 of the side α. The lengths of both sides are obtained through two statistics.

[0086] Get the coordinates of the parallelogram: Determine the intersection point between two intersecting straight lines, where one angle is α and the other is β. The length of the line segment perpendicular to the line with angle α is d1, and the length of the line segment perpendicular to the line with angle β is d2. Find the intersection point of d1 and d2 and confirm this intersection point as the coordinate of the upper left corner of the parallelogram. The coordinate calculation formula is: And then get all the information of the parallelogram.

[0087] In some embodiments, the step of aggregating the parallelogram regions to determine the states of the faces of the target Rubik's Cube object includes the following steps:

[0088] Traverse the quadrilaterals and search for adjacent quadrilaterals along the two directions of the parallelogram. After finding the adjacent quadrilaterals, check whether they are similar to the current quadrilateral.

[0089] After the check is successful, a parallelogram group is created. This parallelogram group will expand itself according to the added parallelogram so that it can contain the added parallelogram;

[0090] The parallelograms added to the group have group numbers. When they expand to adjacent parallelograms or are included by other quadrilaterals, the groups will be automatically merged to form a larger group, which determines the status of each face of the target Rubik's Cube object.

[0091] The following will take specific application scenarios as examples to describe the specific implementation process of the embodiments of the application in detail.

[0092] It can be known through observation that in any state of the Rubik's Cube, the colors of the 9 small blocks on each face are not fixed, and the shapes formed by the small blocks of the same color can be 1x1, 2x2, 1x2, 1x3, or even not rectangular, but L-shaped, concave-shaped, etc. However, they all have a common point, that is, the directions of the edges of these shapes on the same face are only two directions and the directions are the same. Thus, the embodiments of the application obtain a fact that the color blocks on a face of the Rubik's Cube are divided into many regions according to colors, and the directions of the edges of the regions are necessarily the same. According to this fact, the embodiments of the application can describe the color blocks by parallelograms. For the color blocks that are not rectangular, such as L-shaped, concave-shaped, etc., it is also feasible to describe them by parallelograms. The method of describing the color blocks by parallelograms can tell us two most important information: the directions of the two edges and the positions of the color blocks. When a large number of parallelograms with equal edge directions are found to be closely gathered in the picture, it can be deduced that the regions where the parallelograms are located are a face of the Rubik's Cube.

[0093] The method of the application can be roughly divided into three steps in the recognition of the Rubik's Cube:

[0094] 1. Image segmentation - segmenting the image according to colors to form a series of regions, which are described in steps 1-3.

[0095] 2. Parallelogram determination - trying to represent the regions by parallelograms, which are described in steps 4-6.

[0096] 3. Quadrilateral aggregation - aggregating similar quadrilaterals, which is described in step 7.

[0097] 1. Image segmentation:

[0098] There are many existing methods for image segmentation, such as the watershed algorithm, the adaptive threshold algorithm, etc. These algorithms can well overcome the influence of light, camera interference, etc. However, they have essential defects in solving the present problem: 1. They can mostly only process grayscale images, and the distinct color feature of the Rubik's Cube color blocks will be lost in the grayscale images. 2. The operation amount is large, which is a great pressure for mobile terminal devices that need real-time processing tasks. In view of the above, the present method proposes an image segmentation algorithm in the RGB color space that can both take advantage of the distinct color feature of the Rubik's Cube color blocks and make the operation amount small.

[0099] The concept of "color difference" can be reflected in the color space CIE Luv. This algorithm uses the CIE Luv color space as a method to measure color differences. In the CIE Luv space, humans' intuitive perception of color differences can be measured by the Euclidean distance of colors in this space. By utilizing this, the embodiment of the present invention can divide regions so that the color differences in the regions are within a certain range. However, since the RGB space to the CIE Luv space is nonlinear, the amount of calculation is large. In order to reduce the amount of calculation, this program is divided into three steps in image segmentation: Step 1: Reduce the number of colors in the picture. Step 2: Initially divide the region so that the colors within the region are the same. Step 3: Calculate the color differences of adjacent regions through CIE Luv and merge adjacent regions. The following is a detailed description of these three steps.

[0100] Step 1: Reduce the number of colors:

[0101] The purpose of color reduction is to reduce the amount of subsequent operations and prepare for the initial segmentation of the area. Figure 3 The original image of the Rubik's Cube shown in the figure has large color differences between different blocks. Therefore, the color reduction algorithm will reduce the color as aggressively as possible as long as it does not merge the blocks together. Based on a lot of practical experience, we can use formula 1 to reduce the color of the entire image to 16 colors. The image after color reduction is as follows: Figure 4 As shown, R, G, and B are the original color values, ranging from 0 to 255, and R', G', and B' are new values, also ranging from 0 to 255. Through this operation, the color value of each channel is one of {0, 90, 180}, so the embodiment of the present invention reduces the number of colors of the entire picture to 27 (such as Figure 5 shown).

[0102]

[0103] Step 2: Initial area segmentation

[0104] The initial segmentation method is to merge the areas with the same pixels after color subtraction. During the merging process, mark each pixel and record the number of pixels in the area and the RGB value of the pixel corresponding to the accumulated original image. This makes it convenient to obtain the average pixel of the area and update the rectangular range where the area is located (record the leftmost, rightmost, topmost and bottommost coordinates). The method of marking pixels is: each time a new area is encountered, an id is assigned to the area (generally speaking, the id is generated by a non-negative increasing counter) and the pixel is marked with the id, so that we can find the corresponding area through the pixel.

[0105] Step 3: Merge adjacent regions

[0106] like Figure 6As shown, the sum of the accumulated color values ​​of the original image and the number of pixels in the region obtained in step 2 can be used to obtain the average RGB value of the region. Each region is compared with the average RGB value of its neighboring region. When the difference in the CIE Luv color space is less than a certain value (here, 20 is empirically used), they are merged. At the same time, the pixel value accumulator and the number of pixels are merged, and the rectangular region is updated. In this way, the embodiment of the present invention finally obtains the segmented image.

[0107] from Figure 6 It can be seen that the color blocks of the Rubik's Cube are well separated. Figure 6 A represents the image after color reduction; B represents the first stage pixel merging diagram; C represents the second stage pixel merging diagram; D represents the third stage pixel merging diagram. The green box represents the rectangular boundary of the region, and the pink pixels represent the pixels belonging to the region ID. Figure 4 The segmentation defects caused by illumination in the initial segmentation are solved in step 3, as shown in Figure 6 As shown in B. Figure 4 Some pixels in the red block are subtracted to another color, but are merged together again in step 3.

[0108] 2. Parallelogram determination:

[0109] After image segmentation, the present invention obtains a series of regions, each labeled with a different number. The next step is to attempt to fit each region with a parallelogram. This parallelogram fitting process can be divided into three steps: Step 4: Obtain edge segments, using small line segments to outline the region's edges, and obtain a chain of line segments enclosing the region. Step 5: Use statistical methods to determine the directions, lengths, and coordinates of the parallelogram's sides. Step 6: Eliminate incorrectly identified quadrilaterals. The following is a detailed description of these three steps.

[0110] Step 4. Get edge segments:

[0111] refer to Figure 7 ,in, Figure 7 A in the figure represents the edge acquisition diagram; B represents the line segment chain representing the edge; C represents the line segment angle statistics of B; and D represents the parallelogram acquisition. The edge is scanned around the circumscribed rectangle of the region obtained after image segmentation. The scanning method is: scan perpendicular to one side of the circumscribed rectangle of the region every 10 pixels until a pixel in the region is encountered or the rectangle boundary is exceeded. A line segment can be determined between two pixels belonging to the region. Then, the suitability of this line segment is verified by scanning the region boundary pixels between the two points to determine whether these pixels are on the line segment. If they are, it means that the line segment is part of the region edge and is retained. Otherwise, it is discarded (e.g. Figure 7The principle shown in A of FIG. 6). In this way, each edge of the circumscribed rectangle is traversed, and by this method, the embodiments of the present application can express the boundary of the target region by a chain of line segments (as shown in B of FIG. 6). Figure 7 The verification of whether a pixel is on a straight line can be calculated by the method of vectors. Since the end points of the line segment are known, the angle of the line segment can be calculated as α, and the distance d of the point to the line segment can be expressed by formula 2, where the point to be verified is point p(x p ,y p ), and point b(x b ,y b ) is any point at the two ends. When d is less than or equal to 1, the embodiments of the present application can determine that the point is on the straight line.

[0112] d = (x p -x b ) sin α - (y p -y b ) cos α (2)

[0113] Step 5, obtain the angles, lengths, and coordinates of the parallelogram:

[0114] If the region is composed of Rubik's color blocks, then it is obvious that the length of the edge of the region composed of color blocks of the same color on each face corresponds to only two directions of the line segment direction, and then in the set of line segments obtained in the previous step, if classified according to the angle, there are two angles such that the line segments classified in them are the most. According to this conclusion, all angles are converted to 0 to π, and then divided into 100 parts, and all line segments are counted according to the interval, and the first and second largest angles of the peak value of the interval are the angles corresponding to the two edges of the parallelogram. The first and second peak values can be found by the longest non-decreasing subsequence, but this method is easily affected by the large value of a certain point in a local area, so that the second peak value cannot be reached. The solution of this method is: first sort these angles in descending order, and the angle corresponding to the first value is the angle of one edge of the parallelogram, and then limit the minimum angle of the two angles of the parallelogram. When the angle corresponding to the next value is greater than the minimum angle of the first angle, it indicates that the second angle has been found. In this way, the embodiments of the present application obtain the angles of the two edges of the parallelogram, as shown in C of FIG. 6, where white represents the peak value determined by the program. Figure 7

[0115] ​Next, the lengths of two sides of the parallelogram are obtained. When the two angles a, b are found, the embodiment of the present application screens the line segments with one of the angles a, so that the screened line segments correspond to the angle. Let the vector n be (sin a, cos a), and the distance of the screened line segments to the vector n is counted. It is found that there are two distances, and the line segments classified in them are the most. Then it can be inferred that the difference between the two distances is the distance between the sides of the parallelogram with the angle a, and the distance is approximated as the length d1 of the side with the angle b. Similarly, the length d2 of the side with the angle a is obtained. Thus, the lengths of the two sides are obtained by counting twice. Here, the distance formula is formula (2), but x b = 0, y b = 0; p(x p , y p ) is any point on the screened line segment.

[0116] Next, the coordinates of the upper left corner of the parallelogram are obtained, which is an easy mathematical problem: given two intersecting straight lines, the intersection point is (0, 0), one angle is a, and the other angle is b, the length of the line segment perpendicular to the straight line with the angle a is d1, and the length of the line segment perpendicular to the straight line with the angle b is d2. Find the intersection point of d1 and d2, which is the coordinates of the upper left corner of the parallelogram. Through derivation, the coordinates t(x t , y t ) can be obtained by formula (3).

[0117]

[0118] Thus, all the information of the parallelogram is obtained, as shown by D of Figure 7 .

[0119] step 6, eliminate false parallelograms:

[0120] Because the present application attempts to fit parallelograms to all regions, there will be many interfering quadrilaterals. It can be found from formula (3) that this formula has an excellent property. When the non-parallelogram region is determined, due to the unreliability of the angle and side length data, the calculated upper left corner coordinates of the parallelogram will not match the position of the region, so the embodiment of the present application can check whether the center coordinates of the parallelogram fall within the circumscribed rectangle of the region. If not, the parallelogram can be excluded. The comparison of the area can also be used. When the image is segmented, the embodiment of the present application has recorded the number of pixel points contained in the region. The actual area of the region (the number of pixel points) can be compared with the area obtained by the area formula of the parallelogram, and the parallelogram with a large difference in the ratio can be filtered out.

[0121] This method of using statistics to obtain parallelogram has strong robustness, and can infer parallelogram even in the presence of interference and partial image loss, and the calculation amount is small enough to meet the requirements of real-time analysis and processing on embedded and low-power platforms.

[0122] Step 7, parallelogram aggregation:

[0123] Reference Figure 8 , Figure 8 (a) in (a) represents all the parallelograms detected, Figure 8 (b) in (b) represents the aggregated surface. So far, the embodiment of the present application has found all the parallelograms that may exist on the image. Now it is necessary to infer the surface of the magic cube through them, and here the embodiment of the present application uses a search method. The method is as follows: traversing the quadrilateral, finding the adjacent quadrilateral in the two directions of the parallelogram, if the adjacent quadrilateral is found, checking whether it is a similar quadrilateral (the angle error of two sides is within the allowable range), and if the checking is successful, creating a group of parallelograms, and the parallelogram will expand itself according to the added parallelogram, so that it can contain the added parallelogram. The parallelogram added to the group has a group number, and when it expands to the adjacent parallelogram or is contained by other parallelograms, the merging of the group will be automatically implemented, thereby forming a larger group. The advantage of this is that when the magic cube is blocked or some color blocks are not recognized, the correct surface can still be found.

[0124] In summary, the present application has the following advantages:

[0125] 1. The present application has an innovation point in color segmentation, which uses subtractive color, merging, and uses CIE Luv to judge and then merges, so as to segment out the color consistent region consistent with human intuition.

[0126] 2. In the process of parallelogram fitting, the present application uses statistical methods, so that the recognition ability and anti-interference ability are stronger, and the regions of non-parallelogram shapes are effectively excluded.

[0127] Another aspect of the embodiment of the present application also provides a camera-based magic cube state real-time recognition system, comprising:

[0128] The first module is configured to perform image segmentation processing on the camera image of the target magic cube object to obtain a series of segmented object regions, and each object region is configured with a corresponding independent number;

[0129] The second module is configured to perform region fitting on each independent number of the object region by using a parallelogram to determine all the parallelogram regions existing on the image;

[0130] The third module is configured to aggregate each parallelogram region according to each parallelogram region to determine the state of each face of the target Rubik's Cube object.

[0131] It can be understood that the content in the above method embodiments is applicable to the present system embodiments, the present system embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0132] The present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned real-time Rubik's Cube state recognition method based on a camera when executing the computer program. The electronic device can be any smart terminal, such as a tablet computer or a vehicle-mounted computer.

[0133] It can be understood that the content in the above method embodiments is applicable to the present device embodiments, the present device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0134] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0135] The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application;

[0136] The memory 902 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and called and executed by the processor 901 to implement the real-time Rubik's Cube state recognition method based on a camera of the embodiments of the present application;

[0137] The input / output interface 903 is used to realize information input and output;

[0138] The communication interface 904 is configured to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0139] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0140] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected to each other through the bus 905.

[0141] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the above-mentioned camera-based Rubik's cube state real-time identification method.

[0142] It can be understood that the contents in the above-mentioned method embodiments are all applicable to the present storage medium embodiment, the function specifically realized by the present storage medium embodiment is the same as that of the above-mentioned method embodiments, and the beneficial effects achieved by the present storage medium embodiment are also the same as those achieved by the above-mentioned method embodiments.

[0143] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0144] It should be noted that, in each specific embodiment of the present application, when relevant processing needs to be performed according to user information, user behavior data, user historical data, and user location information and other data related to the identity or characteristics of the user, the permission or consent of the user is obtained first, and the collection, use, and processing of the data are performed in compliance with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user is obtained through a pop-up window or a jump to a confirmation page, and after the separate permission or separate consent of the user is obtained, the necessary user-related data for enabling the embodiments of the present application to normally operate is obtained.

[0145] The embodiments described in the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0146] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0147] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0148] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0149] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0150] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0151] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0152] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0153] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0154] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0155] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A real-time recognition method for the Rubik's Cube state based on a camera, characterized in that: The following steps are involved: Performing image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number; Performing region fitting on each independently numbered object region by means of a parallelogram to determine all parallelogram regions existing on the image; According to each parallelogram area, aggregate each parallelogram area to determine the state of each face of the target Rubik's Cube object; The method of performing region fitting on each independently numbered object region by using a parallelogram to determine all parallelogram regions existing on the image comprises the following steps: Scan the edges of the circumscribed rectangle of the region obtained after image segmentation to obtain edge segments; According to the obtained edge segments, the angle, length and coordinates of the parallelogram are obtained; Eliminate incorrect parallelograms by: checking whether the center coordinates of the parallelogram fall within the circumscribed rectangle of the region; if not, exclude the parallelogram; compare the actual area of ​​the region with the area obtained by the parallelogram area formula, and filter out parallelograms with a large difference in ratio; Scanning the edges of the circumscribed rectangle of the region obtained after image segmentation to obtain edge segments comprises the following steps: Scan the edges of the circumscribed rectangle of the region obtained after image segmentation. The scanning method is: scan perpendicularly to one side of the circumscribed rectangle of the region every 10 pixels until a pixel in the region is encountered or the rectangle is beyond the boundary. A line segment can be determined between two pixels belonging to the region. Verify the suitability of this line segment. The verification method is as follows: scan the boundary pixels of the region between the two points and determine whether these pixels are on the line segment. If they are, it means that the line segment is part of the region edge and is retained. Otherwise, it is discarded. Traverse each side of the circumscribed rectangle and use a line segment chain to represent the boundary of the target area; The vector method is used to verify whether the pixel is on the corresponding straight line. Specifically, the angle of the line segment is calculated based on the known endpoints of the line segment, and the angle of the line segment is calculated by the formula d=(x p -x b )sinα-(y p -y b )cosβ to calculate the distance from the point to be verified to the line segment, where the point to be verified is point p(x p ,y p ), point b(x b ,y b ) is any point at either end of the line segment. When the distance d from the point p to be verified to the line segment is less than or equal to 1, the point to be verified is judged to be on the straight line.

2. The method for real-time recognition of the state of a Rubik's Cube based on a camera according to claim 1, characterized in that: The image segmentation process is performed on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number, including the following steps: Reducing the number of colors in the camera image of the target Rubik's Cube object; Merging the regions with the same pixels after color subtraction to complete the initial segmentation of the camera image of the target Rubik's Cube object and obtain the initial segmented regions; Based on the sum of the original image color cumulative values ​​and the number of pixels in the initially segmented area, the average RGB value of the area is obtained. Each area is compared with the average RGB value of its neighboring area. When the difference between them in the CIE Luv color space is less than a preset value, they are merged. At the same time, the pixel value accumulator and the number of pixels are merged, and the rectangular area is updated. Finally, the segmented object area is obtained, and each object area is assigned a corresponding independent number.

3. The method for real-time recognition of the state of a Rubik's Cube based on a camera according to claim 2, characterized in that: The method of reducing the number of colors in the camera image of the target Rubik's Cube object is specifically as follows: The color reduction process is performed using the following formula: Reduce the color of the entire camera image to 16 colors, where R, G, and B are the values ​​of the original color, ranging from 0 to 255. ′ , G ′ 、B ′ is a new value ranging from 0 to 255, and the color value of each channel is determined to be one of {0, 90, 180}, reducing the number of colors of the entire picture to 27; The regions with the same pixels after color reduction are merged to complete the initial segmentation of the camera image of the target Rubik's Cube object, and obtain the initial segmented regions, specifically: Merge the areas with the same pixels after color subtraction. During the merging process, mark each pixel and record the number of pixels in the area and the RGB value of the pixel corresponding to the accumulated original image. Get the average pixel of the area and update the rectangular range where the area is located. The method of marking pixels is: each time a new area is encountered, assign an id to the area and use the id to mark the pixel so that the corresponding area can be found through the pixel.

4. The method for real-time recognition of the state of a Rubik's Cube based on a camera according to claim 1, characterized in that: The method of obtaining the angle, length, and coordinates of the parallelogram based on the acquired edge segments includes the following steps: Obtain the angle of a parallelogram: Convert all angles to a range from 0 to π, then divide them into 100 parts. Count all line segments by interval, and determine the first and second largest angles in the interval as the angles corresponding to the two sides of the parallelogram. The following method is used to find the first and second largest peaks: first sort the angles in descending order, with the angle corresponding to the first value being the angle of one side of the parallelogram. Then, limit the minimum angle between the two angles of the parallelogram. When the angle corresponding to the next value is greater than the minimum angle of the first angle, the second angle has been found, and the angles of the two sides of the parallelogram are obtained. Get the length of the parallelogram: When two angles α and β are found, use one of the angles α to filter the line segments so that the angle corresponding to the filtered line segments is equal to the angle. Let vector n be (sinα, cosα), count the distances between the filtered line segments and this vector n, and find the two distances that have the most line segments classified in them. Infer that the difference between these two distances is the distance between the sides of the parallelogram with angle α. Approximate this distance as the length d1 of the side with angle β, and obtain the length d2 of the side α. The lengths of both sides are obtained through two statistics. Get the coordinates of the parallelogram: Determine the intersection point between two intersecting straight lines, where one angle is α and the other is β. The length of the line segment perpendicular to the line with angle α is d1, and the length of the line segment perpendicular to the line with angle β is d2. Find the intersection point of d1 and d2 and confirm this intersection point as the coordinate of the upper left corner of the parallelogram. The coordinate calculation formula is: And then get all the information of the parallelogram.

5. The method for real-time recognition of the state of a Rubik's Cube based on a camera according to claim 1, characterized in that: The method of aggregating the parallelogram regions according to the parallelogram regions to determine the states of the faces of the target Rubik's Cube object includes the following steps: Traverse the quadrilaterals and search for adjacent quadrilaterals along the two directions of the parallelogram. After finding the adjacent quadrilaterals, check whether they are similar to the current quadrilateral. After the check is successful, a parallelogram group is created. This parallelogram group will expand itself according to the added parallelogram so that it can contain the added parallelogram; The parallelograms added to the group have group numbers. When they expand to adjacent parallelograms or are included by other quadrilaterals, the groups will be automatically merged to form a larger group, which determines the status of each face of the target Rubik's Cube object.

6. A system for implementing the camera-based real-time recognition method for a Rubik's Cube according to any one of claims 1 to 5, characterized in that: include: The first module is used to perform image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number; A second module is configured to perform region fitting on each independently numbered object region through a parallelogram to determine all parallelogram regions existing on the image; The third module is used to aggregate the parallelogram areas according to the parallelogram areas to determine the status of each face of the target Rubik's Cube object.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images