Rubik's cube state real-time identification method and system based on camera

Abstract

The invention discloses a Rubik's cube state real-time identification method and system based on a camera, and the method comprises the steps: carrying out the image segmentation processing of a camera image of a target Rubik's cube object, obtaining a series of segmented object regions, and configuring a corresponding independent number for each object region; performing region fitting on each independently numbered object region through a parallelogram, and determining all parallelogram regions existing on the image; and according to each parallelogram region, aggregating each parallelogram region, and determining the state of each surface of the target magic cube object. The state of the color blocks of the magic cube can be identified, the accuracy is high, the speed is high, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a camera-based real-time recognition method and system for a magic cube state. Background Art

[0002] Most of the existing Rubik's Cube recognition methods use neural network methods, such as YOLO, which is trained by pictures containing Rubik's Cube and can eventually recognize the Rubik's Cube.

[0003] The above method can only identify the position of the Rubik's Cube in the image, but cannot further identify the state of the Rubik's Cube color blocks. It also requires a large amount of calculation, and it is difficult to perform real-time recognition on mobile terminals such as mobile phones. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to provide a method and system for real-time recognition of the state of a Rubik's Cube based on a camera, which can recognize the state of the color blocks of the Rubik's Cube with high accuracy and high speed.

[0005] To achieve the above object, one aspect of an embodiment of the present invention provides a method for real-time recognition of a Rubik's Cube state based on a camera, comprising the following steps:

[0006] Performing image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number;

[0007] Performing region fitting on each independently numbered object region through a parallelogram to determine all parallelogram regions existing on the image;

[0008] According to each parallelogram area, each parallelogram area is aggregated to determine the state of each face of the target Rubik's Cube object.

[0009] In some embodiments, the image segmentation process is performed on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number, including the following steps:

[0010] Reducing the number of colors in the camera image of the target Rubik's Cube object;

[0011] Merging the regions with the same pixels after color reduction to complete the initial segmentation of the camera image of the target Rubik's Cube object and obtain the initial segmentation region;

[0012] According to the sum of the color cumulative values ​​of the original image and the number of pixels in the initially segmented area, the average RGB value of the area is obtained, each area is compared with the average RGB value of its neighboring area, and when the difference between them in the CIE Luv color space is less than a preset value, they are merged, and the pixel value accumulator and the number of pixels are merged at the same time, and the rectangular area is updated, and finally the segmented object area is obtained, and each object area is configured with a corresponding independent number.

[0013] In some embodiments, reducing the number of colors in the camera image of the target Rubik's Cube object is specifically:

[0014] The color reduction process is performed by the following formula: Reduce the color of the entire camera image to 16 colors, where R, G, and B are the values ​​of the original colors, ranging from 0 to 255. ′ , G ′ , B ′ is a new value ranging from 0 to 255, and the color value of each channel is determined to be one of {0, 90, 180}, reducing the number of colors of the entire picture to 27;

[0015] The regions with the same pixels after color reduction are merged to complete the initial segmentation of the camera image of the target Rubik's Cube object to obtain the initial segmentation region, which is specifically:

[0016] Merge the areas with the same pixels after color subtraction. During the merging process, mark each pixel and record the number of pixels in the area and the RGB value of the pixel corresponding to the accumulated original image, obtain the average pixel of the area, and update the rectangular range where the area is located. The method of marking pixels is: each time a new area is encountered, an id is assigned to the area, and the id is used to mark the pixel so that the corresponding area can be found through the pixel.

[0017] In some embodiments, performing region fitting on each independently numbered object region by means of a parallelogram to determine all parallelogram regions existing on the image comprises the following steps:

[0018] Scan the edge of the circumscribed rectangle around the area obtained after image segmentation to obtain edge segments;

[0019] According to the obtained edge segments, the angle, length and coordinates of the parallelogram are obtained;

[0020] Eliminate erroneous parallelograms by: checking whether the center coordinates of the parallelogram fall within the circumscribed rectangle of the region; if not, exclude the parallelogram; compare the actual area of ​​the region with the area obtained by the parallelogram area formula, and filter out parallelograms with too large a difference in ratio.

[0021] In some embodiments, scanning the edge of the circumscribed rectangle around the area obtained after image segmentation to obtain edge segments includes the following steps:

[0022] Scan the edge of the circumscribed rectangle of the region obtained after image segmentation; the scanning method is: scan perpendicularly to one side of the circumscribed rectangle of the region every 10 pixels until a pixel in the region is encountered or beyond the rectangular boundary, and a line segment can be determined between two pixels belonging to the region;

[0023] Verify the suitability of this line segment. The verification method is: scan the region boundary pixels between the two points and determine whether these pixels are on the line segment. If they are, it means that the line segment is part of the region edge and is retained. Otherwise, it is discarded.

[0024] Traverse each edge of the circumscribed rectangle and use a line segment chain to represent the boundary of the target area;

[0025] The vector method is used to verify whether the pixel is on the corresponding straight line. Specifically, the angle of the line segment is calculated based on the known endpoints of the line segment, and the angle of the line segment is calculated by the formula d = (x p -x b )sinα-(y p -y b )cosα to calculate the distance from the endpoint to the line segment, where the point to be verified is point p(x p ,y p ), point b(x b ,y b ) is any point at either end. When the distance d from the endpoint to the line segment is less than or equal to 1, the point is judged to be on the straight line.

[0026] In some embodiments, obtaining the angle, length, and coordinates of the parallelogram according to the acquired edge segments comprises the following steps:

[0027] Get the angle of the parallelogram: convert all angles to 0 to π, divide them into 100 parts, count all line segments by interval, and determine the first and second largest angles of the interval peak as the angles corresponding to the two sides of the parallelogram; the following method is used to find the first and second largest peaks: first sort the angles in descending order, the angle corresponding to the first value is the angle of one side of the parallelogram, then limit the minimum angle of the two angles of the parallelogram, and when the angle corresponding to the next value is greater than the minimum angle of the first angle, it indicates that the second angle has been found, and then the angles of the two sides of the parallelogram are obtained;

[0028] Get the length of the parallelogram: When two angles α and β are found, use one of the angles α to filter the line segments so that the angle corresponding to the filtered line segments is equal to the angle. Let vector n be (sinα, cosα), count the distances from the filtered line segments to this vector n, find the two distances that have the most line segments classified in them, and infer that the difference between the two distances is the distance between the sides of the parallelogram with angle α. Approximate this distance as the length d1 of the side with angle β, and obtain the length d2 of the α side. Obtain the lengths of the two sides through two statistics;

[0029] Get the coordinates of the parallelogram: determine the intersection point between two intersecting straight lines, where one angle is α and the other angle is β. The length of the line segment perpendicular to the line with angle α is d1, and the length of the line segment perpendicular to the line with angle β is d2. Find the intersection point of d1 and d2, and determine the intersection point as the coordinate of the upper left corner of the parallelogram. The coordinate calculation formula is: And then get all the information of the parallelogram.

[0030] In some embodiments, the step of aggregating the parallelogram regions according to the parallelogram regions to determine the states of the faces of the target Rubik's Cube object includes the following steps:

[0031] Traverse the quadrilaterals and search for adjacent quadrilaterals around the current quadrilateral along the two directions of the parallelogram. After finding the adjacent quadrilaterals, check whether they are similar to the current quadrilateral.

[0032] After the check succeeds, a parallelogram group is created, and this parallelogram group will expand itself according to the added parallelogram so that it can contain the added parallelogram;

[0033] The parallelograms added to the group have group numbers. When they expand to the surrounding adjacent parallelograms or are included by other quadrilaterals, the groups will be automatically merged to form a larger group, determining the status of each face of the target Rubik's Cube object.

[0034] Another aspect of the embodiment of the present invention further provides a camera-based real-time recognition system for a Rubik's Cube state, comprising:

[0035] The first module is used to perform image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number;

[0036] The second module is used to perform region fitting on each independently numbered object region through a parallelogram to determine all parallelogram regions existing on the image;

[0037] The third module is used to aggregate the parallelogram regions according to the parallelogram regions to determine the status of each face of the target Rubik's Cube object.

[0038] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0039] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0040] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0041] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a method and system for real-time recognition of the state of a Rubik's Cube based on a camera, the scheme performs image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each object region is configured with a corresponding independent number; region fitting is performed on each independently numbered object region through a parallelogram to determine all parallelogram regions existing on the image; according to each parallelogram region, each parallelogram region is aggregated to determine the state of each face of the target Rubik's Cube object. The embodiments of the present invention can recognize the state of the Rubik's Cube color block with high accuracy and high speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0043] Figure 2 is a flow chart of overall steps provided by an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of an original image provided by an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of a color-reduced image provided by an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of reducing the number of colors of the entire screen to 27 provided by an embodiment of the present invention;

[0047] Figure 6 is a schematic diagram of a pixel merging process provided by an embodiment of the present invention;

[0048] Figure 7 is a schematic diagram of a parallelogram acquisition process provided by an embodiment of the present invention;

[0049] Figure 8 is a schematic diagram of identification and aggregation of various faces of a Rubik's Cube provided by an embodiment of the present invention;

[0050] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0052] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0053] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0055] The camera-based real-time recognition method and system for the state of a Rubik's Cube provided in an embodiment of the present invention relate to the field of computer technology. The camera-based real-time recognition method for the state of a Rubik's Cube provided in an embodiment of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a camera-based real-time recognition method for the state of a Rubik's Cube, etc., but is not limited to the above forms.

[0056] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0057] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0058] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0059] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0060] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0061] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a method for real-time recognition of the state of a Rubik's Cube based on a camera. The following is explained using the example of the real-time recognition of the state of a Rubik's Cube based on a camera applied to a server 101. It can be understood that the method can also be applied to a terminal 102.

[0062] Reference Figure 2 , Figure 2 The flowchart of the method for real-time recognition of the state of a Rubik's Cube based on a camera applied to a server provided in an embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:

[0063] Performing image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number;

[0064] Performing region fitting on each independently numbered object region through a parallelogram to determine all parallelogram regions existing on the image;

[0065] According to each parallelogram area, each parallelogram area is aggregated to determine the state of each face of the target Rubik's Cube object.

[0066] In some embodiments, the image segmentation process is performed on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number, including the following steps:

[0067] Reducing the number of colors in the camera image of the target Rubik's Cube object;

[0068] Merging the regions with the same pixels after color reduction to complete the initial segmentation of the camera image of the target Rubik's Cube object and obtain the initial segmentation region;

[0069] According to the sum of the color cumulative values ​​of the original image and the number of pixels in the initially segmented area, the average RGB value of the area is obtained, each area is compared with the average RGB value of its neighboring area, and when the difference between them in the CIE Luv color space is less than a preset value, they are merged, and the pixel value accumulator and the number of pixels are merged at the same time, and the rectangular area is updated, and finally the segmented object area is obtained, and each object area is configured with a corresponding independent number.

[0070] In some embodiments, reducing the number of colors in the camera image of the target Rubik's Cube object is specifically:

[0071] The color reduction process is performed by the following formula: Reduce the color of the entire camera image to 16 colors, where R, G, and B are the values ​​of the original colors, ranging from 0 to 255. ′ , G ′ , B ′ is a new value ranging from 0 to 255, and the color value of each channel is determined to be one of {0, 90, 180}, reducing the number of colors of the entire picture to 27;

[0072] The regions with the same pixels after color reduction are merged to complete the initial segmentation of the camera image of the target Rubik's Cube object to obtain the initial segmentation region, which is specifically:

[0073] Merge the areas with the same pixels after color subtraction. During the merging process, mark each pixel and record the number of pixels in the area and the RGB value of the pixel corresponding to the accumulated original image, obtain the average pixel of the area, and update the rectangular range where the area is located. The method of marking pixels is: each time a new area is encountered, an id is assigned to the area, and the id is used to mark the pixel so that the corresponding area can be found through the pixel.

[0074] In some embodiments, performing region fitting on each independently numbered object region by means of a parallelogram to determine all parallelogram regions existing on the image comprises the following steps:

[0075] Scan the edge of the circumscribed rectangle around the area obtained after image segmentation to obtain edge segments;

[0076] According to the obtained edge segments, the angle, length and coordinates of the parallelogram are obtained;

[0077] Eliminate erroneous parallelograms by: checking whether the center coordinates of the parallelogram fall within the circumscribed rectangle of the region; if not, exclude the parallelogram; compare the actual area of ​​the region with the area obtained by the parallelogram area formula, and filter out parallelograms with too large a difference in ratio.

[0078] In some embodiments, scanning the edge of the circumscribed rectangle around the area obtained after image segmentation to obtain edge segments includes the following steps:

[0079] Scan the edge of the circumscribed rectangle of the region obtained after image segmentation; the scanning method is: scan perpendicularly to one side of the circumscribed rectangle of the region every 10 pixels until a pixel in the region is encountered or beyond the rectangular boundary, and a line segment can be determined between two pixels belonging to the region;

[0080] Verify the suitability of this line segment. The verification method is: scan the region boundary pixels between the two points and determine whether these pixels are on the line segment. If they are, it means that the line segment is part of the region edge and is retained. Otherwise, it is discarded.

[0081] Traverse each edge of the circumscribed rectangle and use a line segment chain to represent the boundary of the target area;

[0082] The vector method is used to verify whether the pixel is on the corresponding straight line. Specifically, the angle of the line segment is calculated based on the known endpoints of the line segment, and the angle of the line segment is calculated by the formula d = (x p -x b )sinα-(y p -y b )cosα to calculate the distance from the endpoint to the line segment, where the point to be verified is point p(x p ,y p ), point b(x b ,y b ) is any point at either end. When the distance d from the endpoint to the line segment is less than or equal to 1, the point is judged to be on the straight line.

[0083] In some embodiments, obtaining the angle, length, and coordinates of the parallelogram according to the acquired edge segments comprises the following steps:

[0084] Get the angle of the parallelogram: convert all angles to 0 to π, divide them into 100 parts, count all line segments by interval, and determine the first and second largest angles of the interval peak as the angles corresponding to the two sides of the parallelogram; the following method is used to find the first and second largest peaks: first sort the angles in descending order, the angle corresponding to the first value is the angle of one side of the parallelogram, then limit the minimum angle of the two angles of the parallelogram, and when the angle corresponding to the next value is greater than the minimum angle of the first angle, it indicates that the second angle has been found, and then the angles of the two sides of the parallelogram are obtained;

[0085] Get the length of the parallelogram: When two angles α and β are found, use one of the angles α to filter the line segments so that the angle corresponding to the filtered line segments is equal to the angle. Let vector n be (sinα, cosα), count the distances from the filtered line segments to this vector n, find the two distances that have the most line segments classified in them, and infer that the difference between the two distances is the distance between the sides of the parallelogram with angle α. Approximate this distance as the length d1 of the side with angle β, and obtain the length d2 of the α side. Obtain the lengths of the two sides through two statistics;

[0086] Get the coordinates of the parallelogram: determine the intersection point between two intersecting straight lines, where one angle is α and the other angle is β. The length of the line segment perpendicular to the line with angle α is d1, and the length of the line segment perpendicular to the line with angle β is d2. Find the intersection point of d1 and d2, and determine the intersection point as the coordinate of the upper left corner of the parallelogram. The coordinate calculation formula is: And then get all the information of the parallelogram.

[0087] In some embodiments, the step of aggregating the parallelogram regions according to the parallelogram regions to determine the states of the faces of the target Rubik's Cube object includes the following steps:

[0088] Traverse the quadrilaterals and search for adjacent quadrilaterals around the current quadrilateral along the two directions of the parallelogram. After finding the adjacent quadrilaterals, check whether they are similar to the current quadrilateral.

[0089] After the check succeeds, a parallelogram group is created, and this parallelogram group will expand itself according to the added parallelogram so that it can contain the added parallelogram;

[0090] The parallelograms added to the group have group numbers. When they expand to the surrounding adjacent parallelograms or are included by other quadrilaterals, the groups will be automatically merged to form a larger group, determining the status of each face of the target Rubik's Cube object.

[0091] The following describes the specific implementation process of the embodiment of the present invention in detail by taking a specific application scenario as an example:

[0092] It can be seen from observation that in any state, the colors of the 9 small blocks on each face of the Rubik's Cube are not fixed and the shape formed by the small blocks of the same color can be 1x1, 2x2, 1x2, 1x3, or even not a rectangle, but an L-shape, a concave shape, etc. However, they all have one thing in common, that is, on the same face, the directions of the sides of these shapes are only two directions and the directions are the same. Therefore, the embodiment of the present invention obtains such a fact: the color blocks on a certain face of the Rubik's Cube are divided into many areas according to the color, and the straight lines where the edges of these areas are located must be in the same direction. Based on this fact, the embodiment of the present invention can use parallelograms to describe these color blocks. For non-rectangular color blocks, such as L-shaped, concave, etc., it is also feasible to use parallelograms to describe them. The method of describing color blocks with parallelograms can tell us the two most important information: the directions of the two sides and the location of the color blocks. When a large number of parallelograms with equal directions of the sides are found to be closely gathered in the picture, it can be inferred that the area where these parallelograms are located is a face of the Rubik's Cube.

[0093] The method of the present invention can be roughly divided into three steps in Rubik's Cube recognition:

[0094] 1. Image segmentation - segment the image by color to form a series of regions, as described in step 1 to step 3;

[0095] 2. Parallelogram determination - try to use a parallelogram to represent the area, as explained in steps 4 to 6.

[0096] 3. Quadrilateral aggregation - aggregate similar quadrilaterals, as described in step 7.

[0097] 1. Image Segmentation:

[0098] There are many existing methods for image segmentation, such as the watershed algorithm, the adaptive threshold algorithm, etc. These algorithms can effectively overcome the effects of lighting and camera interference, but they have essential defects in solving this problem: 1. Most of them can only process grayscale images, and the distinct color characteristics of the Rubik's Cube color blocks will be lost in grayscale images. 2. The amount of computation is large, which puts a lot of pressure on mobile devices that need to process tasks in real time. In summary, this method proposes an image segmentation algorithm in the RGB color space that can both utilize the distinct color properties of the Rubik's Cube color blocks and reduce the amount of computation.

[0099] The concept of "color difference" can be reflected in the color space CIE Luv. This algorithm uses the CIE Luv color space as a method to measure color differences. In the CIE Luv space, people's intuitive perception of the size of color differences can be measured by the Euclidean distance of colors in this space. Using this point, the embodiment of the present invention can divide the region so that the color difference in the region is within a certain range. However, since the RGB space to the CIE Luv space is nonlinear, the amount of calculation is large. In order to reduce the amount of calculation, this program is divided into 3 steps in image segmentation: Step 1: Reduce the number of colors in the picture. Step 2: Initially divide the region so that the colors in the region are the same. Step 3: Calculate the color difference of adjacent regions through CIE Luv and merge adjacent regions. The following is a detailed description of these 3 steps.

[0100] Step 1 Reduce the number of colors:

[0101] The purpose of color reduction is to reduce the amount of subsequent operations and prepare for the initial segmentation of the area. Figure 3 The original image of the Rubik's Cube shown in the figure has a large color difference between different color blocks of the Rubik's Cube. Therefore, the color reduction algorithm will reduce the color as aggressively as possible as long as it does not merge the different color blocks of the Rubik's Cube together. According to a lot of practical experience, the color reduction can be performed using formula 1 to reduce the color of the entire picture to 16 colors. The image after color reduction is as follows Figure 4 As shown, R, G, B are the original color values, ranging from 0 to 255, and R', G', B' are new values, also ranging from 0 to 255. Through this operation, the color value in each channel is one of {0, 90, 180}, so the embodiment of the present invention reduces the number of colors of the entire picture to 27 (such as Figure 5 shown).

[0102]

[0103] Step 2: Initial area segmentation

[0104] The initial segmentation method is to merge the areas with the same pixels after color subtraction. During the merging process, mark each pixel and record the number of pixels in the area and the RGB value of the pixel corresponding to the accumulated original image. This makes it convenient to obtain the average pixel of the area and update the rectangular range where the area is located (record the leftmost, rightmost, topmost and bottommost coordinates). The method of marking pixels is: each time a new area is encountered, an id is assigned to the area (generally speaking, the id is generated by a non-negative increasing counter) and the id is used to mark the pixel, so that we can find the corresponding area through the pixel.

[0105] Step 3: Merge adjacent areas

[0106] like Figure 6As shown, the sum of the color cumulative values ​​of the original image in the region and the number of pixels in step 2 can be used to obtain the average RGB value of the region. Each region is compared with the average RGB value of its neighboring region. When the difference between them in the CIE Luv color space is less than a certain value (here, 20 is taken based on experience), they are merged. At the same time, the pixel value accumulator and the number of pixels are merged, and the rectangular region is updated. In this way, the embodiment of the present invention finally obtains the segmented image.

[0107] from Figure 6 It can be seen that the color blocks of the Rubik's Cube are well divided. Figure 6 A represents the image after color reduction; B represents the first stage pixel merging diagram; C represents the second stage pixel merging diagram; D represents the third stage pixel merging diagram. The green box represents the rectangular boundary of the region, and the pink pixels represent the pixels belonging to the region ID. Note Figure 4 The segmentation defects caused by illumination in the initial segmentation are solved in step 3, such as Figure 6 As shown in B. Figure 4 Some pixels in the red block are subtracted to another color, but are merged together again in step 3.

[0108] 2. Parallelogram determination:

[0109] After image segmentation, the embodiment of the present invention obtains a series of regions, which are marked with different numbers. The next step is to try to fit each region with a parallelogram. The fitting of the parallelogram can be divided into three steps: step 4. Get the edge segments, use small segments to outline the edge of the region, and obtain the segment chain surrounding the region step 5. Use statistical methods to obtain the direction and length of the two sides of the parallelogram and the coordinates step 6. Eliminate the misjudged quadrilaterals. The following is a detailed description of these three steps.

[0110] Step 4, get the edge segment:

[0111] refer to Figure 7 ,in, Figure 7 A in the figure represents the edge acquisition diagram; B represents the line segment chain representing the edge; C represents the line segment angle statistics of B; and D represents the parallelogram acquisition. The edge is scanned around the circumscribed rectangle of the region obtained after image segmentation. The scanning method is: scan along one side of the circumscribed rectangle of the region perpendicular to the side every 10 pixels until a pixel in the region is encountered or the rectangle boundary is exceeded. A line segment can be determined between two pixels belonging to the region, and then the appropriateness of the line segment is verified. The verification method is: scan the region boundary pixels between the two points to determine whether these pixels are on the line segment. If they are, it means that the line segment is part of the region edge and is retained. Otherwise, it is discarded (such as Figure 7In this way, each side of the circumscribed rectangle is traversed. Through this method, the embodiment of the present invention can use a line segment chain to represent the boundary of the target area (such as Figure 7 B in the figure). To verify whether a pixel is on a straight line, we can use the vector method to calculate. Since the endpoints of the line segment are known, the angle of the line segment can be calculated as α, so the distance d from the point to the line segment can be expressed by formula 2, where the point to be verified is point p(x p ,y p ), point b(x b ,y b ), is any point at both ends. When d is less than or equal to 1, the embodiment of the present invention can determine that the point is on the straight line.

[0112] d=(x p -x b )sinα-(y p -y b )cosα (2)

[0113] Step 5. Get the angle, length, and coordinates of the parallelogram:

[0114] If the area is composed of Rubik's Cube color blocks, then it is obvious that there are only two directions for the side length of the area composed of the same color blocks on each face to correspond to the direction of the line segment. Then in the line segment set obtained in the previous step, if classified by angle, there are two angles, so that the line segments classified in them are the most. Based on this conclusion, this step converts all angles into 0 to π, and then divides them into 100 parts, and counts all the line segments by interval. The angles with the first and second largest interval peaks are the angles corresponding to the two sides of the parallelogram. The first and second peaks can be found using the longest non-decreasing sub-sequence, but this method is easily affected by the large value of a local point, so that the second peak cannot be reached. The solution of this method is: first sort these angles in descending order, the angle corresponding to the first value must be the angle of one side of the parallelogram, and then limit the minimum angle of the two angles of the parallelogram. When the angle corresponding to the next value is greater than the minimum angle with the first angle, it indicates that the second angle has been found. In this way, the embodiment of the present invention obtains the angles of the two sides of the parallelogram, such as Figure 7 C, where white represents the peak value determined by the program.

[0115] Next, obtain the lengths of the two sides of the parallelogram. When two angles α and β are found, the embodiment of the present invention uses one of the angles α to filter the line segments so that the angle corresponding to the filtered line segments is equal to the angle. Let vector n be (sinα, cosα), and count the distances from the filtered line segments to this vector n. It will be found that there are two distances that make the most line segments classified in them. Then it can be inferred that the difference between these two distances is the distance between the sides of the parallelogram with angle α, and this distance is approximated as the length d1 of the side with angle β. The same operation can be used to obtain the length d2 of the α side. Therefore, this step obtains the lengths of the two sides through two statistics. The distance formula here is formula (2), except that x b =0,y b =0; p(x p ,y p ) is any point on the filtered line segment.

[0116] Next, we need to obtain the coordinates of the upper left corner of the parallelogram. This is an easy-to-solve math problem: given two intersecting straight lines, the intersection point is (0,0), one of which has an angle α and the other has an angle β. The length of the line segment perpendicular to the line with angle α is d1, and the length of the line segment perpendicular to the line with angle β is d2. Find the intersection of d1 and d2, which is the coordinates of the upper left corner of the parallelogram. By derivation, we can use formula (3) to obtain the coordinates t(x t ,y t ).

[0117]

[0118] In this way, we get all the information about the parallelogram, such as Figure 7 As shown in D.

[0119] Step 6. Eliminate the wrong parallelogram:

[0120] Because the present invention attempts to fit all regions with parallelograms, there will inevitably be many interfering quadrilaterals. After observing formula (3), it can be found that this formula has excellent properties. When the non-parallelogram region is determined, due to the unreliability of the angle length data, the calculated coordinates of the upper left corner of the quadrilateral will not match the location of the region. Therefore, the embodiment of the present invention can check whether the center coordinates of the parallelogram fall within the circumscribed rectangle of the region. If not, the parallelogram can be excluded. This method can filter out most of the misidentified quadrilaterals. Comparison of areas can also be used. When segmenting the image, the embodiment of the present invention has already recorded the number of pixels contained in the region. The actual area of ​​the region (the number of pixels) can be compared with the area obtained by the parallelogram area formula to filter out parallelograms with too large a ratio difference.

[0121] This method of using statistics to derive a parallelogram is very robust. Even in the presence of interference or partial image loss, it is still possible to infer a parallelogram. The amount of computation is very small, sufficient to meet the requirements of real-time analysis and processing on embedded and low-computing-power platforms.

[0122] Step 7, parallelogram aggregation:

[0123] refer to Figure 8 , Figure 8 (a) in the figure represents all parallelograms detected. Figure 8 (b) in the figure represents the face obtained by aggregation. So far, the embodiment of the present invention has found any possible parallelogram on the image. Now it is necessary to infer the faces of the Rubik's Cube through them. Here, the embodiment of the present invention uses a search method. The method is as follows: traverse the quadrilaterals, and search for adjacent quadrilaterals around the current quadrilateral along the two directions of the parallelogram. If found, check whether it is a similar quadrilateral to itself (the angle errors of the two sides are within the allowable range). After the check is successful, create a group of parallelograms. This parallelogram will expand itself according to the added parallelograms so that it can contain the added parallelograms. The parallelogram added to the group has a group number. When it expands to the surrounding adjacent parallelograms or is included by other quadrilaterals, the group will be automatically merged to form a larger group. The advantage of this is that the correct face can be found when the Rubik's Cube is blocked or some color blocks are not recognized.

[0124] In summary, the present invention has the following advantages:

[0125] 1. The present invention is innovative in color segmentation, using color reduction, merging, and merging after CIE Luv judgment to segment out areas with consistent colors that conform to human intuition.

[0126] 2. The present invention uses statistical methods in the parallelogram fitting process, so that its recognition ability and anti-interference ability are stronger, and the non-parallelogram shape area is effectively excluded.

[0127] Another aspect of the embodiment of the present invention further provides a camera-based real-time recognition system for a Rubik's Cube state, comprising:

[0128] The first module is used to perform image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number;

[0129] The second module is used to perform region fitting on each independently numbered object region through a parallelogram to determine all parallelogram regions existing on the image;

[0130] The third module is used to aggregate the parallelogram regions according to the parallelogram regions to determine the status of each face of the target Rubik's Cube object.

[0131] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0132] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned camera-based real-time recognition method for the state of a Rubik's Cube. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0133] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0135] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0136] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the real-time recognition method of the magic cube state based on the camera of the embodiment of the present invention;

[0137] Input / output interface 903, used to implement information input and output;

[0138] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0139] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0140] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0141] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned camera-based real-time recognition method for the Rubik's Cube state is implemented.

[0142] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0145] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0146] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0147] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0149] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0150] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0151] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0152] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0155] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A method for real-time recognition of Rubik's Cube status based on a camera, characterized in that: The following steps are involved: Performing image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number; Performing region fitting on each independently numbered object region through a parallelogram to determine all parallelogram regions existing on the image; According to each parallelogram area, each parallelogram area is aggregated to determine the state of each face of the target Rubik's Cube object.

2. The method for real-time recognition of the state of a Rubik's Cube based on a camera according to claim 1, characterized in that: The image segmentation process is performed on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number, including the following steps: Reducing the number of colors in the camera image of the target Rubik's Cube object; Merging the regions with the same pixels after color reduction to complete the initial segmentation of the camera image of the target Rubik's Cube object and obtain the initial segmentation region; According to the sum of the color cumulative values ​​of the original image and the number of pixels in the initially segmented area, the average RGB value of the area is obtained, each area is compared with the average RGB value of its neighboring area, and when the difference between them in the CIE Luv color space is less than a preset value, they are merged, and the pixel value accumulator and the number of pixels are merged at the same time, and the rectangular area is updated, and finally the segmented object area is obtained, and each object area is configured with a corresponding independent number.

3. The method for real-time recognition of the state of a Rubik's Cube based on a camera according to claim 2, characterized in that: The method of reducing the number of colors in the camera image of the target Rubik's Cube object is specifically: The color reduction process is performed by the following formula: Reduce the color of the entire camera image to 16 colors, where R, G, and B are the values ​​of the original colors, ranging from 0 to 255. ′ , G ′ , B ′ is a new value ranging from 0 to 255, and the color value of each channel is determined to be one of {0, 90, 180}, reducing the number of colors of the entire picture to 27; The regions with the same pixels after color reduction are merged to complete the initial segmentation of the camera image of the target Rubik's Cube object to obtain the initial segmentation region, which is specifically: Merge the areas with the same pixels after color subtraction. During the merging process, mark each pixel and record the number of pixels in the area and the RGB value of the pixel corresponding to the accumulated original image, obtain the average pixel of the area, and update the rectangular range where the area is located. The method of marking pixels is: each time a new area is encountered, an id is assigned to the area, and the id is used to mark the pixel so that the corresponding area can be found through the pixel.

4. The method for real-time recognition of the state of a magic cube based on a camera according to claim 1, characterized in that: The method of performing region fitting on each independently numbered object region through a parallelogram to determine all parallelogram regions existing on the image comprises the following steps: Scan the edge of the circumscribed rectangle around the area obtained after image segmentation to obtain edge segments; According to the obtained edge segments, the angle, length and coordinates of the parallelogram are obtained; Eliminate erroneous parallelograms by: checking whether the center coordinates of the parallelogram fall within the circumscribed rectangle of the region; if not, exclude the parallelogram; compare the actual area of ​​the region with the area obtained by the parallelogram area formula, and filter out parallelograms with too large a difference in ratio.

5. The method for real-time recognition of the state of a magic cube based on a camera according to claim 4, characterized in that: The step of scanning the edge of the circumscribed rectangle of the region obtained after image segmentation to obtain edge segments comprises the following steps: Scan the edge of the circumscribed rectangle of the region obtained after image segmentation; the scanning method is: scan perpendicularly to one side of the circumscribed rectangle of the region every 10 pixels until a pixel in the region is encountered or beyond the rectangular boundary, and a line segment can be determined between two pixels belonging to the region; Verify the suitability of this line segment. The verification method is: scan the region boundary pixels between the two points and determine whether these pixels are on the line segment. If they are, it means that the line segment is part of the region edge and is retained. Otherwise, it is discarded. Traverse each edge of the circumscribed rectangle and use a line segment chain to represent the boundary of the target area; The vector method is used to verify whether the pixel is on the corresponding straight line. Specifically, the angle of the line segment is calculated based on the known endpoints of the line segment, and the angle of the line segment is calculated by the formula d = (x p -x b )sinα-(y p -y b )cosα to calculate the distance from the endpoint to the line segment, where the point to be verified is point p(x p ,y p ), point b(x b ,y b ) is any point at either end. When the distance d from the endpoint to the line segment is less than or equal to 1, the point is judged to be on the straight line.

6. The method for real-time recognition of the state of a magic cube based on a camera according to claim 4, characterized in that: The method of obtaining the angle, length and coordinates of the parallelogram according to the acquired edge segments comprises the following steps: Get the angle of the parallelogram: convert all angles to 0 to π, divide them into 100 parts, count all line segments by interval, and determine the first and second largest angles of the interval peak as the angles corresponding to the two sides of the parallelogram; the following method is used to find the first and second largest peaks: first sort the angles in descending order, the angle corresponding to the first value is the angle of one side of the parallelogram, then limit the minimum angle of the two angles of the parallelogram, and when the angle corresponding to the next value is greater than the minimum angle of the first angle, it indicates that the second angle has been found, and then the angles of the two sides of the parallelogram are obtained; Get the length of the parallelogram: When two angles α and β are found, use one of the angles α to filter the line segments so that the angle corresponding to the filtered line segments is equal to the angle. Let vector n be (sinα, cosα), count the distances from the filtered line segments to this vector n, find the two distances that have the most line segments classified in them, and infer that the difference between the two distances is the distance between the sides of the parallelogram with angle α. Approximate this distance as the length d1 of the side with angle β, and obtain the length d2 of the α side. Obtain the lengths of the two sides through two statistics; Get the coordinates of the parallelogram: determine the intersection point between two intersecting straight lines, where one angle is α and the other angle is β. The length of the line segment perpendicular to the line with angle α is d1, and the length of the line segment perpendicular to the line with angle β is d2. Find the intersection point of d1 and d2, and determine the intersection point as the coordinate of the upper left corner of the parallelogram. The coordinate calculation formula is: And then get all the information of the parallelogram.

7. The method for real-time recognition of the state of a Rubik's Cube based on a camera according to claim 1, characterized in that: The method of aggregating the parallelogram regions according to the parallelogram regions to determine the states of the faces of the target Rubik's Cube object includes the following steps: Traverse the quadrilaterals and search for adjacent quadrilaterals around the current quadrilateral along the two directions of the parallelogram. After finding the adjacent quadrilaterals, check whether they are similar to the current quadrilateral. After the check succeeds, a parallelogram group is created, and this parallelogram group will expand itself according to the added parallelogram so that it can contain the added parallelogram; The parallelograms added to the group have group numbers. When they expand to the surrounding adjacent parallelograms or are included by other quadrilaterals, the groups will be automatically merged to form a larger group, determining the status of each face of the target Rubik's Cube object.

8. A camera-based real-time recognition system for the Rubik's Cube, characterized in that: include: The first module is used to perform image segmentation processing on the camera image of the target Rubik's Cube object to obtain a series of segmented object regions, each of which is assigned a corresponding independent number; The second module is used to perform region fitting on each independently numbered object region through a parallelogram to determine all parallelogram regions existing on the image; The third module is used to aggregate the parallelogram regions according to the parallelogram regions to determine the status of each face of the target Rubik's Cube object.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images