Rubik's cube recovery method and system based on deep reinforcement learning and electronic equipment

Abstract

The embodiment of the invention provides a Rubik's cube restoration method and system based on deep reinforcement learning, the state of a Rubik's cube is coded through a 6 * N * N tensor matrix, the color of color blocks and coordinate information are completely reserved, high-precision input is provided for subsequent model training, the Rubik's cube restoration method and system are adaptive to any order of Rubik's cubes (such as second order and third order), and the scheme universality is enhanced; a Q-learning optimization recovery strategy network is utilized, network parameters are updated by minimizing a mean square error between a predicted Q value and a real Q value, and the state-action pair value of the Rubik's cube is accurately learned; and based on the recovery strategy network, strategy gradient optimization, advantage function, a gradient rising algorithm and entropy regularization are adopted, strategy network parameters are iteratively updated, exploration and utilization are balanced until the recovery path expects to accumulate the stable rewards, the technical effect is remarkable, the value judgment accuracy, the strategy optimization efficiency and the model generalization ability can be improved, and the method is suitable for popularization and application. And a high-quality recovery path is stably generated, and recovery of the efficient and intelligent magic cube is assisted.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the technical field of artificial intelligence and intelligent hardware cross, and particularly relates to a Rubik's cube restoration method and system based on deep reinforcement learning and electronic equipment. BACKGROUND

[0002] The Rubik's cube, the Chinese checkers and the independent diamond chess are called the three great mysteries in the intelligence game field by foreign intelligence experts, and the Rubik's cube is a classic spatial logic toy, and its restoration technology has been a research hotspot in the field of artificial intelligence algorithm and mechanical control for a long time. With the development of intelligent hardware and deep learning technology, the automatic restoration system has gradually evolved from the traditional mechanical solution based on the preset formula (such as the layer first method and the angle first method) to intelligence and self-adaptation.

[0003] In the prior art, the Rubik's cube restoration scheme is mainly divided into two categories:

[0004] One is a static recognition scheme based on computer vision, the surface image of the Rubik's cube is collected through a camera, the current state is analyzed, and then the optimal solution algorithm (such as the Kociemba algorithm) is called to generate the restoration steps;

[0005] The other is an automatic restoration device based on a mechanical arm, which executes the restoration action through a preset motion trajectory.

[0006] However, the prior art has the following significant defects:

[0007] 1. Poor dynamic adaptability: unable to perceive the physical state changes of the Rubik's cube in real time, such as axis loosening, color block wear, position deviation caused by external collision due to long-term use of the Rubik's cube, and the restoration path calculated only based on the initial state is prone to step failure due to dynamic errors, and the restoration failure rate is as high as 30% or more;

[0008] 2. Poor human-machine collaboration: the restoration instruction sequence generated by the existing algorithm is completely based on the mathematical optimal solution, and the step-by-step logic of human learning (such as the progressive strategy of first restoring the bottom cross and then processing the corner block) is not considered, which makes it difficult for ordinary users to understand and imitate, and the teaching value is limited;

[0009] 3. Weak cross-order adaptability: different orders of the Rubik's cube (such as 3x3 and 4x4) need to be designed separately for state coding and solution logic, and the algorithm has low universality;

[0010] 4. Redundant operation is not optimized: in a complex state, the algorithm may generate continuous reverse operations (such as R followed by R') or repeated invalid actions, increasing the number of restoration steps and reducing efficiency. SUMMARY

[0011] The embodiment of the present application provides a Rubik's cube restoration method and system based on deep reinforcement learning, to solve the problem that the map construction in the prior art depends on the plane assumption, which limits the mapping performance and cannot completely get rid of the dependence on the plane structured environment.

[0012] In a first aspect, the embodiment of the present application provides a Rubik's cube restoration method based on deep reinforcement learning, comprising:

[0013] S1, constructing a state coding model based on the order of the Rubik's cube, converting the state of the Rubik's cube into a 6*N*N tensor matrix; wherein N represents the order of the Rubik's cube, and each element in the tensor matrix corresponds to the color and coordinates of a color block on the Rubik's cube;

[0014] S2, obtaining Rubik's cube restoration sample data, the Rubik's cube restoration sample data comprising an initial state tensor of the Rubik's cube, a target state tensor, and a restoration operation sequence of the Rubik's cube from the initial state tensor to the target state tensor;

[0015] S3, performing neural network training based on the restoration sample data to obtain a restoration strategy network for outputting the restoration operation sequence according to the initial state tensor and the target state tensor, and performing Rubik's cube restoration based on the restoration strategy network.

[0016] Preferably, the step S1 specifically comprises:

[0017] S11, assigning a unique identifier color(0,i) to each color block of the N-order Rubik's cube, wherein i is the color block index, i∈[0,N 2 ×6-1];

[0018] S12, defining a standard position coordinate Location(0,i) for each color block in the completely restored state of the Rubik's cube;

[0019] S13, expanding the six faces of the Rubik's cube into six independent two-dimensional planes, each two-dimensional plane comprising N*N color blocks, and mapping the expanded six planes into a 6*N*N tensor matrix, wherein the unique identifier of each color block is associated with the standard position coordinate.

[0020] Preferably, the step S2 specifically comprises:

[0021] S21, determining a rotation instruction set of the N-order Rubik's cube, each rotation instruction in the rotation instruction set comprising a rotation object, a rotation direction and a rotation angle; and determining a tensor transformation matrix of each rotation instruction based on the rotation object, the rotation direction and the rotation angle.

[0022] S22, collecting Rubik's Cube restoration samples, each restoration sample including an initial state, a target state, and a restoration operation sequence, wherein the restoration operation sequence includes a series of rotation instructions arranged in chronological order;

[0023] Converting the initial state into an initial state tensor and converting the target state into a target state tensor according to the state encoding model in S1;

[0024] Convert each rotation instruction in the restoration operation sequence into a tensor transformation matrix in a time sequence;

[0025] S23. Generate a training sample based on the restored sample.

[0026] Preferably, in step S3, performing neural network training based on the restored sample data specifically includes:

[0027] Based on the initial state and target state in the restored sample, a probability distribution of all possible rotation instructions is generated through the restoration strategy network, and the next rotation instruction is selected based on the probability distribution;

[0028] Determining an updated state of the Rubik's Cube based on the selected next rotation instruction, and determining a reward coefficient corresponding to the next rotation instruction based on a preset reward calculation rule; wherein the preset reward rule includes one or more of the following: the difference between the updated state and the target state, the number of operation steps between two successful color block placements, the degree of match between the operation sequence generated by the restoration strategy network and the standard operation sequence, and a penalty for redundant operations;

[0029] Repeat the above steps until the Rubik's Cube is restored to the target state. Record the rotation instructions, state changes, and corresponding rewards from the initial state to the target state to form a complete restoration path trajectory.

[0030] Preferably, in step S2, after generating a training sample based on the restored sample, the step further includes:

[0031] Update the network parameters of the restoration policy network based on Q-learning, minimize the mean square error between the predicted Q value and the true Q value of the current restoration policy network, and update the network parameters of the restoration policy network based on maximizing the expected cumulative reward; determine the gradient loss function of the restoration policy network, backpropagate the policy gradient loss function, and update the network parameters of the restoration policy network.

[0032] As a preferred method, the network parameters of the restoration strategy network are updated based on Q-learning to minimize the mean square error between the predicted Q value and the true Q value of the current restoration strategy network, specifically including:

[0033] Determine the mean square error between the predicted Q value and the true Q value of the current restoration strategy network based on Q-learning:

[0034]

[0035] In the above formula, Q(s t ,a t ; θ) is the restoration strategy network with network parameter θ, which is used to restore the cube state s at time t t Execute rotation instruction a t The predicted Q value, θ is the network parameter of the current restoration strategy network; s t is the state of the Rubik's Cube at time t; a t is the rotation instruction executed at time t; R t After executing the action, from state s t Transfer to state s t+1 The immediate reward obtained; γ is the discount factor, ranging from 0 to 1; a' is the state s t+1 Optional subsequent rotation instructions; Q(s t+1 ,a';θ - ) is the network parameter θ - The restoration strategy network, at time t+1, is used to restore the Rubik's Cube state s t+1 The predicted Q value of executing the rotation instruction a'; maxa' is the Q value of the state s t+1 Among the optional subsequent rotation instructions, the subsequent rotation instruction that maximizes the predicted Q value; T is the upper limit of the number of time steps of a restoration operation sequence;

[0036] The network parameters of the restoration strategy network are adjusted through the back propagation algorithm so that the mean square error converges.

[0037] Preferably, updating the network parameters of the recovery strategy network based on maximizing the expected cumulative reward specifically includes:

[0038] Determine the average value of the cumulative rewards obtained by all possible Rubik's Cube restoration path trajectories under the network parameters θ of the restoration strategy network:

[0039]

[0040] In the above formula, J(θ) is the expected cumulative reward; τ is the restoration path trajectory, which is determined by the restoration strategy network π θ generate; is the expectation operator; R t To perform action a t The immediate reward obtained after ; T is the upper limit of the time steps of a recovery operation sequence;

[0041] For J(θ -) gradient, and the gradient is updated along the gradient direction using a gradient ascent algorithm to update the network parameters θ:

[0042]

[0043] In the above formula, A t is the advantage function, indicating the rotation instruction a t the value advantage relative to the average level;

[0044] The above steps are iteratively performed until the expected cumulative reward of the restoration path trajectory generated by the restoration strategy network is improved by less than a set threshold.

[0045] In a second aspect, an embodiment of the present application provides a Rubik's cube restoration system based on deep reinforcement learning, comprising:

[0046] An encoding module constructs a state encoding model based on the order of the Rubik's cube, and converts the state of the Rubik's cube into a 6×N×N tensor matrix; wherein N represents the order of the Rubik's cube, and each element in the tensor matrix corresponds to the color and coordinates of a color block on the Rubik's cube;

[0047] A sample collection module acquires restoration sample data of the Rubik's cube, wherein the restoration sample data includes an initial state tensor of the Rubik's cube, a target state tensor, and a restoration operation sequence of the Rubik's cube from the initial state tensor to the target state tensor;

[0048] A restoration strategy network module performs neural network training based on the restoration sample data, obtains a restoration strategy network for outputting the restoration operation sequence according to the initial state tensor and the target state tensor, and performs Rubik's cube restoration based on the restoration strategy network.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the Rubik's cube restoration method based on deep reinforcement learning as described in the first aspect of the present application.

[0050] In a fourth aspect, an embodiment of the present application provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the Rubik's cube restoration method based on deep reinforcement learning as described in the first aspect of the present application.

[0051] The embodiment of the application provides a Rubik's cube restoration method and system based on deep reinforcement learning, an electronic device and a storage medium, which encodes the Rubik's cube state through a 6*N*N tensor matrix, completely retains the color block color and coordinate information, provides high-precision input for subsequent model training, is suitable for any order Rubik's cube (such as a second order, a third order and the like), and enhances the generality of the scheme; the Q-learning is used to optimize the restoration strategy network, network parameters are updated by minimizing the mean square error of the predicted Q value and the real Q value, and the Rubik's cube state-action pair value is accurately learned; based on the restoration strategy network, the strategy gradient optimization is adopted, the advantage function, the gradient ascent algorithm and the entropy regularization are used to iteratively update the strategy network parameters, exploration and utilization are balanced, until the restoration path expected cumulative reward is stable, and the technical effect is remarkable, the value judgment accuracy, the strategy optimization efficiency and the model generalization capability can be improved, a high-quality restoration path is stably generated, and efficient and intelligent Rubik's cube restoration is assisted. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, below the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 It is a flow chart of the Rubik's cube restoration method based on deep reinforcement learning according to the embodiment of the present application.

[0054] Figure 2 It is a specific flow chart of step S1 according to the embodiment of the present application.

[0055] Figure 3 It is a specific flow chart of step S2 according to the embodiment of the present application.

[0056] Figure 4 It is a block diagram of the Rubik's cube restoration system based on deep reinforcement learning according to the embodiment of the present application.

[0057] Figure 5 It is a schematic diagram of the entity structure according to the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0059] The term “and / or” in the embodiments of the present application merely describes an association relationship of associated objects, and indicates that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone.

[0060] The terms “first” and “second” in the embodiments of the present application are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first” and “second” can explicitly or implicitly include at least one of the features. In the description of the present application, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion. For example, a system, product or device including a series of components or units is not limited to the listed components or units, but can optionally include components or units not listed or can optionally include other components or units inherent to these products or devices. In the description of the present application, the meaning of “a plurality of” is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0061] In this document, referring to “embodiments” means that the specific features, structures or properties described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0062] Rubik's cube recovery needs accurate planning of rotation steps. The traditional method relies on manual derivation of formulas, which has obvious limitations: first, the adaptability is poor for different orders and disordered states, and new scenarios need to redesign strategies; second, the recovery path is mostly based on experience, which is difficult to guarantee efficiency (step redundancy and disconnection from human habits). Existing intelligent recovery systems often have low recovery success rate and operation logic that does not meet actual needs due to rough state coding and insufficient combination of reward feedback in strategy learning, and cannot stably output a few-step recovery scheme with high adaptability. It is urgent to build a technical system based on precise state representation and deep reinforcement learning to solve the dynamic adaptation and strategy optimization problem of Rubik's cube recovery.

[0063] To solve the above problems, the application provides a Rubik's cube restoration method and system based on deep reinforcement learning, which encodes the Rubik's cube state through a 6*N*N tensor matrix, completely retains the color and coordinate information of the color blocks, provides high-precision input for subsequent model training, adapts to any order Rubik's cube (such as second order, third order, etc.), and enhances the generality of the scheme; the Q-learning is used to optimize the restoration strategy network, the network parameters are updated by minimizing the mean square error of the predicted Q value and the real Q value, the value of the Rubik's cube state-action pair is accurately learned, and then based on the restoration strategy network, the strategy gradient optimization is adopted, the advantage function, the gradient ascent algorithm and the entropy regularization are used to iteratively update the strategy network parameters, the exploration and utilization are balanced, until the expected cumulative reward of the restoration path is stable, and the technical effect is remarkable, which can improve the value judgment accuracy, the strategy optimization efficiency and the model generalization ability, stably generate a high-quality restoration path, and help efficient and intelligent Rubik's cube restoration. The method of the application will be described below through one or more embodiments.

[0064] The embodiment of the application provides a Rubik's cube restoration method based on deep reinforcement learning, as shown in the formula (1), which comprises the following steps. Figure 1

[0065] S1, a state encoding model is constructed based on the order of the Rubik's cube, and the state of the Rubik's cube is converted into a 6*N*N tensor matrix; wherein N represents the order of the Rubik's cube, and each element in the tensor matrix corresponds to the color and coordinates of a color block on the Rubik's cube;

[0066] Specifically, the purpose of step S1 is to accurately describe the physical state of the Rubik's cube in a standardized mathematical form, and to provide a unified input format for subsequent model training; wherein the "order of the Rubik's cube" refers to the dimension of the Rubik's cube (such as a 3*3 three-order Rubik's cube, a 4*4 four-order Rubik's cube, etc.), which determines the number of color blocks and the structural complexity of the Rubik's cube (a three-order Rubik's cube has 6 faces, and each face has 9 color blocks; a four-order Rubik's cube has 16 color blocks on each face). Based on this, the state of the Rubik's cube is converted into a 6*N*N tensor matrix in this embodiment, "6" corresponds to the 6 faces of the Rubik's cube (top face, bottom face, front face, back face, left face and right face); "N*N" represents the arrangement of the color blocks on each face (N is the order), such as a three-order Rubik's cube N=3, and the tensor sub-matrix of each face is 3*3, corresponding to 9 color blocks; the tensor matrix is in the form of a three-dimensional array, which can completely retain the spatial structure information of each face, the "color" of the element refers to the physical color of the color block (such as white, yellow, etc.), which is quantified through RGB value or preset coding (such as 1 representing white and 2 representing yellow); the "coordinates" of the element refer to the position of the color block on the corresponding face (such as "top face, 2nd row, 3rd column"), so as to realize the marking of all color blocks one by one, clearly define the position of each color block under the complete restoration state, and ensure that the current position of each color block can be accurately compared with the target position.

[0067] ​S2, acquiring a restored sample data of the Rubik's Cube, the restored sample data comprising an initial state tensor of the Rubik's Cube, a target state tensor, and a restored operation sequence of the Rubik's Cube from the initial state tensor to the target state tensor;

[0068] Step S2 is to collect the "input-output" examples for model learning, so that the model learns the rules by imitating the known restoration process. The initial state tensor refers to the state of the Rubik's Cube when it starts to be restored (such as the state after being scrambled), which is represented by a 6xNxN tensor matrix and contains the current color and position of all color blocks. The target state tensor refers to the standard state of the Rubik's Cube after being completely restored, in which the color and coordinates of each color block are in the preset correct position (such as all yellow on the top surface and all white on the bottom surface). The restored operation sequence refers to each step of rotation instruction (such as U, R, L', etc.) from the initial state tensor to the target state tensor, which is the core content that the model needs to learn and output (such as the sequence [U, R, F'] representing "top surface clockwise rotation 90 degrees → right surface clockwise rotation 90 degrees → front surface counterclockwise rotation 90 degrees").

[0069] Further, in order to collect the operation sequence and the state change of each face of the Rubik's Cube during the restoration, the real-time images of the six faces of the Rubik's Cube can be captured by a mobile phone camera, a built-in camera of a smart Rubik's Cube, or an external image sensor. For the Rubik's Cube in a scrambled state, the images of the six faces are captured as the original visual data of the "initial state". For the Rubik's Cube in a completely restored state (each face has uniform color and the color blocks are in place), the images of the six faces are captured as the original visual data of the "target state". The state during the restoration operation can also be extracted through image acquisition. The captured images are processed to eliminate interference such as reflection and shadow through deblurring and light balance. The distortion of the Rubik's Cube plane is corrected through perspective transformation to ensure that the color blocks of each face are arranged in a regular grid. An image segmentation algorithm (such as color threshold-based segmentation and edge detection) is used to segment each face image into N x N independent color block regions. For each segmented color block region, the dominant color tone is extracted (such as matching the preset color library: white, yellow, red, etc. through RGB value), and the color information is converted into digital coding (such as 1 representing white and 2 representing yellow). According to the position of the color block in the image (such as "top surface, 2nd row, 3rd column"), the corresponding element coordinates in the 6xNxN tensor matrix are determined, and the color coding of each face is filled into the N x N sub-matrix according to the coordinates. The sub-matrices of the six faces are combined into the initial state tensor (scrambled state) or the target state tensor (restored state) of 6xNxN.

[0070] The color and position of the real Rubik's Cube are directly collected by the camera, solving the problem that the prior art cannot perceive the physical state of the Rubik's Cube in real time (such as mechanical error), and making the sample data closer to the actual application scenario.

[0071] S3, training a neural network based on the restored sample data to obtain a restoration strategy network for outputting a restoration operation sequence according to an initial state tensor and a target state tensor, and performing a Rubik's Cube restoration based on the restoration strategy network.

[0072] Specifically, the initial state tensor, the target state tensor, and the restoration operation sequence in the sample data are used to enable the model to simulate the restoration process (e.g., starting from the initial state, trying the operation sequence in the sample, and observing whether the target state can be reached). In this embodiment, a Monte Carlo Tree Search (MCTS) and Q-learning hybrid algorithm is used.

[0073] Based on the above embodiment, as a preferred implementation, as shown in Figure 2 the step S1 specifically includes:

[0074] S11, assigning a unique identifier color(0,i) to each color block of the N-order Rubik's Cube, where i is a color block index, i∈[0,N 2 ×6-1]; where the unique identifier is a unique code for each color block, used to distinguish different color blocks (even if the colors are the same, the identifier is different due to the different location attributes); the naming rule can include the reference attributes of the color block (such as the color and initial position in the completely restored state), and needs to ensure that the identifier has both uniqueness and semanticity; the index i covers all color blocks (from 0 to “total number of color blocks-1”), for example, for a three-order Rubik's Cube N=3, the total number of color blocks is 3 2 ×6=54, so i∈[0,53], ensuring that each color block is uniquely indexed.

[0075] S12, defining a standard position coordinate Location(0,i) for each color block in the completely restored state of the Rubik's Cube; the standard position coordinate is the spatial coordinate of the color block in the completely restored state, which needs to be bound to the physical structure of the Rubik's Cube and includes “face index+in-face coordinate”: face index: 0-5 are used to correspond to the 6 faces of the Rubik's Cube (e.g., 0=top face, 1=bottom face, 2=front face, 3=back face, 4=left face, and 5=right face); in-face coordinate: N×N grid for each face, represented by (row index r, column index c) to indicate the position of the color block in the face (r, c∈[0,N-1]), for example, the top-left color block coordinate of the three-order Rubik's Cube top face (face index 0) is Location(0,i)=(0,0,0) (face index 0, row 0, column 0).

[0076] After being bound to the color(0,i) of step S11, a “color block-standard position” mapping relationship is formed, so that the model can determine whether the color block is returned to the original position by comparing the “current position” with the “Location(0,i)”.

[0077] S13, unfolding the 6 faces of the Rubik's Cube into 6 independent two-dimensional planes, each two-dimensional plane including N*N color blocks, and mapping the 6 unfolded planes into a 6*N*N tensor matrix, wherein a unique identifier of each color block is associated with a standard position coordinate in the tensor matrix.

[0078] The step of the embodiment unfolds the 6 faces of the Rubik's Cube into 6 two-dimensional plane graphs through flattening the spatial structure of the Rubik's Cube, and the unfolding rule follows the conventional unfolding logic of the Rubik's Cube (e.g., the top face is on top, the bottom face is on bottom, the front face is facing forward, etc.), so as to ensure that the mapping relationship between the two-dimensional plane and the three-dimensional physical structure is fixed, state ambiguity is avoided, and the purpose is to convert the physical state of the three-dimensional Rubik's Cube into a digital tensor processable by a computer, so as to realize the structured representation and efficient calculation of the state.

[0079] Specifically, the tensor matrix is a three-dimensional array, and the dimensions "6*N*N" correspond to 6 faces (1st dimension), rows of each face (2nd dimension), and columns of each face (3rd dimension), respectively; each element in the matrix stores a unique identifier color(0, i) of a current color block at a corresponding position (face index, row, column), for example, the value of the tensor matrix element (0, 0, 0) is color(0, 5), which indicates that the color block with index 5 is currently placed at the 0th row and 0th column of the top face (face 0).

[0080] The "position index" in the tensor matrix is indirectly associated, that is, the element with coordinates (k, r, c) in the tensor matrix (corresponding to the kth face, row r, and column c) stores color(0, i) corresponding to the standard position coordinate Location(0, i), which should theoretically be (k, r, c) in the completely restored state; when the Rubik's Cube is in a non-restored state, the dislocation degree of the color block can be directly calculated by comparing the coordinates (k, r, c) of the tensor matrix element and the Location(0, i) corresponding to the color(0, i) stored by the element, thereby providing a key input for model training.

[0081] On the basis of the above embodiment, as a preferred implementation manner, as shown in Figure 3 the step S2 specifically includes:

[0082] S21, determining a set of rotation instructions of an N-order Rubik's Cube, each rotation instruction in the set of rotation instructions including a rotation object, a rotation direction, and a rotation angle; and determining a tensor transformation matrix of each rotation instruction based on the rotation object, the rotation direction, and the rotation angle.

[0083] In the embodiment, the conventional rotation instructions of the Rubik's Cube need to be defined first, including R, R', U, U', F, F', L, L', B, B', D, and D', wherein:

[0084] R: right side clockwise rotation 90 degrees (clockwise rotation from the perspective of facing the right side).

[0085] R': right side counterclockwise rotation 90 degrees (counterclockwise rotation from the perspective of facing the right side).

[0086] U: top side clockwise rotation 90 degrees (clockwise rotation from the perspective of facing the top side).

[0087] U': top side counterclockwise rotation 90 degrees (counterclockwise rotation from the perspective of facing the top side).

[0088] F: front side clockwise rotation 90 degrees (clockwise rotation from the perspective of facing the front side).

[0089] F': front side counterclockwise rotation 90 degrees (counterclockwise rotation from the perspective of facing the front side).

[0090] L: left side clockwise rotation 90 degrees (clockwise rotation from the perspective of facing the left side).

[0091] L': left side counterclockwise rotation 90 degrees (counterclockwise rotation from the perspective of facing the left side).

[0092] B: back side clockwise rotation 90 degrees (clockwise rotation from the perspective of facing the back side).

[0093] B': back side counterclockwise rotation 90 degrees (counterclockwise rotation from the perspective of facing the back side).

[0094] D: bottom side clockwise rotation 90 degrees (clockwise rotation from the perspective of facing the bottom side).

[0095] D': bottom side counterclockwise rotation 90 degrees (counterclockwise rotation from the perspective of facing the bottom side).

[0096] The above rotation instructions correspond to the basic rotation directions of the 6 faces (right, top, front, left, back, bottom) of the magic cube. The rotation instruction set generated by the system simulates the state change of the magic cube. For an N-order magic cube, the rotation instruction needs to cover all regular rotations of the 6 faces (such as top, right, front, etc.), to ensure that any state can be restored through combined instructions. Each rotation instruction will cause regular changes in the position of the color blocks on a specific face (for example, when the right face is rotated R, the color blocks on the right face will be shifted clockwise, and the edge color blocks of adjacent faces will also change), and the tensor transformation matrix records this change through coordinate mapping rules (for example, "the color block at the rth row and cth column of the right face will move to the cth row and N-1-rth column after rotation").

[0097] The state tensor after rotation can be directly simulated by matrix operation. The initial state tensor is multiplied by the tensor transformation matrix of the rotation instruction, and the new state tensor after rotation is obtained, which provides a mathematical basis for subsequent model learning of how the operation sequence changes the state.

[0098] S22, collecting a Rubik's cube restoration sample, each restoration sample including an initial state, a target state and a restoration operation sequence, the restoration operation sequence including a series of rotation instructions arranged in time sequence;

[0099] According to the state encoding model in S1, the initial state is converted into an initial state tensor, and the target state is converted into a target state tensor;

[0100] The rotation instructions in the restoration operation sequence are converted into tensor transformation matrices in time sequence.

[0101] Step S22 is to convert the Rubik's cube restoration process in the physical world into a digital sample that can be processed by a computer, realizing the whole process quantization of "state-operation-state". According to the time sequence of the operation sequence (1st step, 2nd step... Tth step), each rotation instruction (such as U of the tth step) is corresponded to the tensor transformation matrix (such as the transformation matrix M_U of U) defined in S21, forming a time sequence matrix sequence [M1, M2,..., MT] (M_t corresponds to the transformation matrix of the tth step instruction). Through the chain operation of matrix multiplication, the state change of the whole restoration process can be simulated. The initial state tensor is multiplied by M1, M2,..., MT in turn, and the final tensor obtained should be consistent with the target state tensor, which provides a digital trajectory for the model to learn the rules of the operation sequence.

[0102] S23, generating a training sample based on the restoration sample.

[0103] Based on the above embodiment, as a preferred embodiment, in step S3, the neural network is trained based on the restoration sample data, specifically including:

[0104] Based on the initial state and the target state in the restoration sample, a probability distribution of all possible rotation instructions is generated through the restoration strategy network, and the next rotation instruction is selected based on the probability distribution; by based on the initial state tensor (s0) and the target state tensor (s target ) in the restoration sample, the restoration strategy network (parameters are θ) outputs the probability distribution (P(a|θ\) of all possible rotation instructions (such as R, U', etc.), that is, the possibility of each instruction being selected.

[0105] Based on the selected next step rotation instruction, the updated state of the Rubik's Cube is determined, and the reward coefficient corresponding to the next step rotation instruction is determined based on the preset reward calculation rule; wherein the preset reward rule includes one or more of the following: the difference between the updated state and the target state, the number of operation steps between two successful color block placements, the matching degree of the operation sequence generated by the recovery strategy network with the standard operation sequence, and the redundancy operation penalty;

[0106] The above steps are repeated until the Rubik's Cube is restored to the target state, and the rotation instructions, state changes and corresponding rewards in the process from the initial state to the target state are recorded to form a complete recovery path trajectory.

[0107] In this embodiment, by designing the reward function, the Rubik's Cube recovery behavior is actually to place a specific Rubik's Cube color block (for example, there are 54 color blocks in a three-order Rubik's Cube) in a specified position, and each Rubik's Cube color block is bound to a specific "block" of the Rubik's Cube (for example, there are 26 blocks in a three-order Rubik's Cube) due to the adjacent color blocks, so the position where each color block should be placed is unique. When the Rubik's Cube color block is successfully placed in a specific position, a reward is obtained. Based on this assumption, the following rewards are set in this embodiment:

[0108] 1. State approximation reward R state , which represents the difference between the updated state and the target state; when the color block triggers successful placement, the difference between the current and the completion of the Rubik's Cube is calculated, the more the difference is reduced, the more the state approximation is considered, and the more the state approximation reward is obtained. The formula is as follows:

[0109]

[0110] Wherein, S current : current state tensor representation of the Rubik's Cube (6xNxN);

[0111] S target : target state tensor representation of the Rubik's Cube;

[0112] ||·|| F : Frobenius norm, which measures the absolute difference of matrix / tensor;

[0113] 2. Operation simplicity reward R step , the number of operation steps between two successful color block placements: when the color block triggers successful placement, the operation steps between this time and the last time are calculated, the fewer the steps, the more the reward. The formula is as follows:

[0114]

[0115] n is the current operation step; n maxis the preset maximum allowed number of rotations (adjusted according to the input training data, such as n = 100); k is the decay coefficient.

[0116] 3. Human strategy matching degree reward R human , that is, the matching degree of the operation sequence generated by the recovery strategy network to the standard operation sequence: the more the output operation instruction conforms to the human strategy, the more the reward.

[0117]

[0118] A model is the operation sequence generated by the model (such as [U, R, F,...]), A human is the human standard operation sequence (such as [U, R, F,...]), d DTW is the dynamic time warping (DTW) distance, which is used to measure the matching degree of two sequences.

[0119] 4. Redundant operation penalty P redundant , which is used to punish consecutive reverse or invalid operations.

[0120]

[0121] a i is the i-th rotation instruction (such as U, R', etc.), I(a i = a i+1 ) is an indicator function (1 if the condition is true, otherwise 0), and β is the penalty coefficient.

[0122] Actual application of the reward function: the model output operation sequence is [R-R'-U-R'-U'-R-U-R'], the Frobenius difference between the target state and the current state is 0.5, and there is a consecutive reverse operation (R, R');

[0123] Calculation:

[0124] a. R state = 1-0.5 = 0.5

[0125] b.

[0126] c.

[0127] d. P redundant = 0.1·1 = 0.1

[0128] e. R total = a1·T state + a2·T step + a3·T human + a4·T redundant

[0129] = 0.4*0.5 + 0.2*0.93 + 0.3*0.33 - 0.1 = 0.385

[0130] On the basis of the above embodiment, as a preferred implementation, in step S2, after generating the training sample based on the recovered sample, further comprising:

[0131] updating the network parameters of the recovery policy network based on Q-learning, minimizing the mean square error of the predicted Q value and the true Q value of the current recovery policy network, updating the network parameters of the recovery policy network based on maximizing the expected cumulative reward; determining the gradient loss function of the recovery policy network, performing back propagation on the policy gradient loss function, and updating the network parameters of the recovery policy network.

[0132] On the basis of the above embodiment, as a preferred implementation, updating the network parameters of the recovery policy network based on Q-learning, minimizing the mean square error of the predicted Q value and the true Q value of the current recovery policy network, specifically comprising:

[0133] determining the mean square error of the predicted Q value and the true Q value of the current recovery policy network based on Q-learning:

[0134]

[0135] In the above formula, Q(s t ,a t ; θ) is the recovery policy network with network parameters θ, the predicted Q value of the Rubik's Cube state s t at time t executing the rotation instruction a t , θ is the network parameter of the current recovery policy network; s t is the Rubik's Cube state at time t; a t is the rotation instruction executed at time t; R t is the immediate reward obtained by moving from state s t to state s t+1 after executing the action; γ is the discount factor, taking values 0-1; a' is the optional subsequent rotation instruction after state s t+1 ; Q(s t+1 ,a' ; θ - ) is the predicted Q value of the Rubik's Cube state s t+1 at time t+1 executing the rotation instruction a' by the recovery policy network with network parameters θ - ; maxa' is the subsequent rotation instruction that maximizes the predicted Q value among the optional subsequent rotation instructions of state s t+1 ; T is the upper limit of the time step number of a recovery operation sequence;

[0136] The network parameters of the recovery strategy network are adjusted through a back propagation algorithm, so that the mean square error converges.

[0137] Based on the above embodiment, as a preferred implementation, the network parameters of the recovery strategy network are updated based on maximizing the expected cumulative reward, specifically including:

[0138] Determine the average value of the cumulative reward obtained by all possible Rubik's Cube recovery path trajectories of the recovery strategy network under network parameters θ:

[0139]

[0140] In the above formula, J(θ) is the expected cumulative reward; τ is the recovery path trajectory generated by the recovery strategy network π θ ; is an expected operator; R t is the immediate reward obtained after performing action a t ; T is the upper limit of the time step of a recovery operation sequence;

[0141] Take the gradient of J(θ - ) and update the network parameters θ in the gradient direction using a gradient ascent algorithm:

[0142]

[0143] In the above formula, A t is the advantage function, representing the value advantage of the rotation instruction a t relative to the average level;

[0144] Iteratively perform the above steps until the expected cumulative reward of the recovery path trajectory generated by the recovery strategy network improves by less than a set threshold.

[0145] When recovering the Rubik's Cube, a trajectory is formed from the initial scrambled state s0 to the successfully recovered state s T . After performing a rotation action a t (such as U, R', etc.) at state s t , the subsequent states s t+1 , s t+1 ,..., s T are continuously transferred.

[0146] For each state-action (s t , a t ), the cumulative reward from time t to the end of the trajectory (recovery completion) needs to be calculated. In the formula, γ is a discount factor (such as 0.9, balancing the weight of current and future rewards) R k is the reward function based on the action performed at time k, and R totalThe obtained immediate reward (such as positive reward for returning a color block or negative reward for disordering) is accumulated through the trajectory time steps, and the rewards at each time are accumulated and discounted by γ to obtain the execution a t The cumulative return expectation starting from s t Measures the long-term value of the action.

[0147] The state value function V(s t ) focuses on the average expected reward of selecting an action according to the policy network output probability in the current state s t V(s t ) = W value ·F c +b value . F c is the feature vector after encoding the state s t (for example, a vector output by the feature extraction layer of a 6xNxN tensor matrix), W value is the weight parameter of the value network, and b value is the bias term. The restored policy network fits the mapping from the state to the expected reward by learning a large number of Rubik's Cube state samples, quickly evaluates the average value of the current state, and avoids the high cost of trajectory-by-trajectory calculation.

[0148] After calculating Q(s t ,a t ) and V(s t ), the advantage function A t = Q(s t ,a t )-V(s t ) is obtained. If A t > 0, it means that the state s t executes the action a t better than the average action, and the policy network should increase the selection probability of this action in the future; if A t < 0, the opposite is true. This difference value can accurately quantify the relative value of the action, and points out the direction for policy gradient update, so that the model can preferentially strengthen high-value actions.

[0149] Determine the gradient loss function of the restored policy network, perform back propagation on the policy gradient loss function, and update the network parameters of the restored policy network, including:

[0150] Calculate the gradient loss function of the restored policy network:

[0151]

[0152] Calculate the logarithm of the action probability, and the formula is:

[0153]

[0154] In the above formula, Wpolicy b is a weight parameter of the recovered policy network, policy b is a bias term, F FC is an input state vector, the log probability is multiplied by the advantage function A t to obtain a gradient update term.

[0155] The log probability is multiplied by the advantage function A to obtain a gradient update term.

[0156] The embodiment also provides a Rubik's cube recovery system based on deep reinforcement learning, based on the Rubik's cube recovery method based on deep reinforcement learning in the above format instance, as shown in Figure 4 The system comprises:

[0157] The encoding module 410 constructs a state encoding model based on the order of the Rubik's cube, converts the state of the Rubik's cube into a 6xNxN tensor matrix, wherein N represents the order of the Rubik's cube, and each element in the tensor matrix corresponds to the color and coordinates of a color block on the Rubik's cube.

[0158] The sample collection module 420 acquires recovery sample data of the Rubik's cube, and the recovery sample data comprises an initial state tensor of the Rubik's cube, a target state tensor, and a recovery operation sequence of the Rubik's cube converted from the initial state tensor to the target state tensor.

[0159] The recovery strategy network module 430 performs neural network training based on the recovery sample data, obtains a recovery strategy network for outputting the recovery operation sequence according to the initial state tensor and the target state tensor, and performs Rubik's cube recovery based on the recovery strategy network.

[0160] Based on the same idea, the embodiment of the application also provides an entity structure diagram, as shown in Figure 5 The server can comprise a processor 510, a communications interface 520, a memory 530 and a communications bus 540, wherein the processor 510, the communications interface 520 and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke the logical instructions in the memory 530 to execute the steps of the Rubik's cube recovery method based on deep reinforcement learning as described in the above embodiments.

[0161] In addition, the logic instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0162] Based on the same concept, the embodiments of the present application also provide a non-transitory computer readable storage medium, which stores a computer program containing at least one code, and the at least one code can be executed by a host device to control the host device to implement the steps of the Rubik's Cube restoration method based on deep reinforcement learning as described in the above embodiments.

[0163] Based on the same technical concept, the embodiments of the present application also provide a computer program, which, when executed by a host device, is used to implement the above method embodiments.

[0164] The program can be stored in whole or in part on a storage medium packaged together with the processor, or in part or in whole on a storage medium not packaged together with the processor.

[0165] Based on the same technical concept, the embodiments of the present application also provide a processor for implementing the above method embodiments. The processor can be a chip.

[0166] The embodiments of the present application can be combined arbitrarily to achieve different technical effects.

[0167] In the above embodiments, all or part of the methods can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the methods can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk), etc.

[0168] Those of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be instructed by a computer program to complete the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The aforementioned storage medium includes ROM or random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0169] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A Rubik's Cube restoration method based on deep reinforcement learning, characterized in that: include: S1. Construct a state encoding model based on the order of the Rubik's Cube to convert the state of the Rubik's Cube into a 6×N×N tensor matrix; where N represents the order of the Rubik's Cube, and each element in the tensor matrix corresponds to the color and coordinates of a color block on the Rubik's Cube; S2. Obtaining restoration sample data of the Rubik's Cube, wherein the restoration sample data includes an initial state tensor and a target state tensor of the Rubik's Cube, and a restoration operation sequence for converting the Rubik's Cube from the initial state tensor to the target state tensor; S3. Perform neural network training based on the restoration sample data to obtain a restoration strategy network for outputting a restoration operation sequence according to an initial state tensor and a target state tensor, and perform Rubik's Cube restoration based on the restoration strategy network.

2. The Rubik's Cube restoration method based on deep reinforcement learning according to claim 1, characterized in that: The step S1 specifically includes: S11. Assign a unique identifier color(0,i) to each color block of the N-order Rubik's Cube, where i is the color block index, i∈[0,N 2 ×6-1]; S12. When the Rubik's Cube is fully restored, define the standard position coordinates Location(0,i) for each color block; S13. Expand the six faces of the Rubik's Cube into six independent two-dimensional planes, each of which includes N×N color blocks. Map the expanded six planes into a 6×N×N tensor matrix, in which a unique identifier of each color block is associated with a standard position coordinate.

3. The Rubik's Cube restoration method based on deep reinforcement learning according to claim 1, characterized in that: The step S2 specifically includes: S21, determining a rotation instruction set for an N-order Rubik's Cube, where each rotation instruction in the rotation instruction set includes a rotation object, a rotation direction, and a rotation angle; and determining a tensor transformation matrix for each rotation instruction based on the rotation object, the rotation direction, and the rotation angle; S22, collecting Rubik's Cube restoration samples, each restoration sample including an initial state, a target state, and a restoration operation sequence, wherein the restoration operation sequence includes a series of rotation instructions arranged in chronological order; Converting the initial state into an initial state tensor and converting the target state into a target state tensor according to the state encoding model in S1; Convert each rotation instruction in the restoration operation sequence into a tensor transformation matrix in a time sequence; S23. Generate a training sample based on the restored sample.

4. The Rubik's Cube restoration method based on deep reinforcement learning according to claim 3, characterized in that: In step S3, neural network training is performed based on the restored sample data, specifically including: Based on the initial state and target state in the restored sample, a probability distribution of all possible rotation instructions is generated through the restoration strategy network, and the next rotation instruction is selected based on the probability distribution; Determining an updated state of the Rubik's Cube based on the selected next rotation instruction, and determining a reward coefficient corresponding to the next rotation instruction based on a preset reward calculation rule; wherein the preset reward rule includes one or more of the following: the difference between the updated state and the target state, the number of operation steps between two successful color block placements, the degree of match between the operation sequence generated by the restoration strategy network and the standard operation sequence, and a penalty for redundant operations; Repeat the above steps until the Rubik's Cube is restored to the target state. Record the rotation instructions, state changes, and corresponding rewards from the initial state to the target state to form a complete restoration path trajectory.

5. The Rubik's Cube restoration method based on deep reinforcement learning according to claim 4, characterized in that: In the step S2, after generating the training sample based on the restored sample, the method further includes: Update the network parameters of the restoration policy network based on Q-learning, minimize the mean square error between the predicted Q value and the true Q value of the current restoration policy network, and update the network parameters of the restoration policy network based on maximizing the expected cumulative reward; determine the gradient loss function of the restoration policy network, backpropagate the policy gradient loss function, and update the network parameters of the restoration policy network.

6. The Rubik's Cube restoration method based on deep reinforcement learning according to claim 5, characterized in that: Update the network parameters of the restoration policy network based on Q-learning to minimize the mean square error between the predicted Q value and the true Q value of the current restoration policy network. Specifically, Determine the mean square error between the predicted Q value and the true Q value of the current restoration strategy network based on Q-learning: In the above formula, Q(s t ,a t ; θ) is the restoration strategy network with network parameter θ, which is used to restore the cube state s at time t t Execute rotation instruction a t The predicted Q value, θ is the network parameter of the current restoration strategy network; s t is the state of the Rubik's Cube at time t; a t is the rotation instruction executed at time t; R t After executing the action, from state s t Transfer to state s t+1 The immediate reward obtained; γ is the discount factor, ranging from 0 to 1; a' is the state s t+1 Optional subsequent rotation instructions; Q(s t+1 ,a';θ - ) is the network parameter θ - The restoration strategy network, at time t+1, is used to restore the Rubik's Cube state s t+1 The predicted Q value of executing the rotation instruction a'; maxa' is the Q value of the state s t+1 Among the optional subsequent rotation instructions, the subsequent rotation instruction that maximizes the predicted Q value; T is the upper limit of the number of time steps of a restoration operation sequence; The network parameters of the restoration strategy network are adjusted through the back propagation algorithm so that the mean square error converges.

7. The Rubik's Cube restoration method based on deep reinforcement learning according to claim 6, characterized in that: Update the network parameters of the restoration strategy network based on maximizing the expected cumulative reward, specifically including: Determine the average value of the cumulative rewards obtained by all possible Rubik's Cube restoration path trajectories under the network parameters θ of the restoration strategy network: In the above formula, J(θ) is the expected cumulative reward; τ is the restoration path trajectory, which is determined by the restoration strategy network π θ generate; is the expectation operator; R t To perform action a t The immediate reward obtained after ; T is the upper limit of the time steps of a recovery operation sequence; For J(θ - ) to find the gradient, and use the gradient ascent algorithm to update the network parameters θ along the gradient direction: In the above formula, A t is the advantage function, which represents the rotation instruction a t Value advantage relative to the average; The above steps are iterated until the increase in the expected cumulative reward of the restoration path trajectory generated by the restoration strategy network is less than the set threshold.

8. A Rubik's Cube restoration system based on deep reinforcement learning, characterized in that: include: The encoding module builds a state encoding model based on the order of the Rubik's Cube and converts the state of the Rubik's Cube into a 6×N×N tensor matrix; where N represents the order of the Rubik's Cube and each element in the tensor matrix corresponds to the color and coordinates of a color block on the Rubik's Cube; A sample acquisition module is configured to obtain restored sample data of the Rubik's Cube, wherein the restored sample data includes an initial state tensor and a target state tensor of the Rubik's Cube, and a restoration operation sequence for converting the Rubik's Cube from the initial state tensor to the target state tensor; The restoration strategy network module performs neural network training based on the restoration sample data to obtain a restoration strategy network for outputting a restoration operation sequence according to an initial state tensor and a target state tensor, and performs Rubik's Cube restoration based on the restoration strategy network.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the Rubik's Cube restoration method based on deep reinforcement learning as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the Rubik's Cube restoration method based on deep reinforcement learning as described in any one of claims 1 to 7 are implemented.

Images