A robot path planning method, device and medium based on quantum search and reinforcement learning

Abstract

The present invention discloses a robot path planning method, device and medium based on quantum search and reinforcement learning. The method includes quantizing the real environment and robot state, dynamically mapping the real world environment and robot state to the quantum field to establish a quantum domain; based on a preset quantum search model, searching in the quantum field using the quantum superposition principle and obtaining multiple search samples; determining one or more target search samples from multiple search samples according to the probability of the robot reaching the target position; predetermining a basic reinforcement learning model, and training the basic reinforcement learning model using the quantum search model and the target search sample, and determining the trained reinforcement learning model as the robot path planning model. The present invention can quickly obtain high-quality target search samples, effectively reduce the number of robot path planning training rounds, and improve the quality and speed of robot path planning.

Description

Technical Field

[0001] The present invention relates to the field of robot path planning, and in particular to a robot path planning method, device and medium based on quantum search and reinforcement learning. Background Art

[0002] With the development of sensors, batteries, and artificial intelligence (AI) technologies, robots have taken on important tasks in various unstructured environments, such as disaster site rescue, search operations within building clusters, and package management in indoor cargo areas. However, although robots can play a role in these complex and changing environments, their adaptation speed is still limited, mainly due to their limited on-board computing power and dynamic interference from the environment, such as dynamic obstacles and changes in battery power. These factors lead to inaccuracies in robot planning and delayed responses in emergency situations.

[0003] Reinforcement Learning (RL) is a machine learning method that allows an intelligent agent (such as a robot) to learn how to take actions to maximize a certain cumulative reward through interaction with the environment. Reinforcement learning provides several key advantages in adapting robots to dynamic environments. Adaptability: Reinforcement learning algorithms allow robots to self-learn and adapt in unknown or changing environments. Through trial and error, robots can gradually understand patterns in the environment and learn to effectively respond to changes in these patterns; Balance between exploration and exploitation: Reinforcement learning algorithms are usually designed with a balance mechanism between exploration and exploitation, which means that robots will not only exploit the best known strategies, but also explore unknown possibilities to discover better strategies. Although the introduction of reinforcement learning and environmental feedback to enhance the robot's exploration ability can increase its adaptation speed, this learning method is not only time-consuming and costly due to the need for a large number of interactions with the environment, but also has certain risks, such as collisions between robots and obstacles.

[0004] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of the present application, nor does it necessarily provide technical guidance. In the absence of clear evidence that the above content has been disclosed before the filing date of the present application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0005] The purpose of the present invention is to provide a robot path planning method, device and medium based on quantum search and reinforcement learning. Different from the traditional quantum reinforcement learning method based on quantum network model instead of neural network model, the present invention is based on the quantization of real environment and quantum superposition mechanism, and quickly obtains high-quality target search samples for path planning strategy training. It can not only improve the accuracy of robot path planning, but also reduce the number of interactions between reinforcement learning and the environment without changing the underlying reinforcement learning algorithm architecture, effectively reduce the number of training rounds, and improve the efficiency and speed of robot path planning.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A robot path planning method based on quantum search and reinforcement learning comprises the following steps:

[0008] Quantize the real environment and robot state, including dynamically mapping the real-world environment and robot state to the quantum domain to establish a quantum domain;

[0009] Based on a preset quantum search model, the quantum superposition principle is used to search in the quantum domain and obtain multiple search samples, each of which corresponds to the probability that the robot reaches the target position;

[0010] Determine one or more target search samples from a plurality of search samples according to the probability that the robot reaches the target position;

[0011] A basic reinforcement learning model is predetermined, and the basic reinforcement learning model is trained using the quantum search model and the target search sample, and the trained reinforcement learning model is determined to be a robot path planning model.

[0012] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, in the process of quantizing the robot state, the robot state is encoded and the robot state is represented as a quantum superposition state;

[0013] Based on a preset quantum search model, the quantum superposition state is used to perform a parallel search in the quantum domain and obtain multiple search samples, wherein the parallel search is to control the robot to simultaneously explore multiple positions in the quantum domain.

[0014] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the real-world environment and robot state are dynamically mapped to the quantum field to establish a quantum domain, wherein the quantum domain includes a quantized simulation environment, quantum state parameters, and quantum state initialization information;

[0015] The method for establishing the quantized simulation environment comprises: based on the reachability of the robot in the real environment, encoding the environment so that each position in the environment has a unique binary code, the binary code corresponds to the orthogonal basis of the quantum code, and obtaining the quantized simulation environment;

[0016] The quantum state parameters are obtained by mapping elements in the real-world reinforcement learning framework to the quantized simulation environment. The elements in the real-world reinforcement learning framework include the current state s t 、Current moment action a t , the next moment state s t+1 and reward r;

[0017] The quantum state initialization information is obtained by mapping the initial position and target position of the robot in the real environment and the obstacle distribution information in the real environment into the quantized simulation environment.

[0018] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the reachability of the robot in the real environment is determined by an adjacency matrix and expressed as the following formula:

[0019]

[0020] Among them, A represents the reachable matrix of the robot in the real environment, A surround Represents the robot reachability matrix under the influence of the surrounding environment characteristics, A obs represents the robot reachability matrix under the influence of obstacle distribution, A robot represents the robot reachability matrix under the influence of robot settings, Represents the element-wise product operator.

[0021] Further, any one of the above-mentioned technical solutions or a combination of multiple technical solutions further includes setting the probability value corresponding to the position of the robot at the initial moment to be greater than the probability values ​​corresponding to other positions; and / or,

[0022] The exit node is set so that when the robot reaches the target position, the robot transfers to the exit node with a probability of 100% at the next moment, and when the robot is at the exit node, the robot will not transfer to the target position at the next moment.

[0023] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the quantum search model is constructed by using a quantum random walk model and a classical random walk model based on the master equation of the open system. The quantum search model is expressed by the following formula:

[0024] dρ / dt=(1-p)L QW (ρ)+pL CRW(ρ) + L target (ρ)

[0025] where ρ represents the probability distribution of the robot state, p represents the weighting coefficient of the quantum random walk model, 1 - p represents the weighting coefficient of the classical random walk model, and L QW (ρ) = -i[A′, ρ] represents the quantum random walk model, where A ′ represents the reachability matrix of the robot in the quantization simulation environment, represents the classical random walk model, where L ij =(A′ ij / d j )|i><j|, d j represents the degree of position j, and |i> represents the i-th orthogonal basis of the Hilbert space, L target (ρ) = 2|n + 1><n|ρ|n><n + 1| - {|n><n|, ρ} means that when the robot state is at the target position (n), the robot state will irreversibly transfer to the exit node (n + 1) at the next moment.

[0026] Furthermore, based on any one of the foregoing technical solutions or a combination of multiple technical solutions, in the quantization simulation environment, the probability of the robot being located in each grid is represented by the diagonal element ρ ij of the probability distribution ρ, where i == j;

[0027] After T-time parallel search, the probability p(T) of the robot reaching the target position in the quantization simulation environment is expressed as: where n represents all target positions in the environment.

[0028] Furthermore, based on any one of the foregoing technical solutions or a combination of multiple technical solutions, one or more target search samples are determined from multiple said search samples by comparing the probabilities of the robot reaching the target position, including:

[0029] Determining the search sample corresponding to the highest value of the probability of the robot reaching the target position as the target search sample; or,

[0030] Determining the search samples corresponding to the probabilities of the robot reaching the target position greater than a preset threshold as the target search samples; or,

[0031] Sorting in descending order according to the probabilities of the robot reaching the target position, and confirming the search samples corresponding to the top k probabilities as the target search samples, where k ≥ 2 and k is a positive integer.

[0032] Furthermore, based on any one of the foregoing technical solutions or a combination of multiple technical solutions, the structure of the target search sample is (s t_b, a t_b ,s t+1_b , r b ), where s t_b The current state of the target search sample in the quantized simulation environment, a t_b The current action of the robot in the quantized simulation environment corresponding to the target search sample, s t+1_b The next moment state corresponding to the target search sample in the quantized simulation environment, including the robot state and the environment state, r b The reward returned by the real environment corresponding to the target search sample.

[0033] Further, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the basic reinforcement learning model is trained using the quantum search model and the target search sample, including the following steps:

[0034] Encapsulating the quantum search model and the target search sample into a quantum meditation function module, wherein the input of the quantum meditation function module is the current state, and the output is the optimal robot action;

[0035] The quantum meditation function module is added to the basic reinforcement learning model as a robot action selection module, and the basic reinforcement learning model is trained with target search samples as a training set.

[0036] Further, based on any one of the above technical solutions or a combination of multiple technical solutions, the optimal robot action is determined by the following method:

[0037] Based on the quantum search model, the robot traverses all possible actions in the quantum field with the current state as input, and obtains the probability that the robot can reach the target position when choosing different actions. According to the probability position of reaching the target position corresponding to each robot action, the robot action with the highest probability of reaching the target position at the current moment is determined as the optimal robot action.

[0038] Furthermore, based on any one of the technical solutions or a combination of multiple technical solutions mentioned above, the robot path planning model is used to obtain a planned path of the robot from an initial position to a target position.

[0039] According to another aspect of the present invention, a robot path planning device is provided, wherein a robot path planning model is provided in the robot path planning device; wherein the robot path planning model is obtained by:

[0040] Quantize the real environment and robot state, including dynamically mapping the real-world environment and robot state to the quantum domain to establish a quantum domain;

[0041] Based on a preset quantum search model, the quantum superposition principle is used to search in the quantum domain and obtain multiple search samples, each of which corresponds to the probability that the robot reaches the target position;

[0042] Determine one or more target search samples from a plurality of search samples according to the probability that the robot reaches the target position;

[0043] A basic reinforcement learning model is predetermined, and the basic reinforcement learning model is trained using the quantum search model and the target search sample, and the trained reinforcement learning model is determined to be a robot path planning model.

[0044] Furthermore, based on any one of the technical solutions or a combination of multiple technical solutions described above, the robot path planning model is obtained based on the robot path planning method based on quantum search and reinforcement learning as described in any one of the technical solutions or a combination of multiple technical solutions described above.

[0045] According to another aspect of the present invention, a computer-readable storage medium is provided for storing program instructions, wherein the program instructions are configured to be called by a processor to execute the steps of the method described in any one of the above technical solutions or a combination of multiple technical solutions.

[0046] The beneficial effects brought by the technical solution provided by the present invention are as follows:

[0047] a. Based on the real environment quantization and quantum superposition mechanism, the present invention can quickly obtain high-quality target search samples for path planning strategy training, which can not only improve the accuracy of robot path planning, but also reduce the number of interactions between reinforcement learning and the environment without changing the underlying reinforcement learning algorithm architecture, effectively reduce the number of training rounds, and improve the efficiency and speed of robot path planning;

[0048] b. The present invention represents the robot state as a quantum superposition state during the process of quantizing the robot state, which can realize parallel search, so that the robot can explore multiple positions at the same time, which can effectively improve the efficiency of obtaining each search sample and improve the speed of robot path planning;

[0049] c. By combining quantum search with parallel search, the present invention can easily obtain useful high-quality target search samples, that is, samples whose rewards are not 0 or provide more target position information. Therefore, the robot path planning method based on quantum search and reinforcement learning provided by the present invention is suitable for general robot learning scenarios, especially for path planning tasks in sparse reward environments;

[0050] d. The present invention determines the reachability of the robot in the real environment based on the characteristics of the surrounding environment, the distribution of obstacles and the robot settings, and maps them into the quantum domain to construct a quantized simulation environment, which can improve the accuracy of the robot's path planning;

[0051] e. The present invention constructs a quantum search model through a weighted combination of a quantum random walk model and a classical random walk model, which can ensure fast search while improving the anti-interference ability during the search process. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flowchart of a robot path planning method based on quantum search and reinforcement learning provided for an exemplary embodiment of the present invention;

[0054] Figure 2 A schematic diagram of factors affecting the quantization of a real environment provided for an exemplary embodiment of the present invention;

[0055] Figure 3 A flow chart of a quantum search process provided for an exemplary embodiment of the present invention;

[0056] Figure 4 A framework diagram of the integration of quantum search and reinforcement learning provided for an exemplary embodiment of the present invention;

[0057] Figure 5 A schematic diagram comparing the number of reinforcement learning training rounds for a model with quantum search provided as an exemplary embodiment of the present invention and the number of reinforcement learning training rounds corresponding to other models. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0059] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0060] In view of the shortcomings of the prior art, the present invention provides a robot path planning method, device and medium based on quantum search and reinforcement learning. The present invention is based on the quantum superposition mechanism and quickly obtains high-quality target search samples for path planning strategy training, which can not only improve the accuracy of robot path planning, but also effectively reduce the number of training rounds and improve the speed of robot path planning.

[0061] In one embodiment of the present invention, a robot path planning method based on quantum search and reinforcement learning is provided. Figure 1 , the method comprises the following steps:

[0062] Quantize the real environment and robot state, including dynamically mapping the real-world environment and robot state to the quantum domain to establish a quantum domain;

[0063] Based on a preset quantum search model, the quantum superposition principle is used to search in the quantum domain and obtain multiple search samples, each of which corresponds to the probability that the robot reaches the target position;

[0064] Determine one or more target search samples from a plurality of search samples according to the probability that the robot reaches the target position;

[0065] A basic reinforcement learning model is predetermined, and the basic reinforcement learning model is trained using the quantum search model and the target search sample, and the trained reinforcement learning model is determined to be a robot path planning model.

[0066] In one embodiment of the present invention, in the process of quantizing the robot state, the robot state is encoded and represented as a quantum superposition state, and the quantum superposition state is configured to represent the probability of the robot being at various positions in the quantum domain at the same time. Based on a preset quantum search model, the quantum superposition state is used to perform parallel search in the quantum domain and obtain multiple search samples, wherein the parallel search is to control the robot to explore multiple positions in the quantum domain at the same time.

[0067] Combine the following Figures 1 to 5 The embodiments of the present invention are further described.

[0068] The real environment and robot state are quantized, including dynamically mapping the real world environment and robot state to the quantum field to establish a quantum domain. The quantum domain includes a quantized simulation environment, quantum state parameters, and quantum state initialization information.

[0069] S1. Quantize the real environment, dynamically map the real-world environment to the quantum field, and establish a quantized simulation environment.

[0070] S1.1. In one embodiment of the present invention, the accessibility of the robot in the real environment is represented by the adjacency matrix method, and the environmental dynamics of the real world in the quantum field are described based on the reachable matrix, which is used to simulate the quantum search model built later. Of course, the method for realizing the quantization of the real environment is not limited to the adjacency matrix method. Any method that can achieve the same technical effect as the adjacency matrix method can be used for the quantization of the real environment, such as lists, trees, and graph methods used to describe the relationship between adjacent states of the robot, which can achieve the same technical effect as the adjacency matrix method.

[0071] The method of using an adjacency matrix to represent the reachability of the robot in the real environment is a preferred embodiment provided by the present invention. In this embodiment, the reachability of the robot in the real environment is determined by encoding the environment, and a quantized simulation environment is established according to the reachability of the robot in the real environment, in which each position has a unique binary code corresponding to an orthogonal basis of the quantum code.

[0072] S1.1.1. For example, a real environment of 400m×400m is divided into 13×13 two-dimensional grids. Of course, in other embodiments, the real environment may be divided into different grid sizes and dimensions according to the size, requirements and coarseness requirements.

[0073] Then, if Figure 2 As shown in the figure, for the path planning task, the following three aspects affect the reachability of the robot in the real environment:

[0074] (1) Characteristics of the surrounding environment: For example, in order to ensure the safety of the robot, in an unstructured and uncertain environment, the robot needs to maintain a first distance (for example, ≥1.0m) from the obstacle, while in a structured environment (the obstacle is a simple geometric structure, such as a rectangle, triangle, circle, etc.), the robot only needs to maintain a second distance (for example, ≥0.5m) from the obstacle, and the second distance is smaller than the first distance. It should be noted that 1.0m and 0.5m in this paragraph are only exemplary and do not limit the scope of protection of the present invention. In other embodiments or specific applications, the first distance can be determined to be 0.5m, 2m, 3m, etc. according to actual needs, and the second distance can be 0.1m, 0.2m, 1m, etc.

[0075] (2) Obstacle distribution: The layout of buildings, trees, and rocks will affect the robot’s accessibility;

[0076] (3) Robot settings: The robot's "capability", "fuel level" and "condition" will affect the robot's accessibility. For example, a small robot cannot cross a muddy road, and a robot with little fuel cannot reach a far location. Of course, in other embodiments, the robot's accessibility can also be determined based on other empirical knowledge.

[0077] Based on the above analysis, the present invention proposes the following formula to determine the reachability of the robot in a real environment:

[0078]

[0079] Among them, A represents the reachable matrix of the robot in the real environment, A surround Represents the robot reachability matrix under the influence of the surrounding environment characteristics, A obs represents the robot reachability matrix under the influence of obstacle distribution, A robot represents the robot reachability matrix under the influence of robot settings, Represents the element-wise product operator. In the reachable matrix, if the robot can reach the environment position corresponding to the matrix element, the value of the matrix element is set to 1, otherwise it is 0.

[0080] S1.1.2. Use Python programming language to build a simulation environment based on the reachable matrix A. Figure 3 (upper right) shows that the simulation environment has similar Figure 3 The maze structure shown in the upper right corner, where the reachable matrix element value is 1 corresponds to the white position in the simulation environment, otherwise it corresponds to the black position in the simulation environment. At the same time, the cyan position in the simulation environment represents the initial position of the robot, and the red represents the target position. Of course, in other embodiments, other software and programming languages ​​can also be used to build the simulation environment.

[0081] S1.2. Quantize the robot state, map the robot state in the real environment to the quantized simulation environment, and encode the robot state to represent the robot state in the quantum domain as a quantum superposition state, so that the robot can explore multiple positions in the quantized simulation environment at the same time.

[0082] S1.2.1. In one embodiment of the present invention, in order to utilize the efficient parallel search principle of the quantum superposition mechanism and to more conveniently integrate with the reinforcement learning framework later, the elements in the real-world reinforcement learning framework (including the current state s t , the current action a t , the next moment state s t+1 and reward r) is mapped into the quantum field to obtain the quantum state parameters. The quantum state parameters include the current state in the quantized simulation environment |s t >、Current moment action|a t >, next moment state |s t+1 > and reward |r>, where |s t > is the current state in the quantized simulation environment, which includes the robot state and the environment state, |a t > is the current action of the robot in the quantized simulation environment, |s t+1 > is the state at the next moment in the quantized simulation environment, which includes the robot state and the environment state, and |r> is the reward in the superposition state returned in the quantum domain.

[0083] In the quantum domain, in order to use quantum superposition to accelerate computing, the quantum search model based on quantum dynamics is in a superposition state. t >As input, the robot can choose multiple actions in a superposition state|a t >, get the |s in superposition state at the next moment t+1 >, and obtain the corresponding reward |r> in the superposition state. In the quantum domain, the robot may be in multiple positions at any time, so a discrete probability distribution ρ is used to represent the superposition state |s t >. The number of possible values ​​in the discrete probability distribution ρ is equal to the number of grids in the simulation environment, which is 13×13=169 in one embodiment of the present invention.

[0084] S1.2.2. Each grid in the quantized simulation environment has a unique binary code corresponding to the orthogonal basis of the quantum code. The number of qubits required is obtained by the following equation:

[0085]

[0086] Among them, N bitrepresents the number of qubits, N cell Indicates the number of grids. In one embodiment of the present invention, the required number of qubits is 8. Of course, in other embodiments, the appropriate number of qubits can also be selected according to actual conditions. Quantum domain is the robot action in the quantized simulation environment | a t > is another probability distribution over all possible actions of the robot. In one embodiment of the present invention, a discrete uniform distribution is used to represent the robot action. t >, of course, other probability distributions may also be used in other embodiments.

[0087] S1.3. Determine quantum state initialization information based on the initial position and target position of the robot in the real environment and the obstacle distribution information in the real environment, wherein the quantum state initialization information includes the initial position, target position and initial position of the obstacle of the robot in the quantized simulation environment.

[0088] Each grid position in the quantized simulation environment has a binary code, which corresponds to the quantum coding orthogonal basis one by one. When the robot searches in parallel, it may be in multiple grid positions at any time, and the probability distribution on the quantum coding orthogonal basis can be used to describe the robot's parallel search. Therefore, the initial positions of the robot, target point and obstacle can be set by initializing the quantum state.

[0089] Specific as Figure 3 As shown in the lower right of the figure, the probability value corresponding to the position where the robot is at the initial moment is set higher than the probability of other positions. Add an exit node (Exit Node) at the position corresponding to the target point. The exit node is usually a virtual node. During parallel search, when the robot reaches the target position, the robot will transfer to the exit node with a probability of 100% at the next moment; on the contrary, if the robot is at the exit node, it will not transfer to the target position at the next moment and will remain at the exit node. Therefore, after a period of search, the probability of the robot being at the exit node represents the total probability that the robot can reach the target point within the search time period, and this total probability is used to judge the quality of the search sample. Therefore, after the quantum random walk parallel search, the probability value of the exit node is regarded as the future cumulative reward |r> obtained by the robot after taking action from the initial position.

[0090] In the present invention, sample quality refers to how much information the sample can provide about where the target point is or about the target point. For example, in traditional reinforcement learning, the samples obtained by searching the environment are likely to be useless samples (the reward is 0, or no information about the location of the target point is provided), but the present invention can easily obtain useful samples (i.e., samples whose reward is not 0 or provide more target location information) by combining quantum search with parallel search. Then, the obtained high-quality samples are used to train reinforcement learning, which can improve the efficiency of path planning learning and training, as well as improve the accuracy and speed of robot path planning.

[0091] S2. Based on a preset quantum search model, the quantum superposition principle is used to perform parallel searches in the quantized simulation environment and obtain multiple search samples. During the parallel search process, each search process starts from the initial position of the robot and ends at the target position to simultaneously search different paths. When the exit node is reached, the search process ends and the search sample corresponding to the search process is obtained.

[0092] By comparing the probability of the robot reaching the target position, one or more target search samples are determined from the plurality of search samples. The target search samples in the present invention are high-quality search samples. There are many ways to determine high-quality target search samples.

[0093] The first way to determine the target search sample is to determine that the search sample corresponding to the highest value of the probability of the robot reaching the target position is the target search sample. Another way is to determine that the search samples corresponding to each probability of the robot reaching the target position greater than a preset threshold are the target search samples. In addition, the target search sample can also be determined in the following way: sorting in descending order according to the probability of the robot reaching the target position, and confirming that the search samples corresponding to the top k (for example, the top three) probabilities in the sorting are the target search samples, k≥2 and k is a positive integer.

[0094] Among the multiple ways of determining the target search samples in the above examples, the first way of determining the target search samples is the best. This way can not only ensure that effective target search samples can be obtained, but also that the obtained target search samples are also the search samples with the highest quality.

[0095] In this embodiment, the open system master equation (Lindblad Master) equation can be used to describe the environmental dynamics of the robot in a quantized simulation environment, and a quantum search model can be constructed for rapid search of high-quality samples. Of course, the method of building a quantum search model is not limited to the Lindblad Master equation. Any method that can achieve the same technical effect as the Lindblad Master equation can be used to build a quantum search model, such as the use of the Lindblad master equation, Redfield relaxation and Floquet theory, as well as Suzuki-Trotter expansion and sparse solver numerical methods and other technologies.

[0096] In a preferred embodiment of the present invention, step S2 includes the following steps.

[0097] S2.1. Based on the open system master equation, a quantum search model is constructed using a quantum random walk model and a classical random walk model. Among them, the classical random walk model can increase the anti-interference ability of the quantum search model. In the search process of the quantized simulation environment, the evolution of the discrete probability distribution ρ over time is controlled by the following open system master equation (LindbladMaster), which is a weighted combination of the quantum random walk model and the classical random walk model, which can ensure fast search and improve the anti-interference ability of the quantum search model:

[0098] dρ / dt=(1-p)L QW (ρ)+pL CRW (ρ)+L target (ρ)

[0099] Among them, ρ represents the discrete probability distribution of the robot state, p represents the weighting coefficient of the quantum random walk model, 1-ρ represents the weighting coefficient of the classical random walk model, and L QW (ρ)=-i[A,ρ] represents a quantum random walk model, wherein A′ represents the reachable matrix of the robot in the quantized simulation environment, and A′ is obtained by converting the reachable matrix A of the robot in the real environment into the quantum domain. represents the classic random walk model, where L ij =(A′ ij / d j )|i> <j|,A′ ij represents the value of the (i, j) element in the reachable matrix A′, indicating whether the robot can reach the position corresponding to (i, j). j represents the degree of position j, |i> represents the i-th orthogonal basis of Hilbert space, L target(ρ) = 2|n + 1><n|ρ|n><n + 1| - {|n><n|, ρ} indicates that when the robot state is at the target position (n), the robot state will irreversibly transfer to the exit node (n + 1) at the next moment.

[0100] In the quantization simulation environment, the probability of the robot being located in each grid is represented by the diagonal element ρ of the probability distribution ρ ij where i == j. After parallel search for T time, the probability p(T) that the robot reaches the target position in the quantization simulation environment is expressed as: where n represents all target positions in the environment.

[0101] S2.2. After the robot selects different actions, by comparing the probabilities of the future robot reaching the target position corresponding to each search sample, and determining the search sample with the largest probability value as the target search sample. The search samples in the present invention have the same structure as the samples in traditional reinforcement learning, and are all composed of the current state s t , the current action a t , the next state s t+1 and the reward r. At the current moment s t , the robot will traverse all possible actions, and according to the quantum search model, obtain the sum of the probabilities that the robot can reach the target position when selecting different actions a According to the probability of reaching the target position corresponding to each robot action, select the optimal robot action at the current moment and denote it as a t .

[0102] S3. Predetermine a basic reinforcement learning model, and use the quantum search model and the target search sample to train the basic reinforcement learning model, and determine the trained reinforcement learning model as the robot path planning model. By integrating high-quality target search samples into the reinforcement learning process and training the basic reinforcement learning model, the present invention can improve the learning efficiency and the quality of the path planning strategy.

[0103] In a preferred embodiment of the present invention, step S3 includes the following sub-steps.

[0104] S3.1. Package the quantum search model and the target search sample into a quantum meditation function module, and the input of the quantum meditation function module is the current state, and the output is the optimal robot action.

[0105] As Figure 3 shown, the input of the quantum meditation function module is the current state s t , and the output is the optimal robot action a t . Given the input s tThe robot will traverse all possible actions in the quantum field and obtain the probability of reaching the target position when the robot chooses different actions a based on the quantum search model. According to the probability of reaching the target position corresponding to each robot action, the optimal robot action at the current moment is selected and recorded as a t , and output. Preferably, the optimal robot action is the robot action corresponding to the maximum probability of the robot reaching the target position.

[0106] S3.2, adding the quantum meditation function module as a robot action selection module to the basic reinforcement learning model, and training the basic reinforcement learning model with the target search sample as the training set. Integrating the quantum meditation function with the traditional reinforcement learning process can improve the learning and training efficiency and the quality of the path planning strategy.

[0107] like Figure 4 As shown in the figure, the quantum meditation function is integrated into the traditional reinforcement learning process as a robot action selection module. In the reinforcement learning training process, the current moment s t Pass it to the quantum meditation function module to obtain the optimal robot action a t Then, under the reinforcement learning framework, the robot performs the optimal action in the simulation environment and obtains the reward r and the next state s t+1 , high-quality target search samples (s t , a t ,s t+1 , r) are stored for training path planning strategies.

[0108] It should be noted that in the quantum domain, the reward is a probability value, and high-quality target search samples are obtained through probability; the reward in the reinforcement learning process is the reward returned by the real environment, and the reward at this time is not a probability value. The reward function in the reinforcement learning training provided by the present invention not only considers the probability of the robot reaching the target position, but also considers the length of the path and the time to reach the target point, as described in detail below.

[0109] In a preferred embodiment of the present invention, the basic learning model can be trained by using the time difference prediction error (TD PredictionError). The time difference prediction error and the parameter update of the reinforcement learning are shown in the following equations:

[0110] TD error =r+γV(s t+1 )-V(s t )

[0111] V(s t )←V(s t )+α[r+γV(s t+1 )-V(st )]

[0112] Among them, r represents the reward returned by the real environment, γ represents the weighting factor of future cumulative rewards, α represents the reinforcement learning rate, V(s t+1 ) represents the state value predicted at time t+1 in the reinforcement learning process, V(s t ) represents the state value predicted at time t in the reinforcement learning process, TD error Represents the difference between the expected value and the predicted value of reinforcement learning.

[0113] In one example of the present invention, the environment reward r is set as follows: if the robot reaches the target position, the reward is +1, otherwise the reward is -0.01. In one embodiment of the present invention, the reinforcement learning learning rate α is set to 0.01.

[0114] The loss function of the above reinforcement learning training can be iteratively trained using the Adam optimization method.

[0115] See also Figure 5 , Figure 5 The horizontal axis represents different environment sizes, and the vertical axis represents the value of the required number of training rounds. Model 1 represents the deep reinforcement learning benchmark model with Replay Buffer (abbreviated as DQN.W), and Model 2 represents the deep reinforcement learning benchmark model with Replay Buffer with Monte Carlo search (abbreviated as MCT DQN.W). This model is the deep reinforcement learning model with Replay Buffer with quantum search proposed by the present invention (abbreviated as QuantumDQN.W). Figure 5 It can be seen that for different environment sizes, the number of training rounds required for the path planning method proposed in the present invention is less than that required for model one and model two, indicating that the present invention can quickly obtain high-quality target search samples for path planning strategy training, reduce the number of training rounds, and improve the accuracy and speed of the robot path planning method.

[0116] In one embodiment of the present invention, the robot path planning model determined in any of the above embodiments is used to obtain a planned path of the robot from an initial position to a target position.

[0117] In one embodiment of the present invention, a robot path planning device is provided, wherein a robot path planning model is provided in the robot path planning device; wherein the robot path planning model is obtained by:

[0118] Quantize the real environment and robot state, including dynamically mapping the real-world environment and robot state to the quantum domain to establish a quantum domain;

[0119] Based on a preset quantum search model, the quantum superposition principle is used to search in the quantum domain and obtain multiple search samples, each of which corresponds to the probability that the robot reaches the target point position;

[0120] Determine one or more target search samples from a plurality of search samples according to the probability that the robot reaches the target point position;

[0121] A basic reinforcement learning model is predetermined, and the basic reinforcement learning model is trained using the quantum search model and the target search sample, and the trained reinforcement learning model is determined to be a robot path planning model.

[0122] In one embodiment of the present invention, a robot path planning model training device is provided, including a quantization module, a quantum search module and a reinforcement learning module.

[0123] The quantization module is configured to quantize the real environment and the robot state, including dynamically mapping the real world environment and the robot state to the quantum domain to establish a quantum domain;

[0124] A quantum search model is preset in the quantum search module. The quantum search module is configured to use the quantum superposition principle to search in the quantum domain and obtain multiple search samples, each search sample corresponds to the probability of the robot reaching the target point position, and the quantum search module is further configured to determine one or more target search samples from the multiple search samples according to the probability of the robot reaching the target point position;

[0125] A basic reinforcement learning model is preset in the reinforcement learning module, and the reinforcement learning module is configured to train the basic reinforcement learning model using the quantum search model and the target search sample, and determine that the trained reinforcement learning model is a robot path planning model.

[0126] In one embodiment of the present invention, a computer-readable storage medium is provided for storing program instructions, wherein the program instructions are configured to be called by a processor to execute the steps of the method described in any of the above embodiments.

[0127] It should be noted that the above-mentioned robot path planning device, robot path planning model training device, and computer-readable storage medium embodiments belong to the same inventive concept as the robot path planning method embodiment based on quantum search and reinforcement learning. The entire content of the robot path planning method embodiment based on quantum search and reinforcement learning is incorporated into the robot path planning device, robot path planning model training device, and computer-readable storage medium embodiments by reference.

[0128] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0129] The above is only a specific implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A robot path planning method based on quantum search and reinforcement learning, characterized in that: The following steps are involved: Quantize the real environment and robot state, including dynamically mapping the real-world environment and robot state to the quantum domain to establish a quantum domain; Based on a preset quantum search model, the quantum superposition principle is used to search in the quantum domain and obtain multiple search samples, each of which corresponds to the probability that the robot reaches the target position; Determine one or more target search samples from a plurality of search samples according to the probability that the robot reaches the target position; A basic reinforcement learning model is predetermined, and the basic reinforcement learning model is trained using the quantum search model and the target search sample, and the trained reinforcement learning model is determined to be a robot path planning model.

2. The robot path planning method based on quantum search and reinforcement learning according to claim 1, characterized in that: In the process of quantizing the robot state, the robot state is encoded and represented as a quantum superposition state; Based on a preset quantum search model, the quantum superposition state is used to perform a parallel search in the quantum domain and obtain multiple search samples, wherein the parallel search is to control the robot to simultaneously explore multiple positions in the quantum domain.

3. The robot path planning method based on quantum search and reinforcement learning according to claim 1, characterized in that: Dynamically mapping the real-world environment and robot state to the quantum field to establish a quantum domain, wherein the quantum domain includes a quantized simulation environment, quantum state parameters, and quantum state initialization information; The method for establishing the quantized simulation environment comprises: based on the reachability of the robot in the real environment, encoding the environment so that each position in the environment has a unique binary code, the binary code corresponds to the orthogonal basis of the quantum code, and obtaining the quantized simulation environment; The quantum state parameters are obtained by mapping elements in the real-world reinforcement learning framework to the quantized simulation environment. The elements in the real-world reinforcement learning framework include the current state s t , current moment action a t , the next moment state s t+1 And rewards r ; The quantum state initialization information is obtained by mapping the initial position and target position of the robot in the real environment and the obstacle distribution information in the real environment into the quantized simulation environment.

4. The robot path planning method based on quantum search and reinforcement learning according to claim 3 is characterized in that: The reachability of the robot in the real environment is determined by the adjacency matrix and expressed as the following formula: ； in, A represents the reachability matrix of the robot in the real environment, A surround Represents the robot reachability matrix under the influence of the characteristics of the surrounding environment, A obs represents the robot reachability matrix under the influence of obstacle distribution, A robot represents the robot reachability matrix under the influence of robot settings, Represents the element-wise product operator.

5. The robot path planning method based on quantum search and reinforcement learning according to claim 3 is characterized in that: It also includes setting the probability value corresponding to the position of the robot at the initial moment to be greater than the probability values ​​corresponding to other positions; and / or, Set the exit node so that when the robot reaches the target position, the robot will transfer to the exit node with a 100% probability at the next moment.

6. The robot path planning method based on quantum search and reinforcement learning according to claim 1, characterized in that: Based on the open system master equation, the quantum random walk model and the classical random walk model are used to construct the quantum search model, which is expressed by the following formula: ； in, ρ represents the probability distribution of the robot state, t express t time, p represents the weight coefficient of the quantum random walk model, 1- p represents the weight coefficient of the classic random walk model, L QW ( ρ ) represents the quantum random walk model, L CRW ( ρ ) represents the classic random walk model, L target ( ρ ) indicates that when the robot state is at the target position, the robot state will be irreversibly transferred to the exit node at the next moment.

7. The robot path planning method based on quantum search and reinforcement learning according to claim 6, characterized in that: In the quantized simulation environment, the probability of the robot being located in each grid is given by the probability distribution ρ The diagonal elements of ρ ij It is indicated that i = j ; go through The parallel search at time, the probability of the robot reaching the target position in the quantized simulation environment p ( T ) is expressed as: ,in, n Represents all target locations in the environment.

8. The robot path planning method based on quantum search and reinforcement learning according to claim 1, characterized in that: Determining one or more target search samples from a plurality of search samples by comparing the probability of the robot reaching the target position includes: Determine that the search sample corresponding to the highest probability of the robot reaching the target position is the target search sample; or, Determine that the search samples corresponding to each probability that the robot reaches the target position is greater than a preset threshold as the target search samples; or, Sort the robots in descending order according to their probability of reaching the target location, and confirm the order before sorting. k The search sample corresponding to the probability is the target search sample. k ≥2 and k Is a positive integer.

9. The robot path planning method based on quantum search and reinforcement learning according to claim 1, characterized in that: The structure of the target search sample is ( s t_b , a t_b , s t+1_b , r b ), in, s t_b The current state of the target search sample in the quantized simulation environment, a t_b searching for the target sample corresponding to the current action of the robot in the quantized simulation environment, s t+1_b The next moment state of the target search sample in the quantized simulation environment, including the robot state and the environment state, r b The reward returned by the real environment corresponding to the target search sample.

10. The robot path planning method based on quantum search and reinforcement learning according to claim 1, characterized in that: The basic reinforcement learning model is trained using the quantum search model and the target search sample, comprising the following steps: Encapsulating the quantum search model and the target search sample into a quantum meditation function module, wherein the input of the quantum meditation function module is the current state, and the output is the optimal robot action; The quantum meditation function module is added to the basic reinforcement learning model as a robot action selection module, and the basic reinforcement learning model is trained with target search samples as a training set.

11. The robot path planning method based on quantum search and reinforcement learning according to claim 10, characterized in that: The optimal robot action is determined by: Based on the quantum search model, the robot traverses all possible actions in the quantum field with the current state as input, and obtains the probability that the robot can reach the target position when choosing different actions. According to the probability of reaching the target position corresponding to each robot action, the robot action with the highest probability of reaching the target position at the current moment is determined as the optimal robot action.

12. The robot path planning method based on quantum search and reinforcement learning according to any one of claims 1 to 11, characterized in that: The robot path planning model is used to obtain a planned path of the robot from an initial position to a target position.

13. A robot path planning device, characterized in that: The robot path planning device is provided with a robot path planning model; wherein the robot path planning model is obtained by: Quantize the real environment and robot state, including dynamically mapping the real-world environment and robot state to the quantum domain to establish a quantum domain; Based on a preset quantum search model, the quantum superposition principle is used to search in the quantum domain and obtain multiple search samples, each of which corresponds to the probability that the robot reaches the target position; Determine one or more target search samples from a plurality of search samples according to the probability that the robot reaches the target position; A basic reinforcement learning model is predetermined, and the basic reinforcement learning model is trained using the quantum search model and the target search sample, and the trained reinforcement learning model is determined to be a robot path planning model.

14. The robot path planning device according to claim 12, characterized in that: The robot path planning model is obtained based on the robot path planning method based on quantum search and reinforcement learning described in any one of claims 2-11.

15. A computer-readable storage medium for storing program instructions, characterized in that: The program instructions are configured to be called by a processor to execute the steps of the method according to any one of claims 1 to 12.

Images