Hierarchical strategy learning-based curling robot dynamic throwing method and system

Abstract

The invention provides a curling robot dynamic throwing method and system based on hierarchical strategy learning. According to the system, through information fusion of the track strategy generation module and the self-adaptive control module, a closed-loop structure from environment state sensing to action control is realized. Target position and track parameters are generated by the upper layer module, and ice surface parameters are extracted by the state estimation module and uniformly input into the control module to generate a control instruction. The mechanism improves the environmental adaptability, robustness and expandability of a control system, and is suitable for curling tasks and other autonomous trajectory control applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and robot motion control, and in particular relates to a curling robot dynamic throwing method and system based on hierarchical strategy learning. Background Art

[0002] Curling robots must complete autonomous throwing tasks in complex ice environments, and their control accuracy relies on trajectory planning and environmental adaptability. Conventional methods currently rely on static parameter configuration, which struggles to effectively handle changes in ice friction and dynamic interference, leading to throwing errors. Trajectory planning and motion control are often designed separately, making unified modeling difficult. This results in weak strategy generalization and difficulty adapting to diverse real-world scenarios. Summary of the Invention

[0003] This invention aims to address the significant trajectory deviations and weak strategy transfer capabilities of existing curling robots in complex environments. By proposing a dynamic throwing method and system for curling robots based on hierarchical strategy learning, the method comprises a trajectory strategy generation phase and an adaptive motion control phase. The upper layer generates target throwing points and trajectory control parameters based on environmental conditions, while the lower layer generates control instructions based on ice surface characteristics, improving the system's strategy generalization and throwing accuracy.

[0004] The present invention is achieved through the following technical solutions. The present invention proposes a dynamic throwing method for a curling robot based on hierarchical strategy learning. The method includes a trajectory strategy generation stage and an adaptive motion control stage. The upper layer generates the target throwing point and trajectory control parameters according to the environmental state, and the lower layer generates control instructions based on the characteristics of the ice surface.

[0005] Trajectory strategy generation stage: According to the current curling game scene state, the strategy network is used to output a structured action vector, including the target throwing point (x * ,y * ) and trajectory control parameters c={c1,c2,...,c m}, used to construct the trajectory function and guide the robot to complete strategic throwing; this stage adopts the deep reinforcement learning method based on SoftActor-Critic for training, which includes five sub-stages: state space modeling sub-stage, action space and trajectory modeling sub-stage, reward function design sub-stage, strategy optimization sub-stage and trajectory optimization and endpoint selection sub-stage. Through these five sub-stages, the end-to-end modeling of the trajectory strategy generation network is completed, providing the target throwing point and corresponding trajectory control parameters for the lower-level adaptive action control stage, realizing the upper-level decision-making function in the hierarchical structure;

[0006] Adaptive motion control stage: This stage receives the target throwing point and trajectory control parameters provided by the trajectory strategy generation stage, combines the ice surface state estimation results, and generates the throwing control instructions for actual execution. exec =[v,ω,θ] to adapt to different ice surface friction and disturbance conditions; this stage includes three sub-stages, namely the ice surface state estimation sub-stage, the throwing action generation sub-stage and the action distribution modeling sub-stage.

[0007] Furthermore, the state space modeling sub-stage is specifically as follows:

[0008] State space modeling of the curling scenario:

[0009] s={X self ,X opp ,μ,κ} (1)

[0010] Among them, s represents the global information set describing the current curling scene, including the curling information of one's own curling stone: the set of position coordinates and distance from the center point (x i ,y i ) is the coordinate of the i-th own curling stone, D i is the Euclidean distance from the i-th curling stone to the center of the base camp; and the other curling stone information: the set of position coordinates and distance center point (x j ,y j ) is the coordinate of the jth opponent's curling stone, d i is the Euclidean distance from the jth opposing curling stone to the center of the base camp; μ is the current ice surface friction coefficient, and κ is the loss coefficient of each collision.

[0011] Furthermore, the action space and trajectory modeling sub-stage is specifically as follows:

[0012] Structured Motion Vector:

[0013] a t =[x * ,y * ,c] (2)

[0014] Among them, (x * ,y * ) must satisfy the constraints r is the base camp radius, c is the trajectory control parameter, which is used to construct a polynomial trajectory curve. To describe the shape of the throwing trajectory curve, an m-order polynomial is used to model the trajectory curve:

[0015]

[0016] Among them, m is the order of trajectory fitting, c iis the curve control coefficient of the i-th order. The smoothness of the trajectory is controlled by constraining the second-order derivative of the curve to reduce the drastic changes and jitters of the trajectory.

[0017] Furthermore, the reward function design sub-stage is specifically as follows:

[0018] Trajectory strategy generation guides strategy learning by designing a reward function R, considering the following indicators:

[0019] Score bonus items:

[0020]

[0021] Among them, ω1 and ω2 are scoring weight coefficients, K is the number of curling stones on the team, and H is the number of curling stones on the opponent team, which encourages the team's curling stones to approach the home camp and inhibits the opponent's curling stones from approaching the home camp; α is the adjustment factor of the friction coefficient on the distance, and λ coll is the collision penalty weight, C represents the number of collisions; increasing μ will amplify the effective distance and thus reduce the score, while increasing κ or C will directly deduct points;

[0022] Target accuracy and trajectory smoothness terms:

[0023]

[0024] Among them, (x end ,y end ) is the final landing point of the trajectory, λ acc is the accuracy penalty coefficient, η is the trajectory smoothness weight coefficient;

[0025] Environmental adaptability items:

[0026] R env =-β|R score (μ+δμ)-R score (μ)|-β|R score (κ+δκ)-R score (κ)| (6)

[0027] Among them, β is the environmental adaptability weight, which measures the sensitivity of the strategy to environmental changes by making small perturbations δμ and δκ to μ and κ; the final reward function is:

[0028] R = tanh(R score +R accuracy +R smooth +R env ) (7).

[0029] Furthermore, the strategy optimization sub-stage is specifically as follows:

[0030] The policy network aims to generate an action vector a that meets the curling tactical goal. t, the action includes the target throwing point and its corresponding trajectory control parameters; since the output action provides a structural planning basis for trajectory generation, strategy optimization needs to take into account both the rationality of the trajectory structure and the maximization of the expected score; in state s t Next, according to the probability distribution π(a t |s t ) Sampling action, interaction to obtain reward r t and the next state s t+1 , using samples from the experience replay pool (s t ,a t ,r t ,s t+1 ) for training; the policy gradient loss function is:

[0031]

[0032] Among them, π(a t |s t ) is the action probability output by the policy network, Q φ (s t ,a t ) is the valuation function of the current state-action pair, and the high-value trajectory generation scheme is preferred;

[0033] The value function network uses the mean square error between the predicted value and the target value as the optimization target, and its loss function is:

[0034]

[0035] Among them, the target value is the TD target value, Q target is the target network for delayed update, γ is the discount coefficient, which is used to control the impact of future rewards on the current score valuation; action a t+1 ~π means that according to s from the current strategy t+1 The sampled data is combined with the trajectory structure to evaluate the potential score at the next moment.

[0036] Furthermore, the trajectory optimization and endpoint selection sub-stage is specifically as follows:

[0037] Receive throw target (x * ,y * ), calculate the end point accuracy loss L traj =||(x * ,y * )-(x end ,y end )|| 2 and trajectory smoothness loss The final optimization objective function is:

[0038]

[0039] Among them, η is the smoothness weight coefficient, argmin represents the optimal solution c for the trajectory control parameters. * , ensuring that the trajectory can reach (x * ,y * ) and smooth enough.

[0040] Furthermore, the ice surface state estimation sub-stage is specifically as follows:

[0041] Based on trial data A multi-layer perceptron encoder is used to model and predict the ice surface dynamics parameters p = [μ, k], where a i represents the actual control action in the i-th trial throw sample, and N is the number of trial throw samples. The model estimates the ice surface friction coefficient and disturbance factor by learning the state-action relationship in the trial throw data. By minimizing the deviation between the predicted value and the actual parameter, it achieves accurate perception of the ice surface environment state.

[0042] The throwing action generation sub-stage is specifically as follows:

[0043] The ice surface dynamics parameter p, trajectory control parameter c and throwing target position (x * ,y * ) Input the policy network to generate the initial throwing action a pred =[v,ω,θ]; During the training process, the loss function consists of two parts: the action prediction error and the action smoothness constraint; the prediction error term is defined as where π ψ Represents the policy network based on state s i Output action a i The probability of ; smoothness constraint where a t represents the action sequence generated by the policy network at consecutive time steps; the total loss is:

[0044] L=L action +λL smooth (11)

[0045] Among them, λ is a hyperparameter used to balance the relationship between action prediction error and action smoothness;

[0046] The action distribution modeling sub-stage is specifically as follows:

[0047] The initial throwing action a pred Modeled as a Gaussian mixture distribution to improve the smoothness and robustness of the policy output; based on the GMM decoder, the action probability density function is constructed and the final executable control instructions are sampled from it:

[0048] a exec~P(a)(12)

[0049] Where P(a) is the probability density function of Gaussian mixture distribution.

[0050] The present invention also proposes a curling robot dynamic throwing system based on hierarchical strategy learning. The system includes a trajectory strategy generation module and an adaptive motion control module. The upper layer generates the target throwing point and trajectory control parameters according to the environmental state, and the lower layer generates control instructions based on the ice surface characteristics.

[0051] Trajectory strategy generation module: Based on the current curling game scene state, the strategy network is used to output a structured action vector, including the target throwing point (x * ,y * ) and trajectory control parameters c={c1,c2,...,c m}, used to construct trajectory functions and guide the robot to complete strategic throwing; this module is trained using a deep reinforcement learning method based on SoftActor-Critic and includes five submodules: state space modeling submodule, action space and trajectory modeling submodule, reward function design submodule, strategy optimization submodule, and trajectory optimization and endpoint selection submodule. Through these five submodules, the end-to-end modeling of the trajectory strategy generation network is completed, providing the target throwing point and corresponding trajectory control parameters for the lower-level adaptive action control stage, realizing the upper-level decision-making function in the hierarchical structure;

[0052] Adaptive motion control module: This module receives the target throwing point and trajectory control parameters provided by the trajectory strategy generation module, combines the ice surface state estimation results, and generates the throwing control instructions for actual execution. exec =[v,ω,θ] to adapt to different ice surface friction and disturbance conditions; this module includes three sub-modules, namely the ice surface state estimation sub-module, the throwing action generation sub-module and the action distribution modeling sub-module.

[0053] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the dynamic throwing method of a curling robot based on hierarchical strategy learning are implemented.

[0054] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the curling robot dynamic throwing method based on hierarchical strategy learning.

[0055] Beneficial effects of the present invention:

[0056] The system described in this paper achieves a closed-loop structure from environmental state perception to motion control through information fusion between the trajectory strategy generation module and the adaptive control module. Target position and trajectory parameters are generated by the upper-level module, while ice surface parameters are extracted by the state estimation module. These are then uniformly input into the control module to generate control instructions. This mechanism improves the control system's environmental adaptability, robustness, and scalability, making it suitable for curling tasks and other autonomous trajectory control applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0058] Figure 1 This is the overall architecture diagram of the curling robot dynamic throwing system based on hierarchical strategy learning described in the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0060] Specifically, combined Figure 1 The present invention proposes a dynamic throwing method for a curling robot based on hierarchical strategy learning. The method includes a trajectory strategy generation stage and an adaptive motion control stage. The upper layer generates the target throwing point and trajectory control parameters according to the environmental state, and the lower layer generates control instructions based on the ice surface characteristics.

[0061] Trajectory strategy generation stage: According to the current curling game scene state, the strategy network is used to output a structured action vector, including the target throwing point (x * ,y * ) and trajectory control parameters c={c1,c2,...,c m}, used to construct the trajectory function and guide the robot to complete strategic throwing; this stage adopts the deep reinforcement learning method based on SoftActor-Critic for training, which includes five sub-stages: state space modeling sub-stage, action space and trajectory modeling sub-stage, reward function design sub-stage, strategy optimization sub-stage and trajectory optimization and endpoint selection sub-stage. Through these five sub-stages, the end-to-end modeling of the trajectory strategy generation network is completed, providing the target throwing point and corresponding trajectory control parameters for the lower-level adaptive action control stage, realizing the upper-level decision-making function in the hierarchical structure;

[0062] The state space modeling sub-stage is specifically as follows:

[0063] State space modeling of the curling scenario:

[0064] s={X self ,X opp ,μ,κ} (1)

[0065] Among them, s represents the global information set describing the current curling scene, including the curling information of one's own curling stone: the set of position coordinates and distance from the center point (x i ,y i ) is the coordinate of the i-th own curling stone, D i is the Euclidean distance from the i-th curling stone to the center of the base camp; and the other curling stone information: the set of position coordinates and distance center point (x j ,y j ) is the coordinate of the jth opponent's curling stone, d i is the Euclidean distance from the jth opposing curling stone to the center of the base camp; μ is the current ice surface friction coefficient, and κ is the loss coefficient of each collision.

[0066] The action space and trajectory modeling sub-stage is specifically as follows:

[0067] Structured Motion Vector:

[0068] a t =[x * ,y * ,c] (2)

[0069] Among them, (x * ,y * ) must satisfy the constraints r is the base camp radius, c is the trajectory control parameter, which is used to construct a polynomial trajectory curve. To describe the shape of the throwing trajectory curve, an m-order polynomial is used to model the trajectory curve:

[0070]

[0071] Among them, m is the order of trajectory fitting, c i is the curve control coefficient of the i-th order. The smoothness of the trajectory is controlled by constraining the second-order derivative of the curve to reduce the drastic changes and jitters of the trajectory.

[0072] The reward function design sub-stage is specifically as follows:

[0073] Trajectory strategy generation guides strategy learning by designing a reward function R, considering the following indicators:

[0074] Score bonus items:

[0075]

[0076] Among them, ω1 and ω2 are scoring weight coefficients, K is the number of curling stones on the team, and H is the number of curling stones on the opponent team, which encourages the team's curling stones to approach the home camp and inhibits the opponent's curling stones from approaching the home camp; α is the adjustment factor of the friction coefficient on the distance, and λ coll is the collision penalty weight, C represents the number of collisions; increasing μ will amplify the effective distance and thus reduce the score, while increasing κ or C will directly deduct points;

[0077] Target accuracy and trajectory smoothness terms:

[0078]

[0079] Among them, (x end ,y end ) is the final landing point of the trajectory, λ acc is the accuracy penalty coefficient, η is the trajectory smoothness weight coefficient;

[0080] Environmental adaptability items:

[0081] R env =-β|R score (μ+δμ)-R score (μ)|-β|R score (κ+δκ)-R score (κ)| (6)

[0082] Among them, β is the environmental adaptability weight, which measures the sensitivity of the strategy to environmental changes by making small perturbations δμ and δκ to μ and κ; the final reward function is:

[0083] R = tanh(R score +R accuracy +R smooth +R env ) (7).

[0084] The strategy optimization sub-stage is specifically as follows:

[0085] The policy network aims to generate an action vector a that meets the curling tactical goal. t , the action includes the target throwing point and its corresponding trajectory control parameters; since the output action provides a structural planning basis for trajectory generation, strategy optimization needs to take into account both the rationality of the trajectory structure and the maximization of the expected score; in state s t Next, according to the probability distribution π(a t |s t ) Sampling action, interaction to obtain reward r t and the next state s t+1 , using samples from the experience replay pool (s t ,a t ,r t ,s t+1 ) for training; the policy gradient loss function is:

[0086]

[0087] Among them, π(a t |s t ) is the action probability output by the policy network, Q φ (s t ,a t ) is the valuation function of the current state-action pair, and the high-value trajectory generation scheme is preferred;

[0088] The value function network uses the mean square error between the predicted value and the target value as the optimization target, and its loss function is:

[0089]

[0090] Among them, the target value is the TD target value, Q target is the target network for delayed update, γ is the discount coefficient, which is used to control the impact of future rewards on the current score valuation; action a t+1 ~π means that according to s from the current strategy t+1 The sampled data is combined with the trajectory structure to evaluate the potential score at the next moment.

[0091] The trajectory optimization and endpoint selection sub-stages are specifically as follows:

[0092] Receive throw target (x * ,y * ), calculate the end point accuracy loss L traj =||(x * ,y * )-(x end ,y end )|| 2 and trajectory smoothness loss The final optimization objective function is:

[0093]

[0094] Among them, η is the smoothness weight coefficient, argmin represents the optimal solution c for the trajectory control parameters. * , ensuring that the trajectory can reach (x * ,y * ) and smooth enough.

[0095] Adaptive motion control stage: The adaptive motion control stage aims to build a control model that integrates environmental state perception and throwing motion generation based on curling trial throw data. This stage receives the target throwing point and trajectory control parameters provided by the trajectory strategy generation stage, combines them with the ice surface state estimation results, and generates the throwing control instructions for actual execution. exec =[v,ω,θ] to adapt to different ice surface friction and disturbance conditions; this stage includes three sub-stages, namely the ice surface state estimation sub-stage, the throwing action generation sub-stage and the action distribution modeling sub-stage.

[0096] The ice surface state estimation sub-stage is specifically as follows:

[0097] Based on trial data A multi-layer perceptron encoder is used to model and predict the ice surface dynamics parameters p = [μ, k], where a i represents the actual control action in the i-th trial throw sample, and N is the number of trial throw samples. The model estimates the ice surface friction coefficient and disturbance factor by learning the state-action relationship in the trial throw data. By minimizing the deviation between the predicted value and the actual parameter, it achieves accurate perception of the ice surface environment state.

[0098] The throwing action generation sub-stage is specifically as follows:

[0099] The ice surface dynamics parameter p, trajectory control parameter c and throwing target position (x * ,y * ) Input the policy network to generate the initial throwing action a pred =[v,ω,θ]; During the training process, the loss function consists of two parts: the action prediction error and the action smoothness constraint; the prediction error term is defined as where π ψ Represents the policy network based on state s i Output action a i The probability of ; smoothness constraint where a t represents the action sequence generated by the policy network at consecutive time steps; the total loss is:

[0100] L=L action +λL smooth (11)

[0101] Among them, λ is a hyperparameter used to balance the relationship between action prediction error and action smoothness;

[0102] The action distribution modeling sub-stage is specifically as follows:

[0103] The initial throwing action a pred Modeled as a Gaussian mixture distribution to improve the smoothness and robustness of the policy output; based on the GMM decoder, the action probability density function is constructed and the final executable control instructions are sampled from it:

[0104] a exec ~P(a)(12)

[0105] Where P(a) is the probability density function of Gaussian mixture distribution.

[0106] The present invention also proposes a dynamic throwing system for a curling robot based on hierarchical strategy learning. The system includes a trajectory strategy generation module and an adaptive action control module. The upper layer generates a target throwing point and trajectory control parameters according to the environmental state, and the lower layer generates control instructions based on the characteristics of the ice surface. In the training phase, trial throwing data is input, and the ice surface parameters p are estimated through a multi-layer perceptron encoder network. The trajectory module generates the target position and trajectory parameters, and the control module generates a predicted action a based on the above information. pred =[v,ω,θ], and then sampled by Gaussian mixture model to output the final control instruction a exec ~P(a). Specifically:

[0107] Trajectory strategy generation module: Based on the current curling game scene state, the strategy network is used to output a structured action vector, including the target throwing point (x * ,y * ) and trajectory control parameters c={c1,c2,...,c m}, used to construct trajectory functions and guide the robot to complete strategic throwing; this module is trained using a deep reinforcement learning method based on SoftActor-Critic and includes five submodules: state space modeling submodule, action space and trajectory modeling submodule, reward function design submodule, strategy optimization submodule, and trajectory optimization and endpoint selection submodule. Through these five submodules, the end-to-end modeling of the trajectory strategy generation network is completed, providing the target throwing point and corresponding trajectory control parameters for the lower-level adaptive action control stage, realizing the upper-level decision-making function in the hierarchical structure;

[0108] Adaptive motion control module: This module receives the target throwing point and trajectory control parameters provided by the trajectory strategy generation module, combines the ice surface state estimation results, and generates the throwing control instructions for actual execution. exec=[v,ω,θ] to adapt to different ice surface friction and disturbance conditions; this module includes three sub-modules, namely the ice surface state estimation sub-module, the throwing action generation sub-module and the action distribution modeling sub-module.

[0109] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the dynamic throwing method of a curling robot based on hierarchical strategy learning are implemented.

[0110] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the curling robot dynamic throwing method based on hierarchical strategy learning.

[0111] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0112] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0113] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0114] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0115] The above is a detailed introduction to the dynamic throwing method and system of a curling robot based on hierarchical strategy learning proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A dynamic throwing method for a curling robot based on hierarchical strategy learning, characterized in that: The method includes a trajectory strategy generation stage and an adaptive action control stage; the upper layer generates the target throwing point and trajectory control parameters according to the environmental state, and the lower layer generates control instructions based on the ice surface characteristics; Trajectory strategy generation stage: According to the current curling game scene state, the strategy network is used to output a structured action vector, including the target throwing point (x * ,y * ) and trajectory control parameters c={c1,c2,...,c m }, used to construct the trajectory function and guide the robot to complete strategic throwing; this stage adopts the deep reinforcement learning method based on SoftActor-Critic for training, which includes five sub-stages: state space modeling sub-stage, action space and trajectory modeling sub-stage, reward function design sub-stage, strategy optimization sub-stage and trajectory optimization and endpoint selection sub-stage. Through these five sub-stages, the end-to-end modeling of the trajectory strategy generation network is completed, providing the target throwing point and corresponding trajectory control parameters for the lower-level adaptive action control stage, realizing the upper-level decision-making function in the hierarchical structure; Adaptive motion control stage: This stage receives the target throwing point and trajectory control parameters provided by the trajectory strategy generation stage, combines the ice surface state estimation results, and generates the throwing control instructions for actual execution. exec =[v,ω,θ] to adapt to different ice surface friction and disturbance conditions; this stage includes three sub-stages, namely the ice surface state estimation sub-stage, the throwing action generation sub-stage and the action distribution modeling sub-stage.

2. The method according to claim 1, characterized in that The state space modeling sub-stage is specifically as follows: State space modeling of the curling scenario: s={X self ,X opp ,m,k} (1) Among them, s represents the global information set describing the current curling scene, including the curling information of one's own curling stone: the set of position coordinates and distance from the center point (x i ,y i ) is the coordinate of the i-th own curling stone, D i is the Euclidean distance from the i-th curling stone to the center of the base camp; and the other curling stone information: the set of position coordinates and distance center point (x j ,y j ) is the coordinate of the jth opponent's curling stone, d i is the Euclidean distance from the jth opposing curling stone to the center of the base camp; μ is the current ice surface friction coefficient, and κ is the loss coefficient of each collision.

3. The method according to claim 2, characterized in that The action space and trajectory modeling sub-stage is specifically as follows: Structured Motion Vector: a t =[x * ,y * ,c] (2) Among them, (x * ,y * ) must satisfy the constraints r is the base camp radius, c is the trajectory control parameter, which is used to construct a polynomial trajectory curve. To describe the shape of the throwing trajectory curve, an m-order polynomial is used to model the trajectory curve: Among them, m is the order of trajectory fitting, c i is the curve control coefficient of the i-th order. The smoothness of the trajectory is controlled by constraining the second-order derivative of the curve to reduce the drastic changes and jitters of the trajectory.

4. The method according to claim 3, characterized in that The reward function design sub-stage is specifically as follows: Trajectory strategy generation guides strategy learning by designing a reward function R, considering the following indicators: Score bonus items: Among them, ω1 and ω2 are scoring weight coefficients, K is the number of curling stones on the team, and H is the number of curling stones on the opponent team, which encourages the team's curling stones to approach the home camp and inhibits the opponent's curling stones from approaching the home camp; α is the adjustment factor of the friction coefficient on the distance, and λ coll is the collision penalty weight, C represents the number of collisions; increasing μ will amplify the effective distance and thus reduce the score, while increasing κ or C will directly deduct points; Target accuracy and trajectory smoothness terms: Among them, (x end ,y end ) is the final landing point of the trajectory, λ acc is the accuracy penalty coefficient, η is the trajectory smoothness weight coefficient; Environmental adaptability items: R env =-β|R score (μ+σμ)-R score (μ)|-β|R score (k+dk)-R score (k)| (6) Among them, β is the environmental adaptability weight, which measures the sensitivity of the strategy to environmental changes by making small perturbations δμ and δκ to μ and κ; the final reward function is: R=tanh(R score +R accuracy +R smooth +R env ) (7)。 5. The method according to claim 4, characterized in that The strategy optimization sub-stage is specifically as follows: The policy network aims to generate an action vector a that meets the curling tactical goal. t , the action includes the target throwing point and its corresponding trajectory control parameters; since the output action provides a structural planning basis for trajectory generation, strategy optimization needs to take into account both the rationality of the trajectory structure and the maximization of the expected score; in state s t Next, according to the probability distribution π(a t |s t ) Sampling action, interaction to obtain reward r t and the next state s t+1 , using samples from the experience replay pool (s t ,a t ,r t ,s t+1 ) for training; the policy gradient loss function is: Among them, π(a t |s t ) is the action probability output by the policy network, Q φ (s t ,a t ) is the valuation function of the current state-action pair, and the high-value trajectory generation scheme is preferred; The value function network uses the mean square error between the predicted value and the target value as the optimization target, and its loss function is: Among them, the target value is the TD target value, Q target is the target network for delayed update, γ is the discount coefficient, which is used to control the impact of future rewards on the current score valuation; action a t+1 ~π means that according to s from the current strategy t+1 The sampled data is combined with the trajectory structure to evaluate the potential score at the next moment.

6. The method according to claim 5, characterized in that The trajectory optimization and endpoint selection sub-stages are specifically as follows: Receive throw target (x * ,y * ), calculate the end point accuracy loss L traj =||(x * ,y * )-(x end ,y end )|| 2 and trajectory smoothness loss The final optimization objective function is: Among them, η is the smoothness weight coefficient, argmin represents the optimal solution c for the trajectory control parameters. * , ensuring that the trajectory can reach (x * ,y * ) and smooth enough.

7. The method according to claim 6, characterized in that The ice surface state estimation sub-stage is specifically as follows: Based on trial data A multi-layer perceptron encoder is used to model and predict the ice surface dynamics parameters p = [μ, k], where a i represents the actual control action in the i-th trial throw sample, and N is the number of trial throw samples. The model estimates the ice surface friction coefficient and disturbance factor by learning the state-action relationship in the trial throw data. By minimizing the deviation between the predicted value and the actual parameter, it achieves accurate perception of the ice surface environment state. The throwing action generation sub-stage is specifically as follows: The ice surface dynamics parameter p, trajectory control parameter c and throwing target position (x * ,y * ) Input the policy network to generate the initial throwing action a pred =[v,ω,θ]; During the training process, the loss function consists of two parts: the action prediction error and the action smoothness constraint; the prediction error term is defined as where π ψ Represents the policy network based on state s i Output action a i The probability of ; smoothness constraint where a t represents the action sequence generated by the policy network at consecutive time steps; the total loss is: L=L action +λL smooth (11) Among them, λ is a hyperparameter used to balance the relationship between action prediction error and action smoothness; The action distribution modeling sub-stage is specifically as follows: The initial throwing action a pred Modeled as a Gaussian mixture distribution to improve the smoothness and robustness of the policy output; based on the GMM decoder, the action probability density function is constructed and the final executable control instructions are sampled from it: a exec ~P(a)(12) Where P(a) is the probability density function of Gaussian mixture distribution.

8. A curling robot dynamic throwing system based on hierarchical strategy learning, characterized in that: The system includes a trajectory strategy generation module and an adaptive motion control module; the upper layer generates the target throwing point and trajectory control parameters according to the environmental state, and the lower layer generates control instructions based on the ice surface characteristics; Trajectory strategy generation module: Based on the current curling game scene state, the strategy network is used to output a structured action vector, including the target throwing point (x * ,y * ) and trajectory control parameters c={c1,c2,...,c m }, used to construct trajectory functions and guide the robot to complete strategic throwing; this module is trained using a deep reinforcement learning method based on SoftActor-Critic and includes five submodules: state space modeling submodule, action space and trajectory modeling submodule, reward function design submodule, strategy optimization submodule, and trajectory optimization and endpoint selection submodule. Through these five submodules, the end-to-end modeling of the trajectory strategy generation network is completed, providing the target throwing point and corresponding trajectory control parameters for the lower-level adaptive action control stage, realizing the upper-level decision-making function in the hierarchical structure; Adaptive motion control module: This module receives the target throwing point and trajectory control parameters provided by the trajectory strategy generation module, combines the ice surface state estimation results, and generates the throwing control instructions for actual execution. exec =[v,ω,θ] to adapt to different ice surface friction and disturbance conditions; this module includes three sub-modules, namely the ice surface state estimation sub-module, the throwing action generation sub-module and the action distribution modeling sub-module.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images