Trajectory generation method and device, robot, and storage medium

Abstract

The present disclosure relates to a trajectory generation method and device, a robot and a storage medium. The method is applied to a robot and includes: generating a plurality of performance combinations according to a plurality of performance positions of the robot and a plurality of performed instruments, wherein each performance combination includes one performance position and at least one performed instrument; for each performance combination in the plurality of performance combinations, determining a corresponding joint motion trajectory of the performance combination according to a position of each performed instrument in the performance combination and a contact speed of the performance position in the performance combination and each performed instrument, wherein the joint motion trajectory includes an angular displacement of a joint at each sampling point; and generating a performance trajectory library of the robot according to the corresponding joint motion trajectory of each performance combination in the plurality of performance combinations.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, specifically to a trajectory generation method, apparatus, robot, and storage medium. Background Technology

[0002] In recent years, robotics technology has been continuously developing, becoming increasingly intelligent and automated, with improvements in the richness, stability, and flexibility of its movements. Current robots can not only perform repetitive, periodic actions such as sweeping and walking, but they can also play musical instruments; for example, a robot can perform a drum set performance. However, when robots perform drum set performances, issues such as slow and incomplete movements persist, meaning the performance quality still needs improvement. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, this disclosure provides a trajectory generation method, apparatus, robot, and storage medium to solve the defects in the related technologies.

[0004] According to a first aspect of the present disclosure, a trajectory generation method is provided, applied to a robot, the method comprising:

[0005] Based on the robot's multiple playing parts and multiple instruments being played, multiple playing combinations are generated, wherein each playing combination includes a playing part and at least one instrument being played.

[0006] For each of the plurality of performance combinations, the joint motion trajectory corresponding to the performance combination is determined based on the position of each instrument played in the performance combination and the contact speed between the playing part in the performance combination and each instrument played. The joint motion trajectory includes the angular displacement of the joint at each sampling point.

[0007] A library of robot performance trajectories is generated based on the joint motion trajectories corresponding to each of the multiple performance combinations.

[0008] In one embodiment, generating multiple performance combinations based on multiple playing parts of the robot and multiple instruments being played includes:

[0009] Each of the plurality of playing parts is combined with each of the plurality of playing instruments to form a playing combination, wherein the playing instruments in the playing combination serve as the starting point and the ending point of the movement trajectory of the playing part.

[0010] Each of the plurality of playing parts is combined with every two of the plurality of playing instruments to form a playing combination, wherein one of the playing instruments in the playing combination serves as the starting point of the movement trajectory of the playing part, and the other playing instrument serves as the ending point of the movement trajectory of the playing part.

[0011] In one embodiment, determining the joint motion trajectory corresponding to the performance combination based on the position of each instrument played within the performance combination and the contact speed between the playing part and each instrument played within the performance combination includes:

[0012] Under at least one constraint, the position where the speed of the playing part in the performance combination is 0 is obtained by minimizing the weighted sum of the distance differences in at least one direction, wherein the at least one constraint is related to the distance differences in at least one direction;

[0013] Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the motion trajectory of the playing part in the performance combination is determined, wherein the motion trajectory includes the position of the playing part at each sampling point.

[0014] Based on the movement trajectory of the playing parts within the playing combination, the joint movement trajectory corresponding to the playing combination is determined.

[0015] In one embodiment, when the performance combination includes a single instrument being played, the at least one constraint includes at least one of the following:

[0016] The vertical distance difference is the vertical distance between the position where the playing part of the performance combination has a speed of 0 and the position of any instrument played in the performance combination.

[0017] The horizontal distance difference is the horizontal distance between the position where the playing part of the performance is at 0 speed and the boundary of any instrument played in the performance combination.

[0018] A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination;

[0019] The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

[0020] In one embodiment, when the performance combination includes two instruments being played, the at least one constraint includes at least one of the following:

[0021] The vertical distance difference is the vertical distance between the position where the playing part of the performance combination has a speed of 0 and the position of any instrument played in the performance combination.

[0022] A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination;

[0023] The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

[0024] In one embodiment, determining the motion trajectory of the playing part within the performance combination based on the position of each instrument played in the performance combination, the contact velocity between the playing part and each instrument played in the performance combination, and the position where the velocity of the playing part in the performance combination is 0, includes:

[0025] Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the sub-trajectory of the playing part from the starting point to the position where the speed is 0, and the sub-trajectory of the playing part from the position where the speed is 0 to the ending point are determined.

[0026] In one embodiment, it also includes:

[0027] Based on the movement trajectory of the playing part within the performance combination, the velocity trajectory of the playing part is determined, wherein the velocity trajectory includes the velocity of the playing part at each sampling moment, and the sampling moment is characterized by the performance frequency;

[0028] Under at least one of the following constraints, the upper limit of the playing frequency of the playing combination is obtained by maximizing the playing frequency using the velocity trajectory of the playing part: the Jacobian matrix used to describe the mapping relationship between the velocity trajectory of the playing part and the joint velocity trajectory corresponding to the playing combination; and the angular velocity of the joint at each sampling point in the joint motion trajectory is within the range of joint angular velocity.

[0029] In one embodiment, it also includes:

[0030] From the performance trajectory library, obtain the joint motion trajectory of each performance event in the music file, wherein each performance event includes the performance combination and performance duration;

[0031] Based on the joint motion trajectory of each performance event in the music file, the robot is controlled to perform the performance actions of the music file.

[0032] In one embodiment, it also includes:

[0033] For each performance event in the music file, if the performance duration is greater than the duration corresponding to the upper limit of the performance frequency of the performance combination, the performance duration is corrected.

[0034] In one embodiment, the plurality of playing parts include the robot's limbs; and / or, the plurality of instruments being played include a plurality of drums and cymbals from a drum kit.

[0035] According to a second aspect of the present disclosure, a trajectory generation apparatus is provided for use in a robot, the apparatus comprising:

[0036] The combination module is used to generate multiple performance combinations based on the robot's multiple playing parts and multiple instruments being played, wherein each performance combination includes a playing part and at least one instrument being played.

[0037] The trajectory module is used to determine the joint motion trajectory corresponding to each of the multiple performance combinations based on the position of each instrument played in the performance combination and the contact speed between the playing part and each instrument played in the performance combination. The joint motion trajectory includes the angular displacement of the joint at each sampling point.

[0038] The storage module is used to generate a library of robot performance trajectories based on the joint motion trajectories corresponding to each of the multiple performance combinations.

[0039] In one embodiment, the combining module is specifically used for:

[0040] Each of the plurality of playing parts is combined with each of the plurality of playing instruments to form a playing combination, wherein the playing instruments in the playing combination serve as the starting point and the ending point of the movement trajectory of the playing part.

[0041] Each of the plurality of playing parts is combined with every two of the plurality of playing instruments to form a playing combination, wherein one of the playing instruments in the playing combination serves as the starting point of the movement trajectory of the playing part, and the other playing instrument serves as the ending point of the movement trajectory of the playing part.

[0042] In one embodiment, the trajectory module is specifically used for:

[0043] Under at least one constraint, the position where the speed of the playing part in the performance combination is 0 is obtained by minimizing the weighted sum of the distance differences in at least one direction, wherein the at least one constraint is related to the distance differences in at least one direction;

[0044] Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the motion trajectory of the playing part in the performance combination is determined, wherein the motion trajectory includes the position of the playing part at each sampling point.

[0045] Based on the movement trajectory of the playing parts within the playing combination, the joint movement trajectory corresponding to the playing combination is determined.

[0046] In one embodiment, when the performance combination includes a single instrument being played, the at least one constraint includes at least one of the following:

[0047] The vertical distance difference is the vertical distance between the position where the playing part of the performance combination has a speed of 0 and the position of any instrument played in the performance combination.

[0048] The horizontal distance difference is the horizontal distance between the position where the playing part of the performance is at 0 speed and the boundary of any instrument played in the performance combination.

[0049] A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination;

[0050] The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

[0051] In one embodiment, when the performance combination includes two instruments being played, the at least one constraint includes at least one of the following:

[0052] The vertical distance difference is the vertical distance between the position where the playing part of the performance combination has a speed of 0 and the position of any instrument played in the performance combination.

[0053] A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination;

[0054] The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

[0055] In one embodiment, when the trajectory module is used to determine the motion trajectory of the playing part within the performance combination based on the position of each instrument played in the performance combination, the contact velocity between the playing part and each instrument played in the performance combination, and the position where the velocity of the playing part is 0 in the performance combination, it is specifically used for:

[0056] Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the sub-trajectory of the playing part from the starting point to the position where the speed is 0, and the sub-trajectory of the playing part from the position where the speed is 0 to the ending point are determined.

[0057] In one embodiment, a frequency module is further included, for:

[0058] Based on the movement trajectory of the playing part within the performance combination, the velocity trajectory of the playing part is determined, wherein the velocity trajectory includes the velocity of the playing part at each sampling moment, and the sampling moment is characterized by the performance frequency;

[0059] Under at least one of the following constraints, the upper limit of the playing frequency of the playing combination is obtained by maximizing the playing frequency using the velocity trajectory of the playing part: the Jacobian matrix used to describe the mapping relationship between the velocity trajectory of the playing part and the joint velocity trajectory corresponding to the playing combination; and the angular velocity of the joint at each sampling point in the joint motion trajectory is within the range of joint angular velocity.

[0060] In one embodiment, a performance module is also included, for:

[0061] From the performance trajectory library, obtain the joint motion trajectory of each performance event in the music file, wherein each performance event includes the performance combination and performance duration;

[0062] Based on the joint motion trajectory of each performance event in the music file, the robot is controlled to perform the performance actions of the music file.

[0063] In one embodiment, a correction module is further included, for:

[0064] For each performance event in the music file, if the performance duration is greater than the duration corresponding to the upper limit of the performance frequency of the performance combination, the performance duration is corrected.

[0065] In one embodiment, the plurality of playing parts include the robot's limbs; and / or, the plurality of instruments being played include a plurality of drums and cymbals from a drum kit.

[0066] According to a third aspect of the present disclosure, a robot is provided, the robot including a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the trajectory generation method described in the first aspect when executing the computer instructions.

[0067] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0068] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0069] The trajectory generation method provided in this disclosure first generates multiple performance combinations based on multiple playing parts of the robot and multiple instruments being played. Then, for each of the multiple performance combinations, the joint motion trajectory corresponding to the performance combination is determined based on the position of each instrument being played within the performance combination and the contact speed between the playing part and each instrument being played within the performance combination. Finally, a performance trajectory library for the robot is generated based on the joint motion trajectories corresponding to each of the multiple performance combinations. Since multiple performance combinations can cover various combinations of playing parts and instruments being played, the performance trajectory library constructed by this method can provide joint motion trajectories for various movements of the robot when playing music. In other words, when the robot plays instruments such as drums based on the performance trajectory library constructed by this method, it can perform more naturally, smoothly, and accurately, making the movements more in line with real human performance scenarios, avoiding problems such as slow movements and inaccurate movements, and significantly improving the performance effect. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0071] Figure 1 This is a flowchart illustrating a trajectory generation method according to an exemplary embodiment of this disclosure;

[0072] Figure 2 This is a schematic diagram illustrating a scenario of a robot playing a drum kit, as shown in an exemplary embodiment of this disclosure;

[0073] Figure 3 This is a schematic diagram of the structure of a drum kit shown in an exemplary embodiment of this disclosure;

[0074] Figure 4 This is a flowchart illustrating a method for determining joint motion trajectories according to an exemplary embodiment of this disclosure;

[0075] Figure 5 This is a schematic diagram illustrating the motion trajectory of the playing part of a robot when it straddles a drum, as shown in an exemplary embodiment of this disclosure;

[0076] Figure 6 This is a schematic diagram of the structure of a trajectory generation apparatus shown in an exemplary embodiment of the present disclosure;

[0077] Figure 7 This is a structural block diagram of a robot illustrated in an exemplary embodiment of this disclosure. Detailed Implementation

[0078] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0079] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0080] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0081] In recent years, robotics technology has been continuously developing, becoming increasingly intelligent and automated, with improvements in the richness, stability, and flexibility of its movements. Current robots can not only perform repetitive, periodic actions such as sweeping and walking, but they can also play musical instruments; for example, a robot can perform a drum set performance. However, when robots perform drum set performances, issues such as slow and incomplete movements persist, meaning the performance quality still needs improvement.

[0082] Based on this, in a first aspect, at least one embodiment of this disclosure provides a trajectory generation method, please refer to the appendix. Figure 1 It illustrates the process of the method, including steps S101 to S103.

[0083] This method can be applied to robots, such as legged robots. The robot may have a torso and limbs. In one possible embodiment, please refer to the appendix. Figure 2 and attached Figure 3This method can be applied to scenarios where robots play drums; where a drum set includes multiple instruments, such as... Figure 3 The robot includes a loud cymbal 1, a nail cymbal 2, a drum sound source 3, a snare drum 4, a tom-tom 5, a tom-tom 6, a tom-tom 7, a hi-hat 8, and a bass drum 9. The robot is a general-purpose robot, not a dedicated robot for playing the drum kit. The robot's lower limbs have 10 degrees of freedom, and its upper limbs also have 10 degrees of freedom, enabling it to perform multiple functions such as walking, jumping, picking up and placing objects, and dancing.

[0084] In step S101, multiple performance combinations are generated based on the robot's multiple playing parts and multiple instruments being played, wherein each performance combination includes a playing part and at least one instrument being played.

[0085] The multiple playing parts may include the robot's limbs, namely the left arm, right arm, left foot, and right foot; the multiple instruments to be played include multiple drums, cymbals, etc. of a drum kit.

[0086] For example, each of the plurality of playing parts is combined with each of the plurality of played instruments (within the range of motion of the playing part) to form a playing combination, wherein the played instruments in the playing combination serve as the start and end points of the movement trajectory of the playing part. In this example, the starting and ending points of the movement trajectory of the playing part are the same played instrument, that is, the playing combination represents the robot continuously playing the same played instrument, for example, in the scenario of the robot playing a drum kit, the robot continuously hits the same drum.

[0087] As another example, each of the plurality of playing parts is paired with every two instruments (within the range of motion of the playing part) to generate a playing combination. In this playing combination, one instrument serves as the starting point of the playing part's trajectory, and the other instrument serves as the ending point. In this example, the starting and ending points of the playing part's trajectory are different instruments, meaning the playing combination represents the robot switching between different instruments. For example, in a scenario where the robot plays a drum kit, it straddles different drums to play.

[0088] In the scenario where the robot is playing the drums, the robot's left and right arms and left and right feet are numbered arm_id = 1 to 4, respectively, and the drums on the drum set are numbered drum_id = 1 to 9 from top to bottom. The performance combination is represented by beat = {arm_id, drum_id1, drum_id2}, where arm_id is the number of the playing part within the performance combination, drum_id1 is the starting point of the movement trajectory of the playing part, and drum_id2 is the ending point of the movement trajectory of the playing part. For example, beat = {1, 1, 4} means that in this performance combination, the robot's left hand moves from the strong cymbal 1 to the snare drum 4 to complete one strike.

[0089] When the humanoid robot plays the drums, following human playing habits, the cymbals can only be struck with the robot's left foot, and the foot drum can only be struck with the robot's right foot. The remaining four drums and three cymbals can be played with either the left or right hand. Therefore, according to the above playing rules, the left foot can obtain one playing combination, namely {3, 8, 8}, and the right foot can obtain one playing combination, namely {4, 9, 9}. The left and right hands can obtain more playing combinations, including single drum strikes of drums (cymbals) 1 to 7, as well as cross-drum strikes between any two drums (cymbals) from 1 to 7, that is, each hand can obtain 28 playing combinations.

[0090] In step S102, for each of the plurality of performance combinations, the joint motion trajectory corresponding to the performance combination is determined based on the position of each instrument played in the performance combination and the contact speed between the playing part in the performance combination and each instrument played. The joint motion trajectory includes the angular displacement of the joint at each sampling point.

[0091] The position of the instrument being played can be the point where the playing part contacts the instrument, such as the striking point of a drum. The contact speed between the playing part and the instrument can be the speed at which the playing part leaves the instrument, such as the speed at which the robot's left hand (or the drumstick in the left hand) leaves the drum, or the speed at which the playing part reaches the instrument, such as the speed at which the robot's left hand (or the drumstick in the left hand) strikes the drum.

[0092] In one possible embodiment, it can be in accordance with the appendix Figure 4 The method shown determines the joint movement trajectory corresponding to the performance combination, including steps S401 to S403.

[0093] In step S401, under at least one constraint, the position where the speed of the playing part in the playing combination is 0 is obtained by minimizing the weighted sum of the distance differences in at least one direction, wherein the at least one constraint is related to the distance differences in at least one direction.

[0094] The position where the velocity of the playing part is 0 can be the position where the playing part changes direction of motion along its trajectory from the starting point to the ending point of the instrument being played. For example, in a scenario where a robot plays a drum kit, the robot's playing part lifts upwards after leaving the starting drum, reaching a maximum velocity of 0, and then descends to the ending drum to strike it. Therefore, the position where the velocity of the playing part is 0 is the highest point of the playing part (in its motion trajectory).

[0095] For example, in the case where the performance combination includes two instruments, i.e., when the playing part switches between different instruments (e.g., strumming a drum), the distance difference in at least one direction may include a horizontal distance difference and a vertical distance difference. Please refer to the appendix. Figure 5 It shows a schematic diagram of the movement trajectory of the playing part when the robot strikes a drum. The starting point of drum 1 is x_ref1, and the ending point of drum 2 is x_ref2. The drumheads of both drums 1 and 2 are on a horizontal plane (i.e., the XOY plane). AB is the line connecting drums 1 and 2 in the horizontal direction (i.e., A is the position of drum 1, and B is the projection of the position of drum 2 onto the horizontal plane where drum 1 is located). x_top is the highest point of the playing part, and s is any point on AB. Normalizing s yields:

[0096]

[0097] C is the projection of the highest point of the movement trajectory of the playing part onto AB. When s is at C, s = k.

[0098] In this example, the at least one constraint may include at least one of the following:

[0099] The vertical distance difference is the vertical distance between the position where the playing part of the performance group has a speed of 0 and the position of any instrument played within the performance group. Figure 5 In the scenario shown, the constraint can be: x top =h ref +Δp, where x top h is the height value of the highest point of the playing part. ref The height value of drum 1 or drum 2, preferably h. ref Δp represents the height of drum 2, i.e., the height of the taller drum, and Δp represents the distance difference in the numerical direction.

[0100] The horizontal distance difference is the horizontal distance between the position where the playing part within the performance combination has a speed of 0 and the boundary of any instrument played within the performance combination. Figure 5In the scenario shown, the constraint can be: Where r1 is the radius of drum 1, r2 is the radius of drum 2, and Δk is the distance difference in the horizontal direction; this constraint condition can ensure that the movement trajectory of the playing part will not touch the drum surface.

[0101] A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the corresponding joints in the performance combination. Figure 5 In the scenario shown, the constraint can be: xz(s) = fk(q) ref (s)), where xz(s) is the trajectory of the playing part, that is, the height value of the playing part is a function of s, that is, the sampling points in the trajectory of the playing part are represented by s, q ref (s) represents the joint motion trajectory, which is a function of the joint's angular displacement with respect to s. In other words, the sampling points in the joint motion trajectory are characterized by s, and fk is the positive kinematic function.

[0102] The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range. Figure 5 In the scenario shown, the constraint can be: q ref (s)∈[q low q up ], where q ref (s) represents the joint motion trajectory, which is a function of the joint's angular displacement with respect to s. In other words, the sampling points in the joint motion trajectory are characterized by s, and q low q represents the lower limit of the joint angle range. up This represents the upper limit of the joint angle range.

[0103] In such Figure 5 In the scenario shown, the weighted sum of the distance differences in at least one direction in this example can be: Kp||Δp||+Kk||Δk||, where Kp is the weighting coefficient of the distance difference in the vertical direction and Kk is the weighting coefficient of the distance difference in the horizontal direction.

[0104] Under the constraints of at least one of the above conditions, the weighted summation of the distance differences in at least one of the above directions is minimized to obtain the value of k and x. top The value is the position of the highest point of the playing part in the horizontal direction and the position in the vertical direction.

[0105] As another example, when the performance combination includes a single instrument being played, i.e., when the same instrument is played continuously by the playing parts (e.g., the same drum is struck continuously), the distance difference in at least one direction may include the distance difference in the vertical direction.

[0106] In this example, the at least one constraint may include at least one of the following:

[0107] The vertical distance difference is the vertical distance between the position where the playing part within the performance combination has a speed of 0 and the position of any instrument played within the performance combination. (Similar to...) Figure 5 In the scenario of simultaneous drumming shown, the constraint can be: x top =h ref +Δp, where x top h is the height value of the highest point of the playing part. ref Let be the height of the drum, and Δp be the distance difference in the numerical direction.

[0108] A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the corresponding joints in the performance combination. Similar to... Figure 5 In the synchronized tapping scenario shown, the constraint can be: xz(s)=fk(q) ref (s)), where xz(s) is the trajectory of the playing part, that is, the height value of the playing part is a function of s, that is, the sampling points in the trajectory of the playing part are represented by s, where s is the position of the playing part in the horizontal direction, and q ref (s) represents the joint motion trajectory, which is a function of the joint's angular displacement with respect to s. In other words, the sampling points in the joint motion trajectory are represented by s, where s is the position of the playing part in the horizontal direction, and fk is the positive kinematic function.

[0109] The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range. Similar to... Figure 5 In the synchronized tapping scenario shown, the constraint can be: q ref (s)∈[q low q up ], where q ref (s) represents the joint motion trajectory, which is a function of the joint's angular displacement with respect to s. The sampling points in the joint motion trajectory are characterized by s, where s is the horizontal position of the playing part, and q... low q represents the lower limit of the joint angle range. up This represents the upper limit of the joint angle range.

[0110] In similar Figure 5 In the synchronized tapping scenario shown, the weighted sum of the distance differences in at least one direction in this example can be: Kp||Δp||, where Kp is the weighting coefficient of the distance difference in the vertical direction.

[0111] Under the constraints of at least one of the above conditions, the weighted summation of the distance differences in at least one of the above directions is minimized to obtain x. top That is, the position of the highest point of the playing part in the vertical direction; as for the position of the highest point of the playing part in the horizontal direction, since the trajectory of the playing part is symmetrical about the highest point when playing the same instrument continuously, k = 0.5.

[0112] In step S402, the motion trajectory of the playing part in the performance combination is determined based on the position of each instrument played in the performance combination, the contact speed between the playing part in the performance combination and each instrument played, and the position where the speed of the playing part in the performance combination is 0. The motion trajectory includes the position of the playing part at each sampling point.

[0113] Specifically, the sub-trajectory of the playing part from the starting point to the position of zero speed and the sub-trajectory of the playing part from the position of zero speed to the end point can be determined based on the position of each instrument played in the playing combination, the contact speed between the playing part and each instrument played in the playing combination, and the position where the speed of the playing part is 0 in the playing combination.

[0114] For example, when the performance combination includes two instruments, i.e., when the playing part switches between different instruments (e.g., strumming a drum), the movement trajectory of the playing part can be described by the following formula:

[0115]

[0116] In this formula, the sampling point of the motion trajectory is represented by the position s in the horizontal direction, and the position of the motion trajectory on the sampling point is represented by the position xz(s) in the vertical direction.

[0117] In this example, the above trajectory satisfies the following condition:

[0118]

[0119]

[0120]

[0121] Because x ref1 x ref2 v beat Given that x top Since k has been obtained in step S201, the above conditions can be substituted into the formula describing the trajectory to obtain a in the formula. 31 a 21 a11 a 01 a 32 a 22 a 12 a 02 The parameters are obtained by solving for the motion trajectory of the playing part.

[0122] As another example, when the performance combination includes a single instrument, i.e., when the playing part continuously plays the same instrument (e.g., continuously striking the same drum), the movement trajectory of the playing part can be described by the following formula:

[0123]

[0124] In this formula, the sampling point of the motion trajectory is represented by the horizontal position s, and the position of the motion trajectory on the sampling point is represented by the vertical position xz(s). The position of s=0.5 in the motion trajectory is the highest point, that is, the motion trajectory is symmetrical about the position of s=0.5.

[0125] In this example, the above trajectory satisfies the following condition:

[0126]

[0127]

[0128]

[0129] Because x ref1 and x ref2 Equal and known, v beat Given that x top Having already obtained this in step S201, the above conditions can be substituted into the formula describing the trajectory to obtain a in the formula. 31 a 21 a 11 a 01 a 32 a 22 a 12 a 02 The parameters are obtained by solving for the motion trajectory of the playing part.

[0130] In step S403, the joint motion trajectory corresponding to the performance combination is determined based on the motion trajectory of the performance parts within the performance combination.

[0131] For example, xz(s) = fk(q) can be used ref(s) and other positive kinematic functions are used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination, so as to solve the joint motion trajectory corresponding to the performance combination.

[0132] In step S103, a performance trajectory library for the robot is generated based on the joint motion trajectory corresponding to each of the multiple performance combinations.

[0133] Among them, the joint motion trajectory corresponding to each performance combination can be saved to the performance trajectory library. That is, the performance trajectory library consists of multiple joint motion trajectories, and each joint motion trajectory is labeled with the corresponding performance combination.

[0134] Based on the performance trajectory library constructed in this step, when the robot plays a certain music file, it can obtain the joint motion trajectory of each performance event in the music file from the performance trajectory library. Each performance event includes the performance combination and performance duration. Then, according to the joint motion trajectory of each performance event in the music file, the robot is controlled to perform the performance action of the music file.

[0135] The music file is presented as a sequence of performance events. The robot completes the performance by sequentially executing each performance event. Since the performance trajectory library contains the joint motion trajectories for every possible performance combination, playing the music file is transformed into executing the joint motion trajectories corresponding to each performance event in sequence, which is convenient, fast, and precise.

[0136] The trajectory generation method provided in this disclosure first generates multiple performance combinations based on multiple playing parts of the robot and multiple instruments being played. Then, for each of the multiple performance combinations, the joint motion trajectory corresponding to the performance combination is determined based on the position of each instrument being played within the performance combination and the contact speed between the playing part and each instrument being played within the performance combination. Finally, a performance trajectory library for the robot is generated based on the joint motion trajectories corresponding to each of the multiple performance combinations. Since multiple performance combinations can cover various combinations of playing parts and instruments being played, the performance trajectory library constructed by this method can provide joint motion trajectories for various movements of the robot when playing music. In other words, when the robot plays instruments such as drums based on the performance trajectory library constructed by this method, it can perform more naturally, smoothly, and accurately, making the movements more in line with real human performance scenarios, avoiding problems such as slow movements and inaccurate movements, and significantly improving the performance effect.

[0137] In some embodiments of this disclosure, after obtaining the motion trajectory of the playing parts within the playing combination, the upper limit of the playing frequency of the playing combination can also be obtained in the following manner, that is, the fastest speed that the robot can achieve when executing the playing combination.

[0138] First, based on the movement trajectory of the playing part within the performance combination, the velocity trajectory of the playing part is determined, wherein the velocity trajectory includes the velocity of the playing part at each sampling moment, and the sampling moment is characterized by the performance frequency.

[0139] In this process, the derivative of the motion trajectory of the playing part with respect to time can be calculated to obtain the velocity trajectory of the playing part, which can be expressed by the following formula:

[0140]

[0141] t∈[0, 60 / tempo]

[0142] In the formula, tempo is the performance frequency, which is the number of times the performance combination is completed within a unit of time (e.g., 1 second or 1 minute).

[0143] Next, under at least one of the following constraints, the upper limit of the playing frequency of the playing combination is obtained by maximizing the playing frequency using the velocity trajectory of the playing part:

[0144] The Jacobian matrix is ​​used to describe the mapping relationship between the velocity trajectory of the playing part and the joint velocity trajectory corresponding to the playing combination. Wherein J(q) ref ) is the Jacobian matrix, The joint velocity trajectory corresponding to the aforementioned performance combination.

[0145] The angular velocity of the joint at each sampling point in the joint velocity trajectory is within the range of joint angular velocity, i.e. Among them, v low v represents the lower limit of the joint velocity range. up This represents the upper limit of the joint velocity range.

[0146] For example, the playing frequency can be gradually increased, and after each increase, the angular velocity of the joint at each sampling point in the joint velocity trajectory can be calculated using the new playing frequency to see if it exceeds the joint velocity range. If it does not exceed the range, the playing frequency can be increased until at least one sampling point in the joint velocity trajectory calculated using the playing frequency obtained after the Nth increase has an angular velocity exceeding the joint velocity range. Then, the playing frequency can be stopped, and the playing frequency obtained after the (N-1)th increase can be determined as the upper limit of the playing frequency.

[0147] Understandably, the upper limit of the performance frequency for each performance combination can also be stored in the performance trajectory library. For example, the upper limit of the performance frequency is also marked when marking the performance combination for each joint movement trajectory.

[0148] Based on the upper limit of the performance frequency determined in the above embodiments, it can be determined whether the robot can complete the performance of a certain music file when playing it. That is, whether the performance duration of each performance event in the music file is less than the duration corresponding to the upper limit of the performance frequency of the performance combination. If it is less, the robot cannot complete the performance event, and the robot can generate a prompt message to inform the user that it cannot complete the music file.

[0149] Alternatively, when the robot plays a music file, for each performance event in the music file, if the performance duration exceeds the duration corresponding to the upper limit of the performance frequency of the performance combination, the performance duration can be corrected. For example, the performance duration of the performance event can be adjusted to the duration corresponding to the upper limit of the performance frequency of the performance combination. This correction can prevent problems such as slow movements caused by the robot's inability to complete music events, and the impact on subsequent performance events.

[0150] According to a second aspect of the present disclosure, a trajectory generation apparatus is provided for use in a robot. Please refer to the attached document. Figure 6 The device includes:

[0151] Combination module 601 is used to generate multiple performance combinations based on multiple playing parts of the robot and multiple instruments to be played, wherein each performance combination includes a playing part and at least one instrument to be played.

[0152] The trajectory module 602 is used to determine the joint motion trajectory corresponding to each of the plurality of performance combinations based on the position of each instrument played in the performance combination and the contact speed between the playing part in the performance combination and each instrument played. The joint motion trajectory includes the angular displacement of the joint at each sampling point.

[0153] The storage module 603 is used to generate a robot performance trajectory library based on the joint motion trajectory corresponding to each of the multiple performance combinations.

[0154] In some embodiments of this disclosure, the combined module is specifically used for:

[0155] Each of the plurality of playing parts is combined with each of the plurality of playing instruments to form a playing combination, wherein the playing instruments in the playing combination serve as the starting point and the ending point of the movement trajectory of the playing part.

[0156] Each of the plurality of playing parts is combined with every two of the plurality of playing instruments to form a playing combination, wherein one of the playing instruments in the playing combination serves as the starting point of the movement trajectory of the playing part, and the other playing instrument serves as the ending point of the movement trajectory of the playing part.

[0157] In some embodiments of this disclosure, the trajectory module is specifically used for:

[0158] Under at least one constraint, the position where the speed of the playing part in the performance combination is 0 is obtained by minimizing the weighted sum of the distance differences in at least one direction, wherein the at least one constraint is related to the distance differences in at least one direction;

[0159] Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the motion trajectory of the playing part in the performance combination is determined, wherein the motion trajectory includes the position of the playing part at each sampling point.

[0160] Based on the movement trajectory of the playing parts within the playing combination, the joint movement trajectory corresponding to the playing combination is determined.

[0161] In some embodiments of this disclosure, when the performance combination includes a single instrument being played, the at least one constraint includes at least one of the following:

[0162] The vertical distance difference is the vertical distance between the position where the playing part of the performance combination has a speed of 0 and the position of any instrument played in the performance combination.

[0163] The horizontal distance difference is the horizontal distance between the position where the playing part of the performance is at 0 speed and the boundary of any instrument played in the performance combination.

[0164] A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination;

[0165] The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

[0166] In some embodiments of this disclosure, when the performance combination includes two instruments being played, the at least one constraint includes at least one of the following:

[0167] The vertical distance difference is the vertical distance between the position where the playing part of the performance combination has a speed of 0 and the position of any instrument played in the performance combination.

[0168] A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination;

[0169] The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

[0170] In some embodiments of this disclosure, when the trajectory module is used to determine the motion trajectory of the playing part within the performance combination based on the position of each instrument played in the performance combination, the contact velocity between the playing part and each instrument played in the performance combination, and the position where the velocity of the playing part within the performance combination is 0, it is specifically used for:

[0171] Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the sub-trajectory of the playing part from the starting point to the position where the speed is 0, and the sub-trajectory of the playing part from the position where the speed is 0 to the ending point are determined.

[0172] In some embodiments of this disclosure, a frequency module is also included, for:

[0173] Based on the movement trajectory of the playing part within the performance combination, the velocity trajectory of the playing part is determined, wherein the velocity trajectory includes the velocity of the playing part at each sampling moment, and the sampling moment is characterized by the performance frequency;

[0174] Under at least one of the following constraints, the upper limit of the playing frequency of the playing combination is obtained by maximizing the playing frequency using the velocity trajectory of the playing part: the Jacobian matrix used to describe the mapping relationship between the velocity trajectory of the playing part and the joint velocity trajectory corresponding to the playing combination; and the angular velocity of the joint at each sampling point in the joint motion trajectory is within the range of joint angular velocity.

[0175] In some embodiments of this disclosure, a performance module is also included, for:

[0176] From the performance trajectory library, obtain the joint motion trajectory of each performance event in the music file, wherein each performance event includes the performance combination and performance duration;

[0177] Based on the joint motion trajectory of each performance event in the music file, the robot is controlled to perform the performance actions of the music file.

[0178] In some embodiments of this disclosure, a correction module is also included, for:

[0179] For each performance event in the music file, if the performance duration is greater than the duration corresponding to the upper limit of the performance frequency of the performance combination, the performance duration is corrected.

[0180] In one embodiment, the plurality of playing parts include the robot's limbs; and / or, the plurality of instruments being played include a plurality of drums and cymbals from a drum kit.

[0181] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the method in the first aspect, and will not be elaborated upon here.

[0182] Thirdly, at least one embodiment of this disclosure provides a robot, please refer to the appendix. Figure 7 The diagram illustrates the structure of the robot, which includes a memory and a processor. The memory stores computer instructions that can run on the processor, and the processor generates a trajectory based on the method described in any of the first aspects when executing the computer instructions.

[0183] Fourthly, at least one embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the first aspects.

[0184] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0185] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A trajectory generation method, characterized in that, Applied to robots, the method includes: Based on the robot's multiple playing parts and multiple instruments being played, multiple playing combinations are generated, wherein each playing combination includes a playing part and at least one instrument being played. For each of the multiple performance combinations, the joint motion trajectory corresponding to the performance combination is determined based on the position of each instrument played in the performance combination and the contact speed between the playing part in the performance combination and each instrument played. The joint motion trajectory includes the angular displacement of the joint at each sampling point. A library of robot performance trajectories is generated based on the joint motion trajectories corresponding to each of the multiple performance combinations. Also includes: From the performance trajectory library, obtain the joint motion trajectory of each performance event in the music file, wherein each performance event includes the performance combination and performance duration; Based on the joint motion trajectory of each performance event in the music file, the robot is controlled to perform the performance actions of the music file.

2. The trajectory generation method according to claim 1, characterized in that, The process of generating multiple performance combinations based on multiple playing parts of the robot and multiple instruments being played includes: Each of the multiple playing parts is combined with each of the multiple instruments being played to form a playing combination, wherein the instruments being played in the playing combination serve as the starting and ending points of the movement trajectory of the playing part. Each of the plurality of playing parts is combined with every two of the plurality of playing instruments to form a playing combination, wherein one of the playing instruments in the playing combination serves as the starting point of the movement trajectory of the playing part, and the other playing instrument serves as the ending point of the movement trajectory of the playing part.

3. The trajectory generation method according to claim 1, characterized in that, The step of determining the joint motion trajectory corresponding to the performance combination based on the position of each instrument played within the performance combination and the contact speed between the playing part and each instrument played within the performance combination includes: Under at least one constraint, the position where the speed of the playing part in the performance combination is 0 is obtained by minimizing the weighted sum of the distance differences in at least one direction, wherein the at least one constraint is related to the distance differences in at least one direction; Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the motion trajectory of the playing part in the performance combination is determined, wherein the motion trajectory includes the position of the playing part at each sampling point. Based on the movement trajectory of the playing parts within the playing combination, the joint movement trajectory corresponding to the playing combination is determined.

4. The trajectory generation method according to claim 3, characterized in that, When the performance combination includes a single instrument being played, the at least one constraint includes at least one of the following: The vertical distance difference is the vertical distance between the position where the playing part of the performance is at a speed of 0 and the position of any instrument played in the performance combination. The horizontal distance difference is the horizontal distance between the position where the playing part of the performance is at 0 speed and the boundary of any instrument played in the performance combination. A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination; The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

5. The trajectory generation method according to claim 3, characterized in that, When the performance combination includes two instruments being played, the at least one constraint includes at least one of the following: The vertical distance difference is the vertical distance between the position where the playing part of the performance is at a speed of 0 and the position of any instrument played in the performance combination. A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination; The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

6. The trajectory generation method according to claim 3, characterized in that, The step of determining the motion trajectory of the playing part within the performance combination based on the position of each instrument played in the performance combination, the contact velocity between the playing part and each instrument played in the performance combination, and the position where the velocity of the playing part in the performance combination is 0, includes: Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the sub-trajectory of the playing part from the starting point to the position where the speed is 0, and the sub-trajectory of the playing part from the position where the speed is 0 to the ending point are determined.

7. The trajectory generation method according to claim 3, characterized in that, Also includes: Based on the movement trajectory of the playing part within the performance combination, the velocity trajectory of the playing part is determined, wherein the velocity trajectory includes the velocity of the playing part at each sampling moment, and the sampling moment is characterized by the performance frequency; Under at least one of the following constraints, the upper limit of the playing frequency of the playing combination is obtained by maximizing the playing frequency using the velocity trajectory of the playing part: the Jacobian matrix used to describe the mapping relationship between the velocity trajectory of the playing part and the joint velocity trajectory corresponding to the playing combination; and the angular velocity of the joint at each sampling point in the joint motion trajectory is within the range of joint angular velocity.

8. The trajectory generation method according to claim 1, characterized in that, Also includes: For each performance event in the music file, if the performance duration is greater than the duration corresponding to the upper limit of the performance frequency of the performance combination, the performance duration is corrected.

9. The trajectory generation method according to any one of claims 1 to 8, characterized in that, The plurality of playing parts include the robot’s limbs; and / or, the plurality of instruments played include a plurality of drums and cymbals of a drum kit.

10. A trajectory generation device, characterized in that, The device, applied to robots, includes: The combination module is used to generate multiple performance combinations based on the robot's multiple playing parts and multiple instruments being played, wherein each performance combination includes a playing part and at least one instrument being played. The trajectory module is used to determine the joint motion trajectory corresponding to each of the multiple performance combinations based on the position of each instrument played in the performance combination and the contact speed between the playing part and each instrument played in the performance combination. The joint motion trajectory includes the angular displacement of the joint at each sampling point. The storage module is used to generate a robot performance trajectory library based on the joint motion trajectory corresponding to each of the multiple performance combinations. It also includes a performance module for: From the performance trajectory library, obtain the joint motion trajectory of each performance event in the music file, wherein each performance event includes the performance combination and performance duration; Based on the joint motion trajectory of each performance event in the music file, the robot is controlled to perform the performance actions of the music file.

11. The trajectory generation device according to claim 10, characterized in that, The combined module is specifically used for: Each of the multiple playing parts is combined with each of the multiple instruments being played to form a playing combination, wherein the instruments being played in the playing combination serve as the starting and ending points of the movement trajectory of the playing part. Each of the plurality of playing parts is combined with every two of the plurality of playing instruments to form a playing combination, wherein one of the playing instruments in the playing combination serves as the starting point of the movement trajectory of the playing part, and the other playing instrument serves as the ending point of the movement trajectory of the playing part.

12. The trajectory generation device according to claim 10, characterized in that, The trajectory module is specifically used for: Under at least one constraint, the position where the speed of the playing part in the performance combination is 0 is obtained by minimizing the weighted sum of the distance differences in at least one direction, wherein the at least one constraint is related to the distance differences in at least one direction; Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the motion trajectory of the playing part in the performance combination is determined, wherein the motion trajectory includes the position of the playing part at each sampling point. Based on the movement trajectory of the playing parts within the playing combination, the joint movement trajectory corresponding to the playing combination is determined.

13. The trajectory generation device according to claim 12, characterized in that, When the performance combination includes a single instrument being played, the at least one constraint includes at least one of the following: The vertical distance difference is the vertical distance between the position where the playing part of the performance is at a speed of 0 and the position of any instrument played in the performance combination. The horizontal distance difference is the horizontal distance between the position where the playing part of the performance is at 0 speed and the boundary of any instrument played in the performance combination. A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination; The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

14. The trajectory generation device according to claim 12, characterized in that, When the performance combination includes two instruments being played, the at least one constraint includes at least one of the following: The vertical distance difference is the vertical distance between the position where the playing part of the performance is at a speed of 0 and the position of any instrument played in the performance combination. A positive kinematic function used to describe the mapping relationship between the motion trajectory of the playing parts within the performance combination and the motion trajectory of the joints corresponding to the performance combination; The angular displacement of the joint at each sampling point in the joint motion trajectory is within the joint angle range.

15. The trajectory generation device according to claim 12, characterized in that, The trajectory module is used to determine the motion trajectory of the playing parts within the performance combination based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part is 0 in the performance combination. Specifically, it is used for: Based on the position of each instrument played in the performance combination, the contact speed between the playing part and each instrument played in the performance combination, and the position where the speed of the playing part in the performance combination is 0, the sub-trajectory of the playing part from the starting point to the position where the speed is 0, and the sub-trajectory of the playing part from the position where the speed is 0 to the ending point are determined.

16. The trajectory generation device according to claim 12, characterized in that, It also includes a frequency module, used for: Based on the movement trajectory of the playing part within the performance combination, the velocity trajectory of the playing part is determined, wherein the velocity trajectory includes the velocity of the playing part at each sampling moment, and the sampling moment is characterized by the performance frequency; Under at least one of the following constraints, the upper limit of the playing frequency of the playing combination is obtained by maximizing the playing frequency using the velocity trajectory of the playing part: the Jacobian matrix used to describe the mapping relationship between the velocity trajectory of the playing part and the joint velocity trajectory corresponding to the playing combination; and the angular velocity of the joint at each sampling point in the joint motion trajectory is within the range of joint angular velocity.

17. The trajectory generation device according to claim 10, characterized in that, It also includes a calibration module for: For each performance event in the music file, if the performance duration is greater than the duration corresponding to the upper limit of the performance frequency of the performance combination, the performance duration is corrected.

18. The trajectory generation apparatus according to any one of claims 10 to 17, characterized in that, The plurality of playing parts include the robot’s limbs; and / or, the plurality of instruments played include a plurality of drums and cymbals of a drum kit.

19. A robot, characterized in that, The robot includes a memory and a processor. The memory stores computer instructions that can be executed on the processor, and the processor implements the trajectory generation method of any one of claims 1 to 9 when executing the computer instructions.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 9.

Images