Method and apparatus for determining flight trajectory of irregular object

Abstract

The present disclosure relates to a method and apparatus for determining the flight trajectory of an irregular object, a method and device for displaying the flight trajectory of an irregular virtual object, and a robot for grasping an irregular object in flight. The method includes: determining parameters in a kinematic model and a dynamic model corresponding to the irregular object based on the attitude data of the irregular object in a first time period and the position data of a pivot point of the irregular object; determining predicted attitude data corresponding to one or more moments in a second time period based on the dynamic model corresponding to the irregular object; and determining predicted position data corresponding to one or more moments in the second time period based on the predicted attitude data corresponding to one or more moments in the second time period of the pivot point and the kinematic model. The present disclosure only needs to observe the pose data of an irregular object during a certain period of flight to determine the future flight trajectory of the object.

Description

Technical Field

[0001] The present disclosure relates to the field of robots and / or the field of virtual reality, and more specifically, to a method for determining the flight trajectory of an irregular object, a method for displaying the flight trajectory of an irregular virtual object, a device for displaying the flight trajectory of an irregular virtual object, a device for determining the flight trajectory of an irregular object, a robot for grasping an irregular object in flight, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the continuous development of technologies in the field of robots, various types of robots such as industrial robots and service robots have been increasingly used in various technical fields, such as intelligent agriculture, intelligent factories, and intelligent warehousing. Generally, an intelligent robot often attempts to use its operating part (e.g., a manipulator) to catch an irregular object. However, since the flight trajectory of the irregular object in the air is difficult to determine, the robot often cannot accurately locate the position where the object will fall, resulting in the inability to catch the irregular object and causing damage to the irregular object.

[0003] In addition, in 3D games, there are often scenarios of simulating the grasping of an irregular virtual object in flight. Currently, in 3D games, the flight trajectory of the irregular virtual object is often simply equivalent to the flight trajectory of a regular virtual object, which has a large difference from the real world and results in a poor gaming experience.

[0004] Therefore, there is a need for a method and device for determining the flight trajectory of an irregular object, a robot for grasping an irregular object in flight, and a robot capable of executing the above method. Summary of the Invention

[0005] The present disclosure provides a method for determining the flight trajectory of an irregular object, a method for displaying the flight trajectory of an irregular virtual object, a device for displaying the flight trajectory of an irregular virtual object, a device for determining the flight trajectory of an irregular object, a robot for grasping an irregular object in flight, an electronic device, and a computer-readable storage medium.

[0006] For example, the present disclosure provides a method for determining the flight trajectory of an irregular object, including: determining parameters in the kinematic model and the dynamic model corresponding to the irregular object based on the attitude data of the irregular object in a first time period and the position data of the pivot point of the irregular object; determining predicted attitude data corresponding to one or more moments in a second time period based on the dynamic model corresponding to the irregular object; and determining predicted position data corresponding to one or more moments in the second time period based on the predicted attitude data corresponding to one or more moments in the second time period of the pivot point and the kinematic model.

[0007] For example, the flight trajectory of the center of mass of the irregular object is a parabola with the Z-axis upward in the world coordinate system, and the flight trajectory of the pivot point of the irregular object is an irregular convex curve in the world coordinate system.

[0008] For example, the parameters of the dynamic model corresponding to the irregular object include the three-axis inertia parameters in the object coordinate system and the viscous drag parameters based on air resistance.

[0009] For example, the parameters of the kinematic model corresponding to the irregular object include the initial linear velocity in the world coordinate system, the offset of the pivot point relative to the center of mass, and the initial position.

[0010] For example, determining the parameters in the kinematic model and the dynamic model corresponding to the irregular object further includes: determining rotation matrices at multiple moments in the first time period based on the attitude data of the irregular object in the first time period, where each rotation matrix at each of the multiple moments indicates the mapping relationship from the object coordinate system to the world coordinate system at that moment; and determining the parameters in the kinematic model corresponding to the irregular object based on the position data of the pivot point of the irregular object in the first time period and the rotation matrices at the multiple moments in the first time period.

[0011] For example, determining the parameters in the kinematic model and the dynamic model corresponding to the pivot point further includes: determining rotation matrices at multiple moments in the first time period based on the attitude data of the irregular object in the first time period, where each rotation matrix at each of the multiple moments indicates the mapping relationship from the object coordinate system to the world coordinate system at that moment; determining the angular velocity and angular acceleration of the pivot point rotating around the three axes of the object coordinate system at at least one moment in the first time period based on the rotation matrices at the multiple moments in the first time period; and determining the parameters of the dynamic model based on the angular velocity and angular acceleration of the pivot point rotating around the three axes of the object coordinate system at at least one moment in the first time period.

[0012] For example, the determination of the parameters in the kinematic model and the dynamic model corresponding to the pivot point further includes: obtaining the attitude data of the irregular object in flight at multiple moments in the first time period and the position data of the pivot point of the irregular object; based on the attitude data and the position data at multiple moments in the first time period, determining the smooth attitude data of the irregular object in flight at multiple moments in the first time period and the smooth position data of the pivot point of the irregular object; and based on the smooth attitude data of the irregular object in flight at multiple moments in the first time period and the smooth position data of the pivot point of the irregular object, using the least squares method to determine the parameters in the kinematic model corresponding to the irregular object.

[0013] For example, the determination of the predicted attitude data corresponding to one or more moments of the pivot point in the second time period further includes: based on the angular velocity and angular acceleration of the pivot point rotating around the three axes of the object coordinate system at one or more moments in the first time period and the dynamic model corresponding to the pivot point, determining the predicted angular velocity and predicted angular acceleration of the pivot point at one or more moments in the second time period; based on the predicted angular velocity and predicted angular acceleration of the pivot point in the second time period, determining the predicted rotation matrix corresponding to one or more moments in the second time period, and each predicted rotation matrix in the predicted rotation matrix corresponding to the one or more moments indicates the mapping relationship from the object coordinate system to the world coordinate system at the moment; and based on the predicted rotation matrix corresponding to one or more moments in the second time period, determining the predicted attitude data corresponding to one or more moments of the pivot point in the second time period.

[0014] For example, the determination of the predicted position data corresponding to one or more moments of the pivot point in the second time period further includes: based on the predicted rotation matrix corresponding to one or more moments in the second time period and the kinematic model, determining the predicted position data corresponding to one or more moments of the pivot point in the second time period.

[0015] For example, the dynamic model is:

[0016]

[0017] Wherein, I x , I y , I z are the inertia parameters corresponding to the x-axis, y-axis and z-axis in the object coordinate system respectively, and k d is the viscous resistance parameter based on air resistance, are the angular accelerations corresponding to the x-axis, y-axis and z-axis in the object coordinate system respectively, and ω x , ω y , ω zThey are the angular velocities corresponding to the x-axis, y-axis, and z-axis in the object coordinate system, respectively.

[0018] For example, the kinematic model is as follows:

[0019]

[0020] Wherein, a1, a2, and a3 are the initial linear velocities of the center of mass in the positive directions of the X-axis, Y-axis, and Z-axis in the world coordinate system, respectively, b1, b2, and b3 are the initial positions of the center of mass on the X-axis, Y-axis, and Z-axis in the world coordinate system, respectively, Δx, Δy, and Δz are the offsets of the pivot point relative to the center of mass on the X-axis, Y-axis, and Z-axis in the object coordinate system, respectively, and x, y, and z are the projections of the pivot point on the X-axis, Y-axis, and Z-axis in the world coordinate system at time t. is the rotation matrix from the world coordinate system to the object coordinate system at time t.

[0021] For example, the present disclosure provides a method for displaying the flight trajectory of an irregular virtual object, including: determining the parameters in the kinematic model and dynamic model corresponding to the irregular virtual object based on the attitude data of the irregular virtual object in flight during a first time period and the position data of the pivot point of the irregular virtual object; determining the predicted attitude data corresponding to one or more moments of the irregular virtual object during a second time period based on the dynamic model corresponding to the irregular virtual object; determining the predicted position data corresponding to one or more moments of the pivot point during the second time period based on the attitude data corresponding to one or more moments of the pivot point during the second time period and the kinematic model; and rendering the flight screen of the irregular virtual object during the second time period and displaying the flight screen based on the predicted attitude data corresponding to one or more moments of the irregular virtual object during the second time period and the predicted position data corresponding to one or more moments of the pivot point during the second time period.

[0022] For example, the present disclosure provides a device for displaying the flight trajectory of an irregular virtual object, including: a processor configured to: determine parameters in the kinematic model and dynamic model corresponding to the irregular virtual object based on the attitude data of the irregular virtual object in flight during a first time period and the position data of the pivot point of the irregular virtual object; determine predicted attitude data corresponding to one or more moments in a second time period for the irregular virtual object based on the dynamic model corresponding to the irregular virtual object; determine predicted position data corresponding to one or more moments in the second time period for the pivot point based on the attitude data corresponding to the one or more moments in the second time period for the pivot point and the kinematic model; and render a flight screen for the irregular virtual object in the second time period based on the attitude data corresponding to the one or more moments in the second time period for the irregular virtual object and the predicted position data corresponding to the one or more moments in the second time period for the pivot point; a display screen configured to: display the flight screen.

[0023] For example, the present disclosure provides a device for determining the flight trajectory of an irregular object, including: a parameter determination module configured to: determine parameters in the kinematic model and dynamic model corresponding to the irregular virtual object based on the attitude data of the irregular object in flight during a first time period and the position data of the pivot point of the irregular object; a trajectory prediction module configured to: determine predicted attitude data corresponding to one or more moments in a second time period for the pivot point based on the dynamic model corresponding to the irregular virtual object; and determine predicted position data corresponding to one or more moments in the second time period for the pivot point based on the predicted attitude data corresponding to the one or more moments in the second time period for the pivot point and the kinematic model.

[0024] For example, the present disclosure provides a robot for grasping an irregular object in flight, the robot having: an operating part for grasping the irregular object in flight; a controller for controlling the operating part to adjust its pose, the controller being provided on the robot and configured to execute the above method.

[0025] In addition, the present disclosure further provides an electronic device, including: a processor; a memory storing computer instructions, which when executed by the processor implement the above method.

[0026] In addition, the present disclosure further provides a computer-readable storage medium having computer instructions stored thereon, which when executed by a controller implement the above method.

[0027] According to another aspect of the present disclosure, there is provided a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A controller of a robot reads the computer instructions from the computer-readable medium, and the controller executes the computer instructions, so that the robot executes the methods provided in the above aspects or various alternative implementations of the above aspects.

[0028] In summary, the present disclosure provides a method for determining the flight trajectory of an irregular object, a method for displaying the flight trajectory of an irregular virtual object, a device for displaying the flight trajectory of an irregular virtual object, a device for determining the flight trajectory of an irregular object, a robot for grasping an irregular object in flight, an electronic device, and a computer-readable storage medium. The present disclosure only needs to observe the position data and attitude data of an irregular object at a certain moment or a certain position during the flight process, and can determine the future flight trajectory of the object, including the linear velocity, angular velocity, position, and attitude of the object within a certain period of time in the future.

[0029] In addition, the present disclosure also uses methods of rigid body kinematics, rigid body dynamics, and aerodynamics to achieve high-precision prediction of the flight trajectory of irregular objects with a relatively low computational cost. Compared with the existing technology that needs to collect a large number of motion trajectories of irregular objects in advance as samples to predict the flight trajectory of irregular objects, the present disclosure greatly reduces the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some exemplary embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0031] Herein, in the drawings:

[0032] Figure 1 Shows a flowchart of a method for determining the flight trajectory of an irregular object according to an embodiment of the present disclosure.

[0033] Figure 2 Shows a schematic diagram of a method for determining the flight trajectory of an irregular object according to an embodiment of the present disclosure, which shows the flight trajectory of an irregular object taking a hammer as an example.

[0034] Figure 3 Shows the variation of the angular velocity of an irregular object around three rotation principal axes with time according to an embodiment of the present disclosure.

[0035] Figure 4Shows another flowchart of a method for determining the flight trajectory of an irregular object according to an embodiment of the present disclosure.

[0036] Figure 5 Shows a flowchart of determining each parameter in a dynamic model according to an embodiment of the present disclosure.

[0037] Figure 6A Shows a comparison graph of the angular velocities of three axes predicted by a dynamic model corresponding to the Euler equation without external torque and the actual angular velocities of the three axes according to an embodiment of the present disclosure.

[0038] Figure 6B Shows a comparison graph of the rotational kinetic energy predicted by a dynamic model corresponding to the Euler equation without external torque and the actual rotational kinetic energy according to an embodiment of the present disclosure.

[0039] Figure 7A Shows a comparison graph of the angular velocities of three axes predicted by a dynamic model corresponding to the Euler equation with air resistance and the actual angular velocities of the three axes according to an embodiment of the present disclosure.

[0040] Figure 7B Shows a comparison graph of the rotational kinetic energy predicted by a dynamic model corresponding to the Euler equation with air resistance and the actual rotational kinetic energy according to an embodiment of the present disclosure.

[0041] Figure 8 Shows a schematic diagram of a method for displaying the flight trajectory of an irregular virtual object according to an embodiment of the present disclosure.

[0042] Figure 9 Shows a structural diagram of a robot for grasping an irregular object in flight according to an embodiment of the present disclosure.

[0043] Figure 10 Shows a schematic diagram of the process of attempting to grasp an irregular object according to an embodiment of the present disclosure. Detailed implementation manners

[0044] In order to make the objectives, technical solutions, and advantages of the present disclosure more obvious, exemplary embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. It should be understood that the present disclosure is not limited by the exemplary embodiments described herein.

[0045] In addition, in this specification and the accompanying drawings, steps and elements having substantially the same or similar steps and elements are denoted by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted.

[0046] In addition, in this specification and the accompanying drawings, according to the embodiments, elements are described in the singular or plural form. However, the singular and plural forms are appropriately selected for the proposed cases solely for convenience of explanation and are not intended to limit the present disclosure thereto. Therefore, the singular form may include the plural form, and the plural form may also include the singular form, unless the context clearly indicates otherwise.

[0047] In addition, in this specification and the accompanying drawings, the terms "first / second" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. Understandably, "first / second" can be interchanged in a specific order or sequence when allowed, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0048] In addition, in this specification and the accompanying drawings, the terms related to orientation or positional relationship such as "upper", "lower", "vertical", "horizontal", etc. are only used for convenience in describing the embodiments according to the present disclosure and are not intended to limit the present disclosure thereto. Therefore, it should not be construed as a limitation of the present disclosure.

[0049] In addition, in this specification and the accompanying drawings, unless otherwise clearly stated, "connection" does not necessarily mean "direct connection" or "direct contact". Here, "connection" can represent both a fixing effect and an electrical connection.

[0050] In addition, in this specification and the accompanying drawings, unless otherwise clearly stated, "motion trajectory" and "flight trajectory" are not only the displacement, trajectory or attitude of an object. Here, as commonly used in robot dynamics, "motion trajectory" and "flight trajectory" not only represent the displacement of an object or the angle formed in space, but also represent the velocity, acceleration, angular velocity, and angular acceleration of the object at each moment.

[0051] In addition, in this specification and the accompanying drawings, unless otherwise clearly stated, "driving force" should be understood in a broad sense, that is, "driving force" does not necessarily mean "force". For the specific driving object of this "driving force", "driving force" can represent both a narrow sense of force and "driving torque", "driving torque" (for example, for a pivot point).

[0052] As an example, the present disclosure can be applied to the field of intelligent sensors combined with Artificial Intelligence (AI). Among them, Artificial Intelligence is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, Artificial Intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial Intelligence also studies the design principles and implementation methods of various intelligent machines to enable the machines to have the functions of perception, reasoning, and decision-making.

[0053] Artificial Intelligence technology is an interdisciplinary subject with a wide range of fields involved, including both hardware-level technologies and software-level technologies. The basic technologies of Artificial Intelligence generally include technologies such as sensors, dedicated Artificial Intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of Artificial Intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0054] Currently, with the research and progress of Artificial Intelligence technology, Artificial Intelligence technology has been studied and applied in multiple fields. For example, common ones include smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless, autonomous driving, drones, robots, smart healthcare, smart customer service, etc. Currently, by utilizing the perception, reasoning, and decision-making functions of Artificial Intelligence, Artificial Intelligence has been combined with various types of robots and applied to various fields such as intelligent agriculture, intelligent factories, and intelligent warehousing to achieve the purpose of replacing manual labor to perform desired operations on various objects in the above fields (such as moving an object to a target location) and significantly improving the automation level of the above fields and reducing human resource costs.

[0055] How to predict the flight trajectory of irregular objects has always been a difficult problem in the industry. Currently, various solutions have been proposed to predict the flight trajectory of irregular objects. The following will briefly compare and introduce these solutions with the present disclosure.

[0056] The EPFL-LASA laboratory uses machine learning algorithms to learn and predict the flight trajectories of irregular objects (for the specific solution, see Estimating the non-linear dynamics of free-flying objects). This solution does not rely on physical models and can predict the flight trajectories of non-rigid bodies. However, this solution requires a large amount of trajectories similar to the trajectories to be predicted to be collected in advance for model training, with high costs, low success rates, and low reproducibility rates. Compared with this solution, the present disclosure does not require a large amount of data to be collected in advance to train the model, greatly saving the deployment cost.

[0057] Zhejiang University also announced a trajectory prediction solution (Neural Motion Prediction for In-flight Uneven Object Catching), which is also a solution that uses a neural network model to learn the trajectories of irregular objects. This solution can only predict the centroid trajectory but not the attitude of the object. Compared with this solution, the present disclosure not only does not require training a neural network model but also can predict the attitude of the object.

[0058] In addition, when the above-mentioned machine learning algorithm is used to predict the trajectory of an irregular object with rotation, since the trajectory of the Euler angles is fitted, it is impossible to predict the rotation situation where the Euler angles are discontinuous. However, the embodiments of the present disclosure use a dynamics solution to predict the angular velocity of the irregular object, effectively avoiding the situation where the Euler angles are discontinuous.

[0059] Iowa State University developed an algorithm for estimating the motion trajectory of irregular objects (for the specific solution, see Motion Estimation of Free-Flying Objects: Aerodynamics, Constrained Filtering, and Graph-Based Feature Tracking). This algorithm uses two monocular cameras to observe the flying irregular object and estimate and predict the attitude of the object. Although this solution has a very low cost and can accurately observe the current pose of the flying object, the predicted motion trajectory is not ideal, and the amount of calculation is too large to predict in real time.

[0060] The present disclosure only needs to observe the position data and attitude data of an irregular object at a certain period of time (or several moments) or several positions during the flight process to determine the future flight trajectory of the object, including the linear velocity, angular velocity, position, and attitude of the object within a future period of time / at any future moment. The present disclosure also uses methods of rigid body kinematics, rigid body dynamics, and aerodynamics to achieve high-precision prediction of the flight trajectory of irregular objects with a relatively low amount of computation.

[0061] The following further describes various embodiments of the present disclosure with reference to the accompanying drawings.

[0062] First, refer to Figures 1 to 3 To further illustrate the method for determining the flight trajectory of an irregular object according to an embodiment of the present disclosure. Figure 1 A flow chart of a method 100 for determining a flight trajectory of an irregular object according to an embodiment of the present disclosure is shown. Figure 2 A schematic diagram of a method 100 for determining a flight trajectory of an irregular object according to an embodiment of the present disclosure is shown, which shows the flight trajectory of an irregular object taking a hammer as an example. Figure 3 The graph shows the variation of the angular velocity of an irregular object around three main axes of rotation over time according to an embodiment of the present disclosure.

[0063] like Figure 1 As shown, the method 100 for determining the flight trajectory of an irregular object according to an embodiment of the present disclosure exemplarily includes steps S110 to S130. The embodiment of the present disclosure may also include more or fewer steps, and the present disclosure is not limited thereto.

[0064] In step S110, based on the posture data of the irregular object in flight in the first time period and the position data of the pivot point of the irregular object, the parameters in the kinematic model and the dynamic model corresponding to the irregular object are determined. The first time period includes at least two adjacent moments.

[0065] refer to Figure 2 , the enlarged image of the posture of the irregular object, taking a hammer as an example, in the first time period is as follows Figure 2 As shown in the left figure, its center of mass com is located at the junction of the handle and the end, and its pivot point (Pivot Point) P is located at the end of the handle. The flight trajectory of the irregular object is shown in Figure 2 As shown in the right figure of , the trajectory of the center of mass com is shown in dotted lines, and the trajectory of the pivot point P is shown in solid lines. In various possible application scenarios, the pivot point can also be called a grasping point. For example, when a robot attempts to grasp an irregular object in flight, it is necessary to determine the position of the pivot point P to achieve accurate grasping. In fact, the pivot point in the embodiments of the present disclosure can be any point on the irregular object except the center of mass. Figure 2 The description is made by taking point P at the end of the handle as an example, but the present disclosure is not limited thereto.

[0066] For example, the posture data refers to data related to the posture of the irregular object, such as the tilt angle of the irregular object, etc. For example, the posture data can be represented by the posture relationship between the object coordinate system corresponding to the irregular object and the world coordinate system.

[0067] The origin of the object coordinate system (also known as the body coordinate system) corresponding to the irregular object is fixed at the center of mass com, and rotates with the rotation of the rigid body and translates with the translation of the rigid body. The origin of the world coordinate system can be fixed at any position, and the world coordinate system is stationary.

[0068] For example, in a scenario where a robot attempts to grasp a flying irregular object, the lower left corner or the center of the observation image captured by the camera on the robot can be used as the origin of the world coordinate system. Another example is that in a virtual 3D game scenario, the lower left corner or the center of the rendered observation image / or the position of a certain fixed object can be used as the origin of the world coordinate system. The present disclosure does not impose any restrictions on the setting method of the origin O of the world coordinate system.

[0069] For example, the X-axis of the world coordinate system points in the direction perpendicular to the observation image and outward, the Y-axis points in the horizontal right direction in the observation image, and the Z-axis points in the vertical upward direction in the observation image. The Y1-axis of the object coordinate system is a straight line passing through the center of mass, which is the first principal axis of rotation of the irregular object. The X1-axis of the object coordinate system is a straight line passing through the center of mass and the X1-axis is perpendicular to the Y1-axis, which is the second principal axis of rotation of the irregular object. The Z1-axis of the object coordinate system is a straight line passing through the center of mass and the Z1-axis is perpendicular to the plane formed by the X1-axis and the Y1-axis, which is the third principal axis of rotation of the irregular object. The above-mentioned first principal axis of rotation, second principal axis of rotation, and third principal axis of rotation have the following characteristics: when the irregular object rotates around one of the principal axes of rotation at a certain angular velocity, the angular momentum of the irregular object is equal to the product of the angular velocity and the moment of inertia corresponding to the principal axis of rotation. The value of the moment of inertia indicates how much torque is required to generate a unit angular acceleration around the principal axis of rotation (in other words, to make it rotate faster or slower). The highest moment of inertia requires the largest torque, while the lowest moment of inertia requires the smallest torque.

[0070] Another example is that for an irregular object such as a hammer, the Y1-axis of the object coordinate system coincides with the vertical central axis, and the positive direction points outward from the hammer. The X1-axis is perpendicular to the central axis where the Y1-axis is located and is parallel to the transverse central axis of the hammer end, and the positive direction points to the right. The Z1-axis is perpendicular to the plane formed by the Y1-axis and the X1-axis, and Z1 and the Y1-axis and the X1-axis form a right-handed rectangular coordinate system. The establishment scheme of the object coordinate system varies with the different shapes of different irregular objects, and the present disclosure does not limit this.

[0071] Another example is that the attitude data can be represented by attitude angles (also known as Euler angles). Euler angles are used to represent the relative angular position relationship between the coordinate system of the irregular object and the world coordinate system. Among them, the pitch angle θ is the angle between the Y1-axis and the plane formed by the X-axis and the Y-axis of the world coordinate system; the yaw angle is the angle between the projection of the Y1 axis on the plane formed by the X axis and the Y axis of the world coordinate system and the Y axis; the roll angle γ is the angle between the Z1 axis and the vertical plane containing the Y1 axis. The rotational relationship between the object coordinate system and the world coordinate system is carried out in the order of yaw angle, pitch angle and roll angle, that is, first rotate by (yaw angle) around the Z1 axis, then rotate by θ (pitch angle) around the X1 axis on this basis, and finally rotate by γ (roll angle) around the Y1 axis on the basis of the previous two rotations.

[0072] For another example, the attitude data can also be represented by a rotation matrix. In space, any complex angular position relationship between two coordinate systems can be regarded as a combination of a finite number of basic rotations, and their corresponding transformation matrices are equal to the product of the transformation matrices determined by their basic rotations (that is, rotation around the X axis, rotation around the Y axis and rotation around the Z axis), and the multiplication order is arranged from right to left according to the order of the basic rotations. According to robotics, the rotation matrix can indicate the following information: ① the attitude representation of the world coordinate system 0 in the object coordinate system 1; ② the transformation matrix from the world coordinate system 0 to the object coordinate system 1; ③ the mapping relationship from the object coordinate system 1 to the world coordinate system 0. These three meanings may be mentioned later, and will not be elaborated hereafter. For example, for the above-mentioned yaw angle pitch angle θ and roll angle γ, the rotation matrix from the object coordinate system to the world coordinate system can be directly obtained by transformation Those skilled in the art should be aware of the conventional conversion methods between the rotation matrix and the Euler angles, so the present disclosure will not elaborate here.

[0073] For another example, the attitude data can also be represented by quaternions. Quaternions and rotation matrices / Euler angles can be converted to each other. Compared with Euler angle operations, quaternion operations can reduce the amount of calculation and avoid the Euler angle deadlock phenomenon. A quaternion is a hypercomplex number and can be regarded as a vector in four-dimensional space, which can describe the fixed-point rotation of a rigid body. It consists of four elements q0, q1, q2, q3, including three imaginary parts i, j, k and one real part. Those skilled in the art should be aware of the rotation matrix / the conventional conversion methods between Euler angles and quaternions, so the present disclosure will not elaborate here.

[0074] For example, the position data corresponding to the mass point com can be represented by the vector of the mass point com relative to the origin of the world coordinate system in the world coordinate system. For example, the position data of the center of mass com can be expressed as: x0, y0, z0 are the projections of the mass point com on the X, Y, and Z axes of the world coordinate system respectively. Similarly, the position data corresponding to the pivot point P can be represented by the vector of the pivot point P relative to the origin of the world coordinate system in the world coordinate system. For example, the position data of the pivot point P can be expressed as: x, y, and z are the projections of the pivot point P on the X, Y, and Z axes of the world coordinate system respectively. The position data corresponding to the pivot point P can also be represented by adding the position data of the mass point com to the offset of the pivot point P relative to the center of mass com. to represent, is based on the object coordinate system. For example, the position data of the pivot point P can be represented as:

[0075]

[0076] where is the rotation matrix indicating the mapping relationship from the object coordinate system to the world coordinate system as described above. In the representation methods of each of the above and following data, if the upper left superscript is 0, it indicates that the data is in the world coordinate system, and if the upper left superscript is 1, it indicates that the data is in the object coordinate system. The rotation matrix can be directly obtained by conversion according to the above Euler angles and / or quaternions, or directly obtained through observation acquisition / calculation.

[0077] Continue to refer to Figure 2 , the flight trajectory of the center of mass of the irregular object is a parabola with the Z axis upward in the world coordinate system (as shown by the dotted line in its right figure), and the flight trajectory of the pivot point of the irregular object is an irregular convex curve in the world coordinate system (such as Figure 2 shown by the solid line in the right figure). Those skilled in the art should understand that examples of irregular objects can also include objects such as bottles, tennis rackets, and irregular stones that cannot be completely equivalent to balls, cuboids, and cubes. Although the present disclosure is described by taking a hammer as an example, the present disclosure is not limited thereto.

[0078] Among them, the attitude data of the irregular object in flight during the first time period and the position data of the pivot point of the irregular object satisfy the constraint conditions and the correlation relationships between the parameters corresponding to the dynamic model and the kinematic model of the irregular object. The dynamic model can be used to characterize the relationship between the motion trajectory (for example, the pose) of the irregular object and the force condition of the irregular object. The kinematic model can be used to characterize the relationship between the position of the irregular object and its linear velocity and angular velocity. The present disclosure does not limit the specific construction methods of the dynamic model or the kinematic model herein, as long as they can be used to predict the flight trajectory of the irregular object.

[0079] Compared with regular objects, the dynamic model of irregular objects is relatively complex. Figure 3Shows the angular velocity of the observed hammer rotating around the X, Y, and Z axes in the object coordinate system during flight. The vertical axis is the angular velocity of the three axes, with the unit of (rad / s); the horizontal axis is time, with the unit of seconds. For example, 2.35711 on the horizontal axis indicates the moment of 23571.1 seconds. In addition, as Figure 3 shown, Figure 3 the observed data in

[0080] contains a large amount of noise, which also causes difficulties in predicting the attitude of irregular objects. Figure 3 Further referring to

[0081] , from the perspective of the attitude change in the hammer flight trajectory, the hammer without external torque does not exhibit a stable three-axis angular velocity during flight, but fluctuates continuously and even flips rapidly, which conforms to the intermediate axis theorem (also known as the tennis racket theorem).

[0082] Optionally, the embodiments of the present disclosure further fit the dynamic model of the irregular object through the intermediate axis theorem. The embodiments of the present disclosure solve the parameters in the dynamic model based on the attitude data of the irregular object in the first time period during flight and the position data of the pivot point of the irregular object. Then, reference will be made to Figures 4 - 7B to further describe examples of determining the parameters in the dynamic model and the kinematic model, and the present disclosure will not elaborate herein.

[0083] Thus, through step S110, the dynamic model and the kinematic model of the irregular object can be determined. Optionally, the parameters of the dynamic model corresponding to the irregular object include the three-axis inertia parameters in the object coordinate system. Optionally, the parameters of the dynamic model corresponding to the irregular object further include the viscous resistance parameters based on air resistance. Optionally, the parameters of the kinematic model corresponding to the irregular object include the initial linear velocity in the world coordinate system, the offset of the pivot point relative to the centroid, and the initial position. Then, reference will be made to Figures 4 to 7B to further describe these parameters, and details will not be elaborated here.

[0084] Next, in step S120, based on the dynamic model corresponding to the irregular object, determine the predicted attitude data corresponding to one or more moments of the pivot point in the second time period. The above-mentioned second time period may be a set of several moments before or after the first time period, and it includes at least one moment. Although the embodiments of the present disclosure only predict the attitude data of a certain second time period / a certain moment, those skilled in the art should understand that the attitude data of multiple second time periods can also be predicted in step S120, and the present disclosure is not limited thereto. For example, according to the dynamic model corresponding to the irregular object, the angular acceleration of the pivot point at each moment can be determined, and thus the angular velocity of the pivot point at any moment can be correspondingly solved by integration, and thus the attitude data of the pivot point at any moment can be predicted.

[0085] In step S130, based on the predicted attitude data corresponding to one or more moments of the pivot point in the second time period and the kinematic model, determine the predicted position data corresponding to one or more moments of the pivot point in the second time period.

[0086] As described above, the flight trajectory of the center of mass of the irregular object is a parabola with the Z-axis upward in the world coordinate system. Therefore, the position changes of the center of mass of the irregular object in the X-axis and Y-axis directions can be equivalent to uniform linear motion. Therefore, the relevant models of uniform linear motion can be used to predict the predicted position data of the irregular object in the second time period for the X-axis and Y-axis directions. The position change of the center of mass of the irregular object in the Z-axis direction can be equivalent to the relevant model of motion with a constant acceleration. Therefore, the constant acceleration model can be used to predict the predicted position data of the irregular object in the second time period for the Z-axis direction.

[0087] For another example, the flight trajectory of the pivot point of the irregular object is an irregular convex curve in the world coordinate system. However, the offset of the pivot point of the irregular object relative to the center of mass is invariant for a rigid body. Therefore, after solving the offset of the pivot point P relative to the center of mass com by using the attitude data of the irregular object in flight in the first time period and the position data of the pivot point of the irregular object as described above subsequently, the above formula (1) can be used to determine the predicted position data of the pivot point P in the second time period.

[0088] In addition, the present disclosure correspondingly discloses an apparatus for determining the flight trajectory of an irregular object, the apparatus including: a parameter determination module configured to determine parameters in a kinematic model and a dynamic model corresponding to the irregular virtual object based on attitude data of the irregular object in flight during a first time period and position data of a pivot point of the irregular object; a trajectory prediction module configured to determine predicted attitude data corresponding to one or more moments in a second time period for the pivot point based on the dynamic model corresponding to the irregular virtual object; and determine predicted position data corresponding to one or more moments in the second time period for the pivot point based on the predicted attitude data corresponding to one or more moments in the second time period for the pivot point and the kinematic model.

[0089] In an embodiment of the present disclosure, only by observing the position data and attitude data of an irregular object at a certain time period or certain positions during flight, it is possible to determine the future flight trajectory of the object, including the linear velocity, angular velocity, position, and attitude of the object within a future period of time. In addition, the present disclosure also uses methods of rigid body kinematics, rigid body dynamics, and aerodynamics to achieve high-precision prediction of the flight trajectory of an irregular object with a relatively low computational load. Compared with the prior art solutions that require collecting a large number of motion trajectories of irregular objects in advance as samples to predict the flight trajectory of irregular objects, the present disclosure greatly reduces the computational load.

[0090] The following refers to Figures 4 to 7B to describe an example of determining each parameter in the dynamic model. Figure 4 FIG. shows another flowchart of a method for determining the flight trajectory of an irregular object according to an embodiment of the present disclosure. Figure 5 FIG. shows a flowchart of determining each parameter in the dynamic model according to an embodiment of the present disclosure. Figure 6A FIG. shows a comparison diagram of the angular velocities of three axes predicted by a dynamic model corresponding to Euler's equation without external torque and the actual angular velocities of the three axes according to an embodiment of the present disclosure. Figure 6B FIG. shows a comparison diagram of the rotational kinetic energy predicted by a dynamic model corresponding to Euler's equation without external torque and the actual rotational kinetic energy according to an embodiment of the present disclosure. Figure 7A FIG. shows a comparison diagram of the angular velocities of three axes predicted by a dynamic model corresponding to Euler's equation with air resistance and the actual angular velocities of the three axes according to an embodiment of the present disclosure. Figure 7B FIG. shows a comparison diagram of the rotational kinetic energy predicted by a dynamic model corresponding to Euler's equation with air resistance and the actual rotational kinetic energy according to an embodiment of the present disclosure.

[0091] Just as Figure 3As shown, the data obtained through observation includes a large amount of noise. For example, the observation data in the first time period obtained through the built-in motion capture system of the robot may contain a large amount of noise. Hereinafter, the attitude data of the irregular object at multiple moments in the first time period and the position data of the pivot point of the irregular object are also referred to as observation data.

[0092] Optionally, as Figure 4 shown, in step S110, the attitude data of the irregular object in flight at multiple moments in the first time period (for example, the attitude data represented by quaternions (q′0, q′1, q′2, q′3) at multiple first moments), the position data of the pivot point of the irregular object (for example, x′, y′, z′) can be obtained, and then based on the attitude data and position data at multiple moments in the first time period, the smooth attitude data of the irregular object in flight at multiple moments in the first time period (for example, multiple (q0, q1, q2, q3)) and the smooth position data of the pivot point of the irregular object (for example, Figure 4 and Figure 2 x, y, z in) can be determined. Those skilled in the art should understand that although the rotation matrix / Euler angles can be directly obtained through observation in S110, due to the possible gimbal lock problem of Euler angles and the large amount of data of the rotation matrix, which is not conducive to the transmission and storage of data by the motion capture system, the motion capture system of the present disclosure adopts the scheme of capturing quaternions as the attitude data. The present disclosure is not limited thereto.

[0093] For example, various filters can be used to filter the attitude data at multiple moments in the first time period and the position data of the pivot point of the irregular object, so as to remove the noise information from the attitude data and position data. Optional filters include Kalman filters, SVR filters, Fourier fitting filters, and so on. The embodiments of the present disclosure do not limit the types of filters, as long as they are helpful for removing noise information. For another example, the least squares method can also be used to fit the smooth curve corresponding to the observation data at multiple moments in the first time period, and the values on this smooth curve are used as the observation data after removing noise. The observation data at multiple moments in the first time period processed by the filter and / or the least squares method will be approximately located on a smooth curve, and at this time, it is also called smooth observation data. For example, smooth attitude data (for example, q0, q1, q2, q3) and smooth position data (for example, Figure 4 and Figure 2 x, y, z in).

[0094] Next, further refer to Figure 4, the quaternion sequence of the first time period can be determined by smoothing the observed data as the attitude data of the first time period. For example, step S110 may further include: determining the rotation matrices at multiple moments in the first time period based on the attitude data of the irregular object during flight, and each rotation matrix at the multiple moments indicates the mapping relationship from the object coordinate system to the world coordinate system at that moment. As described above, each set of quaternions q0, q1, q2, q3 can be directly converted into a 3*3 rotation matrix Next, based on the position data of the pivot point of the irregular object during flight in the first time period (e.g., Figure 4 and Figure 2 the x, y, z in) and the rotation matrices at multiple moments in the first time period the parameters in the kinematic model corresponding to the irregular object can be determined.

[0095] Among them, the parameters of the kinematic model corresponding to the irregular object include the initial linear velocity in the world coordinate system, the offset of the pivot point relative to the centroid, and the initial position. For example, as described above, the position data of the pivot point P in the world coordinate system can be represented by the above formula (1).

[0096]

[0097] Substituting the specific value into formula (1) can obtain the following formula (2).

[0098]

[0099] Among them, the position data of the centroid com is x0, y0, z0 are the projections of the mass point com on the XYZ axes of the world coordinate system respectively. Further, since the flight trajectory of the centroid of the irregular object is a parabola with the Z axis upward in the world coordinate system, the position data of the centroid com can be expressed as a function of time t. Specifically, the position changes of the centroid com of the irregular object in the X-axis and Y-axis directions can be equivalent to uniform linear motion, so the position data of the irregular object at time t for the X-axis and Y-axis directions can be predicted by the formulas x0 = a1t + b1 and y0 = a2t + b2 of uniform linear motion respectively. The position change of the centroid of the irregular object in the Z-axis direction can be equivalent to a relevant model of motion with a constant acceleration, and the acceleration is the gravitational acceleration 9.8m / s 2 , therefore, it can be expressed by the formula z0 = -4.9t 2+33t + b3. Here, 31, 32, and 33 are the initial linear velocities of the centroid in the positive directions of the X-axis, Y-axis, and Z-axis in the world coordinate system respectively, and b1, b2, and b3 are the initial positions of the centroid on the X-axis, Y-axis, and Z-axis in the world coordinate system respectively.

[0100] Therefore, among them, the kinematic model corresponding to the pivot point can be expressed as Formula (3) or Formula (4) or Formula (5). Formulas (3) to (5) are all deformations of Formula (1) and have the same physical meaning.

[0101]

[0102]

[0103]

[0104] Among them, Δx, Δy, and Δz are the offsets of the pivot point relative to the centroid on the X-axis, Y-axis, and Z-axis in the world coordinate system respectively, and x, y, and z are the initial positions of the centroid on the X-axis, Y-axis, and Z-axis in the world coordinate system respectively. is the rotation matrix between the world coordinate system and the object coordinate system at time t.

[0105] At time t in the first time period, the variables in Formulas (3) - (5): the position data of the pivot point P at time t in the first time period and the rotation matrix are all known. Therefore, each parameter in the kinematic model corresponding to the pivot point can be solved. For example, Formulas (3) to (5) can be deformed into Formula (6) to solve each parameter in the kinematic model: the initial linear velocity in the world coordinate system (for example, a1, a2, a3), the offset of the pivot point relative to the centroid (for example, Δx, Δy, Δz), and the initial position (for example, b1, b2, b3).

[0106]

[0107] Optionally, the solved parameters can be further calibrated in consideration of the error in the observation data or the fitting error in the smoothed observation data. For example, the above-mentioned step S110 can further include: based on the smoothed posture data of the irregular object in flight at multiple moments in the first time period and the smoothed position data of the pivot point of the irregular object, the least squares method is used to determine the parameters in the kinematic model corresponding to the irregular object. For example, the observation data at multiple moments in the first time period can be used to solve multiple a1 values, and then the final a1 can be determined by the least squares method. Other parameter values can also be similarly solved using the least squares method. Those skilled in the art should understand that the least squares method is only an example, and the present disclosure can also use other methods to similarly determine any parameters in the kinematic model / dynamic model.

[0108] Next, see Figure 5 To further describe how to determine the parameters in the dynamic model. Optionally, step S110 may also include: based on the posture data of the irregular object in flight, determining the rotation matrix at multiple moments in the first time period, wherein each of the rotation matrices at the multiple moments indicates the mapping relationship from the object coordinate system to the world coordinate system at the moment. Similarly, the posture data of the irregular object may be smooth posture data (e.g., Figure 4 q0, q1, q2, q3) in the above formula can also directly use the posture data of the first time period observed (for example, Figure 4 After mathematical transformation, the quaternion-based attitude data will be converted into rotation matrices at multiple moments in the first time period.

[0109] Then, in step S110, based on the rotation matrix at multiple moments in the first time period, the angular velocity and angular acceleration of the pivot point at at least one moment in the first time period are determined; and based on the angular velocity and angular acceleration of the pivot point at at least one moment in the first time period, the parameters of the dynamic model are determined.

[0110] For example, according to rigid body dynamics, the derivative of the rotation matrix with respect to time t can be used to obtain the angular velocity of the pivot point described in the form of the rotation matrix, and the derivative of the angular velocity of the pivot point with respect to time t can be used to obtain the angular acceleration of the pivot point. Then, based on the correspondence between the rotation matrix and the Euler angle, the angular velocity and angular acceleration of the three-axis rotation around the world coordinate system in the form of Euler angles can be obtained. Then, the angular velocity and angular acceleration of the three-axis rotation around the world coordinate system in the form of Euler angles can be further converted to obtain the angular velocity and angular acceleration of the three-axis rotation of the pivot point around the object coordinate system in the form of Euler angles.

[0111] The parameters of the determined kinetic model vary with different kinetic models.

[0112] For example, as an illustration, the Euler equation without external torque can be used to fit the kinetic model of the above irregular object during flight. Since the irregular object is not subject to external torque during flight, according to the intermediate axis theorem, the kinetic model corresponding to the Euler equation without external torque as shown in formula (7) can be established.

[0113]

[0114] where I x , I y , I z are the inertia parameters corresponding to the x-axis, y-axis, and z-axis in the object coordinate system respectively, are the angular accelerations corresponding to the x-axis, y-axis, and z-axis in the object coordinate system respectively, ω x , ω y , ω z are the angular velocities corresponding to the x-axis, y-axis, and z-axis in the object coordinate system respectively.

[0115] Further transforming formula (7) can obtain formula (8).

[0116]

[0117] where, ω x , ω y , ω z are all obtained by solving the attitude data in the above step S110. Therefore, the inertia parameters I x , I y , I z corresponding to the x-axis, y-axis, and z-axis in the object coordinate system can also be solved correspondingly. Thus, all the parameters in the kinetic model based on the Euler equation without external torque are determined.

[0118] For example, as another illustration, the Euler equation with air resistance can also be used to establish the kinetic model of the above irregular object during flight, and this kinetic model further takes into account the decrease in angular kinetic energy caused by air resistance. At this time, the kinetic model corresponding to the Euler equation with air resistance can be shown as formula (9).

[0119]

[0120] where N x , N y , N z respectively represent the air resistance generated when rotating around the three axes of the object coordinate system. The air resistance conforms to the hydrodynamic resistance equation (10).

[0121]

[0122] Among them, F D represents the air resistance, ρ represents the air density, C D represents the drag coefficient, v represents the relative velocity of the object with respect to the air, and A represents the relative area. Since the rotation process of the irregular object will cause changes in the relative velocity and relative area, the air resistance is essentially a force that dynamically changes with time. To simplify the model and reduce the amount of calculation, considering the direction of the air resistance, N x , N y , N z can be simplified by formula (11).

[0123]

[0124] Among them, k d is the viscous resistance parameter based on the air resistance. That is, according to formula (10) and formula (11), the dynamic model based on the Euler equation with air resistance can be further shown as formula (12) or formula (13). Formula (12) and formula (13) have the same meaning, and are only different deformations of the same formula.

[0125]

[0126]

[0127] Among them, ω x , ω y , ω z are all obtained by solving the attitude data in the above step S110. Therefore, the inertia parameters I x , I y , I z corresponding to the x-axis, y-axis, and z-axis in the object coordinate system can also be solved correspondingly. Thus, all the parameters in the dynamic model based on the Euler equation without external torque are determined.

[0128] Those skilled in the art should understand that although the present disclosure only exemplarily describes the determination schemes of the parameters of two dynamic models, the determination of the parameters of other dynamic models can also refer to the above schemes. Other dynamic models are, for example, dynamic models considering precise air resistance, dynamic models considering Magnus force (for example, for objects with relatively light mass), etc., and the present disclosure does not limit this.

[0129] Furthermore, continue to refer to Figure 5, Step S120 may further include: determining predicted angular velocities and predicted angular accelerations of the pivot point at one or more moments in a second time period based on angular velocities and angular accelerations of the pivot point at one or more moments in a first time period and a dynamic model corresponding to the pivot point. In step S110, the three-axis inertia parameters in the object coordinate system in the dynamic model corresponding to the pivot point have been determined based on the attitude data in the first time period, and optionally, the viscous resistance parameter based on air resistance. Therefore, based on multiple angular accelerations in the first time period and multiple ω in the first time period x , ω y , ω z , the predicted angular velocities and predicted angular accelerations (for rotations about the three axes of the object coordinate system) at one or more moments in the second time period can be obtained through integral operations.

[0130] For example, then, step S120 may further include: determining predicted rotation matrices corresponding to one or more moments in the second time period based on the predicted angular velocities and predicted angular accelerations of the pivot point at one or more moments in the second time period, where each predicted rotation matrix in the predicted rotation matrices corresponding to the one or more moments indicates the mapping relationship from the object coordinate system to the world coordinate system at the moment; and determining predicted attitude data corresponding to the pivot point at one or more moments in the second time period based on the predicted rotation matrices corresponding to the one or more moments in the second time period.

[0131] For example, the predicted angular velocities of the pivot point at one or more moments in the second time period (e.g., predicted angular velocities in the form of Euler angles) can be obtained through integral operations starting from a certain moment in the first time period. Then, the predicted angular velocities of the three-axis rotations in the form of rotation matrices of the pivot point at one or more moments in the second time period are converted. Integrating the predicted angular velocities in the form of rotation matrices can obtain the predicted attitude data corresponding to the pivot point at one or more moments in the second time period (e.g., predicted rotation matrices / predicted quaternion sequences corresponding to one or more moments in the second time period).

[0132] The following respectively refer to Figures 6A - 6B and Figures 7A - 7B to describe the fitting effects corresponding to different dynamic models.

[0133] See Figure 6A , which illustrates 6 curves of a dynamic model corresponding to Euler's equations without external torques, where the horizontal axis is time (in seconds) and the vertical axis is angular velocity (in rad / s). As Figure 6AAs shown by the markings, they are respectively the variation of the predicted angular velocity and the actual angular velocity of the XYZ axes in the world coordinate system over time. Refer to Figure 6B , which illustrates two curves. Among them, the horizontal axis is time (in seconds), and the vertical axis is the rotational kinetic energy (also known as angular kinetic energy, in joules). As Figure 6B shown by the markings, they are respectively the variation of the predicted rotational kinetic energy and the actual rotational kinetic energy in the world coordinate system over time. It can be seen that if the angular velocities of the three axes are predicted by the dynamic model of the Euler equation without external torque, there is a fitting error that increases over time when fitting the flight trajectory of the hammer, and it is unable to simulate the decreasing trend of the angular kinetic energy of the hammer, resulting in low accuracy. In addition, those skilled in the art should understand that Figure 6B in, the irregular decrease of the actual rotational kinetic energy is mainly because the identified moment of inertia of the object is not accurate enough (the fundamental reason is still the insufficient measurement or observation accuracy), so Figure 6B the data in can only roughly show that the actual rotational kinetic energy is decreasing, but it cannot accurately describe the process of the kinetic energy decrease.

[0134] Refer to Figure 7A , which illustrates six curves corresponding to the dynamic model of the Euler equation with air resistance. Among them, the horizontal axis is time (in seconds), and the vertical axis is the angular velocity (in rad / s). As Figure 7A shown by the markings, they are respectively the variation of the predicted angular velocity and the actual predicted angular velocity of the XYZ axes in the world coordinate system over time. Refer to Figure 7B , which illustrates two curves. Among them, the horizontal axis is time (in seconds), and the vertical axis is the rotational kinetic energy (also known as angular kinetic energy, in joules). As Figure 7B shown by the markings, they are respectively the variation of the predicted rotational kinetic energy and the actual rotational kinetic energy in the world coordinate system over time. It can be seen that if the angular velocities of the three axes are predicted by the dynamic model of the Euler equation with air resistance, the fitting error is significantly reduced and the accuracy is improved.

[0135] For example, then referring to Figure 4 , step S130 may further include: determining the predicted position data of the pivot point corresponding to one or more moments in the second time period based on the predicted rotation matrix corresponding to one or more moments in the second time period and the kinematic model. Combining the predicted attitude data of the above-mentioned second time period and the predicted position data corresponding to the second time period can obtain the flight motion trajectory.

[0136] Embodiments of the present disclosure only need to observe the position data and attitude data of an irregular object at certain moments or certain positions during flight, and can determine the future flight trajectory of the object, including the linear velocity, angular velocity, position, and attitude of the object within a future period of time. In addition, the present disclosure also uses methods of rigid body kinematics, rigid body dynamics, and aerodynamics to achieve high-precision prediction of the flight trajectory of irregular objects with a relatively low computational load. Compared with the prior art solutions that need to collect a large number of motion trajectories of irregular objects in advance as samples to predict the flight trajectory of irregular objects, the present disclosure greatly reduces the computational load.

[0137] The following refers to Figures 8 to 10 Briefly describe an example scenario of applying the embodiments of the present disclosure. Figure 8 FIG. shows a schematic diagram of a method for displaying the flight trajectory of an irregular virtual object according to an embodiment of the present disclosure. Figure 9 FIG. shows a structural diagram of a robot for grasping an irregular object in flight according to an embodiment of the present disclosure. Figure 10 FIG. shows a schematic diagram of the process of attempting to grasp an irregular object.

[0138] Refer to Figure 8 , Figure 8 FIG. schematically depicts a user in a virtual reality environment in an immersive environment. The user wears a virtual reality headset, and a three-dimensional scene of a virtual hammer being thrown at him is displayed on the display screen of the virtual reality headset. The virtual hammer is an exemplary irregular virtual object. Those skilled in the art should understand that the irregular virtual object can also be in any other form.

[0139] Optionally, in Figure 8 the method for displaying the flight trajectory of the irregular virtual object (i.e., the virtual hammer) includes: determining the parameters in the kinematic model and dynamic model corresponding to the irregular virtual object based on the attitude data of the irregular virtual object during a first time period and the position data of the pivot point of the irregular virtual object; determining the predicted attitude data corresponding to one or more moments of the irregular virtual object during a second time period based on the dynamic model corresponding to the irregular virtual object; determining the predicted position data corresponding to one or more moments of the pivot point during the second time period based on the attitude data corresponding to one or more moments of the pivot point during the second time period and the kinematic model; and rendering the flight images of the irregular virtual object at one or more moments during the second time period and displaying the flight images based on the predicted attitude data corresponding to one or more moments of the irregular virtual object during the second time period and the predicted position data corresponding to one or more moments of the pivot point during the second time period.

[0140] For example, the parameters in the kinematic model and the dynamic model for determining the irregular virtual object, the predicted attitude data corresponding to one or more moments of the irregular virtual object in the second time period, and the predicted position data corresponding to one or more moments of the pivot point in the second time period are similar to steps S110 to S130 in method 100, and the present disclosure will not elaborate herein.

[0141] Correspondingly, the present disclosure also discloses a device for displaying the flight trajectory of an irregular virtual object, including: a processor configured to: determine the parameters in the kinematic model and the dynamic model corresponding to the irregular virtual object based on the attitude data of the irregular virtual object in flight during the first time period and the position data of the pivot point of the irregular virtual object; determine the predicted attitude data corresponding to one or more moments of the irregular virtual object in the second time period based on the dynamic model corresponding to the irregular virtual object; determine the predicted position data corresponding to one or more moments of the pivot point in the second time period based on the attitude data corresponding to one or more moments of the pivot point in the second time period and the kinematic model; and render the flight images corresponding to one or more moments of the irregular virtual object in the second time period based on the attitude data corresponding to one or more moments of the irregular virtual object in the second time period and the predicted position data corresponding to one or more moments of the pivot point in the second time period; a display screen configured to: display the flight images. Optionally, the above-mentioned processor may be provided on the VR glasses or may be provided separately from the VR glasses.

[0142] Compared with the existing method for displaying the flight trajectory of an irregular virtual object in a virtual display scenario, the embodiments of the present disclosure can display a flight trajectory of an irregular object closer to the real scenario, thereby bringing an immersive experience to the user.

[0143] Reference Figure 9 , the robot 900 for grasping an irregular object in flight according to an embodiment of the present disclosure has: an operation unit 920 for grasping the irregular object in flight; a controller 910 for controlling the operation unit to adjust its pose, the controller being provided on the robot and configured to execute method 100. According to a more detailed embodiment of the robot of the present disclosure, the controller 910 may be implemented, for example, as any device capable of executing method 100, including but not limited to FPGA, DSP, ARM single-chip microcomputer, CPU, etc. For more details on the functions of the controller 910, reference may be made to the description of method 100 of the present disclosure above, which will not be elaborated herein.

[0144] According to another optional design of the robot for grasping irregular objects according to the present disclosure, considering the computing power of the robot itself is limited or restricted, instead of analyzing the grasping position by the controller of the robot itself, dynamic analysis can be performed by an additional computing device arranged outside the robot, and the analysis result of the grasping position, that is, the grasping position data corresponding to different moments during movement, is sent to the robot as a grasping instruction. Thus, even a robot with limited computing power can also execute the method for grasping irregular objects according to the present disclosure.

[0145] Reference Figure 10 , which further shows the schematic flight process of the hammer (or virtual hammer). Multiple grasping positions (or virtual grasping positions) are shown by small gray hammers, and the three lines on the hammer are the three coordinate axes of the schematic object coordinate system. Figure 10 The three coordinate axes of [] are the XYZ axes of the world coordinate system, with the unit of meter. When the robot / the user wearing the VR glasses observes the hammer or virtual hammer, the flight progress of the hammer or virtual hammer has reached 10%, and it has flown for 0.983 seconds and 4500 millimeters. At this time, the estimated landing point of the hammer or virtual hammer is 49.9 millimeters away from the actual landing point of the hammer or virtual hammer, and the estimated rotation angles of the hammer or virtual hammer when landing relative to the XYZ axes of the world coordinate system of the actual rotation angles of the hammer or virtual hammer when landing have errors of 8.7°, 0.8°, and -10.3° respectively. As the flight process continues, as shown in the right figure of Figure 10 , both the distance error and the angle error are gradually decreasing. That is, the robot / the user wearing the VR glasses can continuously adjust their estimation of the pose of the hammer or virtual hammer, so as to achieve a more accurate grasp of the flying hammer or virtual hammer.

[0146] In summary, in addition, the present disclosure also provides an electronic device, including: a processor; a memory, and the memory stores computer instructions, and when the computer instructions are executed by the processor, the above-mentioned method is implemented.

[0147] In addition, the present disclosure also provides a computer-readable storage medium, on which computer instructions are stored, and when the computer instructions are executed by the controller, the above-mentioned method is implemented.

[0148] According to another aspect of the present disclosure, a computer program product or computer program is provided, and the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The controller of the robot reads the computer instructions from the computer-readable medium, and the controller executes the computer instructions, so that the robot executes the methods provided in the above-mentioned various aspects or various optional implementation manners of the above-mentioned various aspects.

[0149] In summary, the present disclosure provides a method for determining the flight trajectory of an irregular object, a method for displaying the flight trajectory of an irregular virtual object, a device for displaying the flight trajectory of an irregular virtual object, a device for determining the flight trajectory of an irregular object, a robot for grasping an irregular object in flight, an electronic device, and a computer-readable storage medium. The present disclosure only needs to observe the position data and attitude data of an irregular object at a certain period or certain positions during the flight process, and can determine the future flight trajectory of the object, including the linear velocity, angular velocity, position, and attitude of the object within a certain period in the future.

[0150] In addition, the present disclosure also uses the methods of rigid body kinematics, rigid body dynamics, and aerodynamics to achieve high-precision prediction of the flight trajectory of an irregular object with a relatively low computational load. Compared with the prior art solutions that need to collect a large number of motion trajectories of irregular objects in advance as samples to predict the flight trajectory of irregular objects, the present disclosure greatly reduces the computational load.

[0151] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art should understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for determining the flight trajectory of an irregular object, comprising: Determining parameters in the kinematic model and dynamic model corresponding to the irregular object based on the attitude data of the irregular object in flight during a first time period and the position data of the pivot point of the irregular object; Determining predicted attitude data corresponding to one or more moments in a second time period based on the dynamic model corresponding to the irregular object; And Determining predicted position data corresponding to one or more moments in the second time period based on the predicted attitude data corresponding to one or more moments in the second time period of the pivot point and the kinematic model; Wherein, the determining of the predicted attitude data corresponding to one or more moments in the second time period of the pivot point includes: Determining the predicted angular velocity and predicted angular acceleration of the pivot point at one or more moments in the second time period based on the angular velocity and angular acceleration of the pivot point at one or more moments in the first time period and the dynamic model corresponding to the pivot point; Determining a predicted rotation matrix corresponding to one or more moments in the second time period based on the predicted angular velocity and predicted angular acceleration of the pivot point at one or more moments in the second time period, and each predicted rotation matrix in the predicted rotation matrix corresponding to the one or more moments indicates the mapping relationship from the object coordinate system to the world coordinate system at the moment; and Determining the predicted attitude data corresponding to one or more moments in the second time period of the pivot point based on the predicted rotation matrix corresponding to one or more moments in the second time period.

2. The method according to claim 1, wherein, The flight trajectory of the center of mass of the irregular object is a parabola with the Z-axis upward in the world coordinate system, and the flight trajectory of the pivot point of the irregular object is an irregular convex curve in the world coordinate system.

3. The method according to claim 1, wherein, The parameters of the dynamic model corresponding to the irregular object include the three-axis inertia parameters in the object coordinate system and the viscous drag parameters based on air resistance.

4. The method according to claim 1, wherein, The parameters of the kinematic model corresponding to the irregular object include the initial linear velocity in the world coordinate system, the offset of the pivot point relative to the center of mass, and the initial position.

5. The method according to any one of claims 1-4, wherein, The determining of the parameters in the kinematic model and dynamic model corresponding to the irregular object further includes: Determining the rotation matrix at multiple moments in the first time period based on the attitude data of the irregular object in flight during the first time period, and each rotation matrix at each of the multiple moments indicates the mapping relationship from the object coordinate system to the world coordinate system at the moment; and Determining the parameters in the kinematic model corresponding to the irregular object based on the position data of the pivot point of the irregular object in flight during the first time period and the rotation matrix at multiple moments in the first time period.

6. The method according to any one of claims 1-4, wherein The determining of the parameters in the kinematic model and dynamic model corresponding to the pivot point further includes: Determining the rotation matrix at multiple moments in the first time period based on the attitude data of the irregular object in flight, and each rotation matrix at each of the multiple moments indicates the mapping relationship from the object coordinate system to the world coordinate system at the moment; Determine the angular velocity and angular acceleration of the pivot point rotating about the three axes of the object coordinate system at at least one moment in the first time period based on the rotation matrices at multiple moments in the first time period; and Determine the parameters of the dynamic model based on the angular velocity and angular acceleration of the pivot point rotating about the three axes of the object coordinate system at at least one moment in the first time period.

7. The method according to any one of claims 1-4, wherein The determining of the parameters in the kinematic model and the dynamic model corresponding to the pivot point further includes: Obtain the attitude data of the irregular object in flight at multiple moments in the first time period and the position data of the pivot point of the irregular object. Based on the attitude data and position data at multiple moments in the first time period, determine the smooth attitude data of the irregular object in flight at multiple moments in the first time period and the smooth position data of the pivot point of the irregular object; and Based on the smooth attitude data of the irregular object in flight at multiple moments in the first time period and the smooth position data of the pivot point of the irregular object, use the least squares method to determine the parameters in the kinematic model corresponding to the irregular object.

8. The method according to claim 1, wherein, The determining of the predicted position data corresponding to the pivot point at one or more moments in the second time period further includes: Based on the predicted rotation matrix corresponding to one or more moments in the second time period and the kinematic model, determine the predicted position data corresponding to the pivot point at one or more moments in the second time period.

9. The method according to claim 3, wherein, The dynamic model is: Among them, I x , I y , I z are the inertia parameters corresponding to the x-axis, y-axis, and z-axis in the object coordinate system respectively, and k d is the viscous resistance parameter based on air resistance, are the angular accelerations corresponding to the x-axis, y-axis, and z-axis in the object coordinate system respectively, and ω x , ω y , ω z are the angular velocities corresponding to the x-axis, y-axis, and z-axis in the object coordinate system respectively.

10. The method according to claim 4, wherein, The kinematic model is: Wherein, a1, a2, and a3 are respectively the initial linear velocities of the centroid in the positive directions of the X-axis, Y-axis, and Z-axis in the world coordinate system, b1, b2, and b3 are respectively the initial positions of the centroid on the X-axis, Y-axis, and Z-axis in the world coordinate system, Δx, Δy, and Δz are respectively the offsets of the pivot point relative to the centroid on the X-axis, Y-axis, and Z-axis in the object coordinate system, and x, y, and z are respectively the projections of the pivot point on the X-axis, Y-axis, and Z-axis of the world coordinate system at time t. is the rotation matrix from the world coordinate system to the object coordinate system at time t.

11. A method for displaying the flight trajectory of an irregular virtual object, including: Determine the parameters in the kinematic model and the dynamic model corresponding to the irregular virtual object based on the attitude data of the irregular virtual object in flight in the first time period and the position data of the pivot point of the irregular virtual object; Determine the predicted attitude data corresponding to the irregular virtual object at one or more moments in the second time period based on the dynamic model corresponding to the irregular virtual object; Based on the attitude data corresponding to the pivot point at one or more moments in the second time period and the kinematic model, determine the predicted position data corresponding to the pivot point at one or more moments in the second time period; And Based on the predicted attitude data corresponding to the irregular virtual object at one or more moments in the second time period and the predicted position data corresponding to the pivot point at one or more moments in the second time period, render the flight screen of the irregular virtual object in the second time period and display the flight screen; Wherein, the determining of the predicted attitude data corresponding to the pivot point at one or more moments in the second time period includes: Based on the angular velocity and angular acceleration of the pivot point at one or more moments in the first time period and the dynamic model corresponding to the pivot point, determine the predicted angular velocity and predicted angular acceleration of the pivot point at one or more moments in the second time period; Determine a predicted rotation matrix corresponding to one or more moments in a second time period based on the predicted angular velocity and predicted angular acceleration of the pivot point at one or more moments in the second time period, where each predicted rotation matrix in the predicted rotation matrices corresponding to the one or more moments indicates the mapping relationship from the object coordinate system to the world coordinate system at that moment; and Determine predicted pose data corresponding to one or more moments in the second time period for the pivot point based on the predicted rotation matrices corresponding to the one or more moments in the second time period.

12. A device for displaying a flight trajectory of an irregular virtual object, comprising: A processor, configured to: Determine parameters in the kinematic model and dynamic model corresponding to the irregular virtual object based on the pose data of the irregular virtual object in a first time period and the position data of the pivot point of the irregular virtual object; Determine predicted pose data corresponding to one or more moments in a second time period for the irregular virtual object based on the dynamic model corresponding to the irregular virtual object; Determine predicted position data corresponding to one or more moments in the second time period for the pivot point based on the pose data corresponding to the one or more moments in the second time period for the pivot point and the kinematic model; and And Render a flight screen of the irregular virtual object in the second time period based on the pose data corresponding to the one or more moments in the second time period for the irregular virtual object and the predicted position data corresponding to the one or more moments in the second time period for the pivot point, A display screen, configured to: display the flight screen, wherein determining the predicted pose data corresponding to one or more moments in the second time period for the pivot point includes: Determine the predicted angular velocity and predicted angular acceleration of the pivot point at one or more moments in the second time period based on the angular velocity and angular acceleration of the pivot point at one or more moments in the first time period and the dynamic model corresponding to the pivot point; Determine a predicted rotation matrix corresponding to one or more moments in the second time period based on the predicted angular velocity and predicted angular acceleration of the pivot point at one or more moments in the second time period, where each predicted rotation matrix in the predicted rotation matrices corresponding to the one or more moments indicates the mapping relationship from the object coordinate system to the world coordinate system at that moment; and Determine the predicted pose data corresponding to one or more moments in the second time period for the pivot point based on the predicted rotation matrices corresponding to the one or more moments in the second time period.

13. A device for determining a flight trajectory of an irregular object, comprising: A parameter determination module, configured to: determine parameters in the kinematic model and dynamic model corresponding to the irregular object based on the pose data of the irregular object in a first time period and the position data of the pivot point of the irregular object; A trajectory prediction module, configured to: Determine predicted pose data corresponding to one or more moments in the second time period for the pivot point based on the dynamic model corresponding to the irregular virtual object; Based on the predicted attitude data corresponding to one or more moments in the second time period of the pivot point and the kinematic model, determine the predicted position data corresponding to the pivot point at one or more moments in the second time period. Wherein, the determination of the predicted attitude data corresponding to the pivot point at one or more moments in the second time period includes: Based on the angular velocity and angular acceleration of the pivot point at one or more moments in the first time period and the dynamic model corresponding to the pivot point, determine the predicted angular velocity and predicted angular acceleration of the pivot point at one or more moments in the second time period; Based on the predicted angular velocity and predicted angular acceleration of the pivot point at one or more moments in the second time period, determine the predicted rotation matrix corresponding to one or more moments in the second time period, and each predicted rotation matrix in the predicted rotation matrix corresponding to one or more moments indicates the mapping relationship from the object coordinate system to the world coordinate system at that moment; and Based on the predicted rotation matrix corresponding to one or more moments in the second time period, determine the predicted attitude data corresponding to the pivot point at one or more moments in the second time period.

14. A robot for grasping an irregular object in flight, the robot having: An operating part for grasping the irregular object in flight; A controller for controlling the operating part to adjust its pose, the controller being provided on the robot and configured to execute the method according to any one of claims 1 to 10.

15. An electronic device, comprising: A processor; A memory storing computer instructions, which when executed by the processor implement the method according to any one of claims 1 - 11.

16. A computer-readable storage medium, having stored thereon computer instructions, which when executed by a processor implement the method according to any one of claims 1 - 11.

Images