Smart mobile terminal and control method, control device, computer program product thereof

Abstract

The application relates to an intelligent mobile terminal and a control method, a control device and a computer program product thereof, and comprises the following steps: acquiring Euler angles and accelerations detected by a plurality of inertial measurement units; wherein the plurality of inertial measurement units are arranged on the body of an operator one by one, and the plurality of inertial measurement units correspond to a plurality of terminal components one by one; determining the moving direction and the moving speed of the plurality of terminal components according to the Euler angles and the accelerations detected by the plurality of inertial measurement units; and generating a control instruction for controlling the movement of the plurality of terminal components according to the moving direction and the moving speed of the plurality of terminal components. In the application, the operator can remotely control the movement of the intelligent mobile terminal through the body posture, and does not need to hold an operation handle with both hands, so that both hands can be deftly used to remotely control the grasping of objects, thereby improving the operation experience of the intelligent mobile terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent mobile terminal, in particular to an intelligent mobile terminal, a control method and device thereof, and a computer program product. BACKGROUND

[0002] The intelligent mobile terminal refers to a mobile device with an intelligent operating system. The intelligent mobile terminal can interact with an operator and move and perform tasks according to the instructions issued by the operator. Some intelligent mobile terminals have a mechanical arm for performing the task of grabbing objects. Currently, the operation of the intelligent mobile terminal is generally active control movement, using a gamepad or other controller to control the movement of the intelligent mobile terminal. However, this control method limits the use of the operator's hands, so that the operator's hands can only hold the handle during remote control and cannot perform additional hand operations. SUMMARY

[0003] The present application aims to provide an intelligent mobile terminal, a control method and device thereof, and a computer program product, which can control the movement of the intelligent mobile terminal without an operation handle, so that the user's hands can perform other operations simultaneously during the movement of the intelligent mobile terminal.

[0004] To achieve the above-mentioned purpose, according to the first aspect of the present application, a control method of an intelligent mobile terminal is provided, the intelligent mobile terminal is composed of a plurality of terminal components, and the method comprises:

[0005] obtaining Euler angles and accelerations detected by a plurality of inertial measurement units; wherein the plurality of inertial measurement units are respectively arranged on the body of an operator, and the plurality of inertial measurement units correspond one-to-one to the plurality of terminal components;

[0006] determining the moving direction and speed of the plurality of terminal components according to the Euler angles and accelerations detected by the plurality of inertial measurement units;

[0007] generating a control instruction for controlling the movement of the plurality of terminal components according to the moving direction and speed of the plurality of terminal components.

[0008] According to the second aspect of the present application, a control device of an intelligent mobile terminal is provided, comprising:

[0009] a perception information acquisition module, configured to obtain Euler angles and accelerations detected by a plurality of inertial measurement units; wherein the plurality of inertial measurement units are respectively arranged on the body of an operator, and the plurality of inertial measurement units correspond one-to-one to the plurality of terminal components;

[0010] a moving state determination module, configured to determine the moving direction and speed of the plurality of terminal components according to the Euler angles and accelerations detected by the plurality of inertial measurement units.

[0011] a control instruction generation module configured to generate control instructions for controlling the movement of the plurality of terminal components according to the moving direction and the moving speed of the plurality of terminal components.

[0012] According to a third aspect of the present application, an intelligent mobile terminal is provided, comprising the intelligent mobile terminal control device and the plurality of terminal components.

[0013] According to a fourth aspect of the present application, a computer program product is provided, comprising computer program instructions for instructing a computer device to perform operations corresponding to the above method.

[0014] The intelligent mobile terminal and its control method, control device and computer program product provided by the present application have the following beneficial effects:

[0015] The traditional remote control method of the intelligent mobile terminal requires the operator to hold a gamepad or other controller, which limits the flexibility of the hands. However, the method of the present application allows the operator to directly control the intelligent mobile terminal through body posture without holding any device, thereby greatly improving the convenience of operation. Using body posture as control input is more in line with the natural behavior habits of human beings, making the operation more intuitive and easy to understand. Since the hands are freed, the operator can perform other operations such as grabbing objects or using other tools while controlling the intelligent mobile terminal. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 A flow chart of an intelligent mobile terminal control method in an embodiment of the present application.

[0018] Figure 2 A schematic diagram of generating control instructions in an embodiment of the present application.

[0019] Figure 3 A device schematic diagram of an intelligent mobile terminal control device in an embodiment of the present application.

[0020] Figure 4 A device schematic diagram of an intelligent mobile terminal in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The detailed description of the drawings is intended as an explanation of the current embodiments of the application, and is not intended to represent the only forms in which the application can be practiced. It is to be understood that the same or equivalent functions can be accomplished by different embodiments that are intended to be encompassed within the spirit and scope of the application.

[0022] Referring to Figure 1 One embodiment of the present application provides a smart mobile terminal control method, the smart mobile terminal is composed of a plurality of terminal components, the method comprises:

[0023] Step S10, obtaining the Euler angles and accelerations detected by a plurality of inertial measurement units; wherein the plurality of inertial measurement units are respectively arranged on the body of the operator, and the plurality of inertial measurement units correspond one-to-one to the plurality of terminal components.

[0024] Specifically, the execution subject of the method of the embodiment can be a control device installed on the smart mobile terminal, which can receive the sensing information of the plurality of inertial measurement units (IMUs) through Bluetooth. IMU is a sensor that can detect the motion state of an object, which can measure the acceleration, angular velocity and magnetic field of the object in three-dimensional space, etc. The IMU usually outputs a quaternion to represent the direction of the object (in this embodiment, the body posture of the operator). The quaternion is a mathematical method for representing and operating rotation in three-dimensional space. Based on the quaternion output by the IMU, the corresponding Euler angles can be converted. Euler angles are three angles describing the orientation of an object relative to a reference coordinate system, including pitch, yaw and roll. These angles can be used to determine the orientation of the object in three-dimensional space.

[0025] The acceleration output by the IMU is a three-dimensional vector, usually represented as (a x , a y , a z ), a x , a y , a z represent the acceleration components of the x-axis, y-axis and z-axis in the coordinate system respectively. Acceleration provides the change in velocity of an object in each direction. In the IMU, the accelerometer measures the acceleration of the object relative to the free-fall state, which includes the gravitational acceleration g and the acceleration caused by motion.

[0026] The plurality of IMUs are placed on different positions of the body of the operator, such as wrists, chests, ankles, etc. The specific position depends on the components of the smart mobile terminal that need to be controlled. Each IMU corresponds to one terminal component of the smart mobile terminal, that is, if the smart mobile terminal has multiple parts that can move independently, each part will be controlled by a specific IMU.

[0027] Step S20: Determine the moving direction and moving speed of the multiple terminal components based on the Euler angles and accelerations detected by the multiple inertial measurement units.

[0028] Specifically, based on Euler angles and acceleration, the direction in which the end effector should move can be determined. For example, if the robot arm needs to move upward, the pitch angle can determine how many degrees the arm needs to rotate upward; and the acceleration can determine the speed of the end effector in each direction.

[0029] Step S30: Generate control commands for controlling the movement of the plurality of terminal components based on their movement direction and speed.

[0030] Specifically, after determining the direction and speed of movement of each terminal component, control commands can be generated to control each terminal component. These control commands are used to drive motors or actuators, causing the terminal components to move in a predetermined direction and speed.

[0031] In summary, compared to traditional remote control methods for smart mobile terminals that require the operator to hold a game controller or other controller, limiting hand dexterity, the method in this embodiment allows the operator to directly control the smart mobile terminal through body posture, without holding any device, thus greatly improving the convenience of operation. Using body posture as control input is more in line with natural human behavior, making the operation more intuitive and easier to understand. Since the hands are freed, the operator can perform other operations while controlling the smart mobile terminal, such as grasping objects or using other tools.

[0032] In some embodiments, see Figure 2 Step S20 specifically includes the following parallel sub-steps:

[0033] In sub-step S201, if at least one of the pitch angle, yaw angle, and roll angle detected by any inertial measurement unit is not 0, and the absolute value of the acceleration is greater than the preset gravitational acceleration, then the moving direction and moving speed of the terminal component corresponding to the inertial measurement unit are determined according to the acceleration of the inertial measurement unit.

[0034] Specifically, if at least one of the pitch, yaw, and roll angles detected by the IMU is not zero, it indicates that the operator's body, where the IMU is located, has rotated relative to the reference coordinate system. The pitch angle describes the rotation of the object around its horizontal axis (x-axis), the yaw angle describes the rotation of the object around its vertical axis (z-axis), and the roll angle describes the rotation of the object around its vertical axis (y-axis).

[0035] At the same time, is the absolute value of the acceleration vector detected by the IMU greater than the preset gravitational acceleration (usually 9.81 m / s²)?2 The absolute value of the acceleration vector refers to its magnitude, which can be calculated by sqrt{a}. x 2 +a y 2 +a z 2} We obtain that the acceleration output by the IMU includes a x a y a z a x a y a z These are the components of the acceleration vector along the x-axis, y-axis, and z-axis, respectively.

[0036] If at least one of the pitch, yaw, and roll angles is not zero, and the absolute value of the acceleration is greater than a preset gravitational acceleration, the operator's body where the IMU is located has rotated, and the magnitude of its acceleration vector exceeds the gravitational acceleration. This indicates that the operator's body where the IMU is located is not only rotating, but also experiencing additional acceleration in a certain direction, i.e., it is accelerating. Therefore, further based on the IMU's a x a y a z To determine the direction and speed of movement of the terminal component.

[0037] Specifically, the direction of movement is determined by the direction of the acceleration component; for example, if a x If it is a positive value, then the direction of movement has a component in the positive x-axis direction; if a y If the value is positive, then the direction of movement has a component in the positive y-axis direction, and so on. The velocity can be calculated by integrating the acceleration component over time, or by using the following formula, where the velocity V in the x-axis direction is... x For example, V x =a x *t*w, where t is the preset time step and w is the preset weight coefficient. Similarly, the movement speed V in the y-axis direction can be obtained. y and the moving speed V in the z-axis direction z .

[0038] In sub-step S202, if at least one of the pitch angle, yaw angle, and roll angle detected by any inertial measurement unit is not 0, and the absolute value of the acceleration is equal to the preset gravitational acceleration, then it is determined that the moving direction and moving speed of the terminal component corresponding to the inertial measurement unit remain in the current state.

[0039] Specifically, if the absolute value of the acceleration vector detected by the IMU (i.e. the magnitude of the acceleration vector) is equal to the preset gravitational acceleration value, it indicates that the only acceleration experienced by the operator's body in the vertical direction is gravity, while there is no additional acceleration in the horizontal direction. In other words, the terminal component does not have a tendency to accelerate or decelerate in the horizontal direction.

[0040] Therefore, if at least one of the pitch, yaw, and roll angles detected by the IMU is not zero, and the absolute value of the acceleration is equal to the preset gravitational acceleration, then it can be determined that although the operator's body is rotating, its horizontal movement has not changed. Therefore, the direction and speed of movement of the terminal unit corresponding to the IMU remain unchanged, including the following cases: if the terminal unit was previously stationary, it remains stationary; if the terminal unit was previously moving at a certain speed, it continues to move at the same speed and direction.

[0041] In sub-step S203, if the pitch angle, yaw angle and roll angle detected by any inertial measurement unit are all 0, then it is determined that the terminal component corresponding to that inertial measurement unit is stationary.

[0042] Specifically, if the pitch, yaw, and roll angles detected by the IMU are all 0, the operator's body where the IMU is located is stationary. This not only means that the operator's body where the IMU is located is not rotating, but also that its speed in all directions is 0, that is, it is not moving.

[0043] In some embodiments, step S20 specifically includes:

[0044] The Euler angles and accelerations detected by the multiple inertial measurement units are subjected to sliding filtering, and the moving direction and moving speed of the multiple terminal components are determined based on the Euler angles and accelerations after sliding filtering (i.e., the above parallel sub-steps S201 to S203).

[0045] The model of the sliding filter is shown in the following mathematical expression (1):

[0046] y(k)=y(k-1)+[x(k)-x(kN)] / N(1)

[0047] Where y(k) is the filtered output at time step k; y(k-1) is the filtered output at the previous time step k-1; x(k) is the Euler angle or acceleration detected by the inertial measurement unit at the current time step k; x(kN) is the Euler angle or acceleration detected by the inertial measurement unit N time steps ago; N is the size of the sliding window, which is the amount of historical data considered by the filter. The window size N determines the smoothness of the data by the filter. The larger N is, the smoother the filtering effect, but it may reduce the response speed to rapid changes.

[0048] Specifically, the filtering model in this embodiment calculates the difference between the measurement value x(k) at the current time step k and the measurement value x(kN) N time steps ago. This difference is divided by the size N of the sliding window to obtain an average value, which represents the average rate of change of the data during the window period. This average rate of change is added to the filtered output y(k-1) of the previous time step k-1 to obtain the filtered output y(k) at the current time step k. In this way, the sliding filter can smoothly track the trend of the data instead of reacting to each individual measurement value. This helps to reduce short-term fluctuations caused by IMU measurement noise, thereby providing a more stable and reliable estimate for determining the direction and speed of movement of the terminal component.

[0049] See Figure 3 Corresponding to the intelligent mobile terminal control method of the above embodiments, another embodiment of this application provides an intelligent mobile terminal control device 1 capable of executing the method of the above embodiments, comprising:

[0050] The perception information acquisition module 11 is used to acquire Euler angles and accelerations detected by multiple inertial measurement units; wherein, the multiple inertial measurement units are respectively installed on the operator's body, and the multiple inertial measurement units correspond one-to-one with the multiple terminal components;

[0051] The movement state determination module 12 is used to determine the movement direction and movement speed of the multiple terminal components based on the Euler angles and accelerations detected by the multiple inertial measurement units.

[0052] The control command generation module 13 is used to generate control commands for controlling the movement of the multiple terminal components based on their movement direction and speed.

[0053] In some embodiments, the Euler angles include pitch angle, yaw angle, and roll angle;

[0054] The movement state determination module 12 is used for:

[0055] If at least one of the pitch angle, yaw angle, and roll angle detected by any inertial measurement unit is not 0, and the absolute value of the acceleration is greater than the preset gravitational acceleration, then the moving direction and moving speed of the terminal component corresponding to the inertial measurement unit are determined according to the acceleration of the inertial measurement unit.

[0056] If at least one of the pitch angle, yaw angle, and roll angle detected by any inertial measurement unit is not 0, and the absolute value of the acceleration is equal to the preset gravitational acceleration, then it is determined that the moving direction and moving speed of the terminal component corresponding to that inertial measurement unit remain in the current state.

[0057] If the pitch, yaw, and roll angles detected by any inertial measurement unit are all 0, then the terminal component corresponding to that inertial measurement unit is determined to be stationary.

[0058] In some embodiments, the movement state determination module 12 is configured to:

[0059] The Euler angles and accelerations detected by the multiple inertial measurement units 20 are subjected to sliding filtering, and the moving direction and moving speed of the multiple terminal components 2 are determined based on the Euler angles and accelerations after sliding filtering.

[0060] The model of the sliding filter is shown in the following mathematical expression (1):

[0061] y(k)=y(k-1)+[x(k)-x(kN)] / N(1)

[0062] Where y(k) is the filtered output at time step k; y(k-1) is the filtered output at the previous time step k-1; x(k) is the Euler angle or acceleration detected by the inertial measurement unit at the current time step k; x(kN) is the Euler angle or acceleration detected by the inertial measurement unit N time steps ago; and N is the size of the sliding window.

[0063] It should be noted that the intelligent mobile terminal control device in this embodiment corresponds to the intelligent mobile terminal control method described in the foregoing embodiments. Therefore, any details of the intelligent mobile terminal control device not described in this embodiment can be obtained by referring to the content of the intelligent mobile terminal control method in the foregoing embodiments, and will not be repeated in this embodiment.

[0064] See Figure 4 Another embodiment of this application provides a smart mobile terminal 10, including the smart mobile terminal control device 1 described in the above embodiments and a plurality of terminal components 2.

[0065] In this embodiment, the plurality of inertial measurement units 20 may include:

[0066] The first inertial measurement unit is positioned on the operator's chest;

[0067] The second inertial measurement unit is located on the operator's upper left arm;

[0068] The third inertial measurement unit is located on the operator's left forearm;

[0069] The fourth inertial measurement unit is located on the operator's upper right arm;

[0070] The fifth inertial measurement unit is located on the operator's right forearm.

[0071] In this embodiment, the plurality of terminal components 2 may include:

[0072] The terminal body corresponds to the first inertial measurement unit;

[0073] The upper left arm of the terminal corresponds to the second inertial measurement unit; the first end of the upper left arm of the terminal is connected to the left side of the terminal body through a first joint element;

[0074] The left forearm of the terminal corresponds to the third inertial measurement unit; the first end of the left forearm of the terminal is connected to the second end of the left upper arm of the terminal through a second joint element, and the second end of the left forearm of the terminal is provided with a robotic hand for grasping objects, and the robotic hand can be controlled by the operator's hands;

[0075] The upper right arm of the terminal corresponds to the fourth inertial measurement unit; the first end of the upper right arm of the terminal is connected to the right side of the terminal body through a first joint element;

[0076] The right forearm of the terminal corresponds to the fifth inertial measurement unit; the first end of the right forearm of the terminal is connected to the second end of the right upper arm of the terminal through a second joint element, and the second end of the right forearm of the terminal is provided with a robotic hand for grasping objects, which can be controlled by the operator's hands.

[0077] Another embodiment of this application provides a computer program product, including computer program instructions, which instruct a computer device to perform the operation corresponding to the intelligent mobile terminal control method described in the above embodiments.

[0078] Specifically, the computer program product includes a series of computer program instructions that can instruct a computer device to execute the intelligent mobile terminal control method described in this application. These instructions are code written in a computer program, defining how to perform specific operations. In this embodiment, these instructions are used to execute the intelligent mobile terminal control method of the above embodiments.

[0079] These program instructions are designed to be loaded onto a computer device and to instruct the device to perform specific operations, which refer to the various steps in the intelligent mobile terminal control method described in the above embodiments.

[0080] In this way, the computer program product provides a complete software solution that can run on various computer devices to implement the intelligent mobile terminal control method described in the above embodiments.

[0081] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for controlling an intelligent mobile terminal, wherein the intelligent mobile terminal comprises multiple terminal components, characterized in that, The method includes: The system acquires Euler angles and accelerations detected by multiple inertial measurement units (IMUs); wherein the multiple IMUs are respectively mounted on the operator's body, and each IMU corresponds to one of the multiple terminal components; the Euler angles include pitch angle, yaw angle, and roll angle. The moving direction and moving speed of the multiple terminal components are determined based on the Euler angles and accelerations detected by the multiple inertial measurement units. Control commands for controlling the movement of the multiple terminal components are generated based on their movement direction and speed. The determination of the moving direction and moving speed of the multiple terminal components based on the Euler angles and accelerations detected by the multiple inertial measurement units includes: If at least one of the pitch angle, yaw angle, and roll angle detected by any inertial measurement unit is not 0, and the absolute value of the acceleration is greater than the preset gravitational acceleration, then the moving direction and moving speed of the terminal component corresponding to the inertial measurement unit are determined according to the acceleration of the inertial measurement unit. If at least one of the pitch angle, yaw angle, and roll angle detected by any inertial measurement unit is not 0, and the absolute value of the acceleration is equal to the preset gravitational acceleration, then it is determined that the moving direction and moving speed of the terminal component corresponding to that inertial measurement unit remain in the current state. If the pitch, yaw, and roll angles detected by any inertial measurement unit are all 0, then the terminal component corresponding to that inertial measurement unit is determined to be stationary.

2. The method according to claim 1, characterized in that, Determining the moving direction and moving speed of the plurality of terminal components based on the Euler angles and acceleration includes: The Euler angles and accelerations detected by the multiple inertial measurement units are subjected to sliding filtering, and the moving direction and moving speed of the multiple terminal components are determined based on the Euler angles and accelerations after sliding filtering. The sliding filter model is shown in the following mathematical expression: y(k) = y(k-1) + [x(k) - x(kN)] / N Where y(k) is the filtered output at time step k; y(k-1) is the filtered output at the previous time step k-1; x(k) is the Euler angle or acceleration detected by the inertial measurement unit at the current time step k; x(kN) is the Euler angle or acceleration detected by the inertial measurement unit N time steps ago; and N is the size of the sliding window.

3. A smart mobile terminal control device, characterized in that, include: The perception information acquisition module is used to acquire Euler angles and accelerations detected by multiple inertial measurement units; wherein, the multiple inertial measurement units are respectively installed on the operator's body, and the multiple inertial measurement units correspond one-to-one with multiple terminal components; the Euler angles include pitch angle, yaw angle and roll angle; The movement state determination module is used to determine the movement direction and movement speed of the multiple terminal components based on the Euler angles and accelerations detected by the multiple inertial measurement units. A control command generation module is used to generate control commands for controlling the movement of the plurality of terminal components based on their movement direction and speed; wherein: If at least one of the pitch angle, yaw angle, and roll angle detected by any inertial measurement unit is not 0, and the absolute value of the acceleration is greater than the preset gravitational acceleration, then the moving direction and moving speed of the terminal component corresponding to the inertial measurement unit are determined according to the acceleration of the inertial measurement unit. If at least one of the pitch angle, yaw angle, and roll angle detected by any inertial measurement unit is not 0, and the absolute value of the acceleration is equal to the preset gravitational acceleration, then it is determined that the moving direction and moving speed of the terminal component corresponding to that inertial measurement unit remain in the current state. If the pitch, yaw, and roll angles detected by any inertial measurement unit are all 0, then the terminal component corresponding to that inertial measurement unit is determined to be stationary.

4. The intelligent mobile terminal control device according to claim 3, characterized in that, The control command generation module is used for: The Euler angles and accelerations detected by the multiple inertial measurement units are subjected to sliding filtering, and the moving direction and moving speed of the multiple terminal components are determined based on the Euler angles and accelerations after sliding filtering. The sliding filter model is shown in the following mathematical expression: y(k) = y(k-1) + [x(k) - x(kN)] / N Where y(k) is the filtered output at time step k; y(k-1) is the filtered output at the previous time step k-1; x(k) is the Euler angle or acceleration detected by the inertial measurement unit at the current time step k; x(kN) is the Euler angle or acceleration detected by the inertial measurement unit N time steps ago; and N is the size of the sliding window.

5. A smart mobile terminal, characterized in that, It includes the intelligent mobile terminal control device as described in any one of claims 3 to 4, and a plurality of terminal components.

6. The intelligent mobile terminal according to claim 5, characterized in that, The plurality of inertial measurement units include: The first inertial measurement unit is positioned on the operator's chest; The second inertial measurement unit is located on the operator's upper left arm; The third inertial measurement unit is located on the operator's left forearm; The fourth inertial measurement unit is located on the operator's upper right arm; The fifth inertial measurement unit is located on the operator's right forearm.

7. The intelligent mobile terminal according to claim 6, characterized in that, The plurality of terminal components include: The terminal body corresponds to the first inertial measurement unit; The upper left arm of the terminal corresponds to the second inertial measurement unit; the first end of the upper left arm of the terminal is connected to the left side of the terminal body through a first joint element; The left forearm of the terminal corresponds to the third inertial measurement unit; the first end of the left forearm of the terminal is connected to the second end of the left upper arm of the terminal through a second joint element, and the second end of the left forearm of the terminal is provided with a robotic hand for grasping objects; The upper right arm of the terminal corresponds to the fourth inertial measurement unit; the first end of the upper right arm of the terminal is connected to the right side of the terminal body through a first joint element; The right forearm of the terminal corresponds to the fifth inertial measurement unit; the first end of the right forearm of the terminal is connected to the second end of the right upper arm of the terminal through a second joint element, and the second end of the right forearm of the terminal is provided with a robotic hand for grasping objects.

8. A computer program product, characterized in that, It includes computer program instructions that instruct a computer device to perform an operation corresponding to the method as described in any one of claims 1 to 2.

Images