A method and device for motion recognition based on multiple sensors

Abstract

The present invention discloses a method and device for multi-sensor motion recognition. The method includes: acquiring data from an accelerometer when determining that a user has begun exercising; determining whether a wearable device is in a stationary state based on the accelerometer data; if so, acquiring data from a geomagnetic sensor; determining the user's hand posture based on the geomagnetic sensor data; and evaluating the user's motion process and posture based on the postures of the wearable device in two consecutive stationary states. By combining data from multiple sensors, various motions and postures can be recognized on the wearable device, assisting in determining whether the results of the exercise are valid.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of intelligent devices, and in particular to a method and apparatus for multi-sensor based motion recognition. Background Art

[0002] Wearable devices have become extremely popular in recent years. Wearable devices are portable devices that are worn directly on the body or integrated into clothing or accessories. Wearable devices are more than just hardware devices; they enable powerful functionality through software support, data interaction, and cloud-based interaction. Wearable devices are poised to significantly transform our lives and perceptions.

[0003] Currently, wearable devices can collect motion status, but there is an urgent need for a solution to assist in determining the user's motion status. Summary of the Invention

[0004] The embodiments of the present invention provide a method and apparatus for multi-sensor based motion recognition, which can recognize various motions and postures of a user during exercise and assist in determining whether the results of the exercise are valid.

[0005] In a first aspect, an embodiment of the present invention provides a method for multi-sensor based motion recognition, comprising:

[0006] When it is determined that the user starts to move, data from the acceleration sensor is obtained;

[0007] Determining whether the wearable device is in a stationary state based on data from the acceleration sensor;

[0008] If yes, collecting data from a geomagnetic sensor, and determining the hand posture of the user based on the data from the geomagnetic sensor;

[0009] The user's movement process and posture are evaluated based on the postures of the wearable device in two adjacent static states.

[0010] Optionally, determining whether the wearable device is in a stationary state based on data from the acceleration sensor includes:

[0011] Determine whether the data of the acceleration sensor is less than an acceleration threshold; if so, determine that the wearable device is in a stationary state; otherwise, determine that the wearable device is not in a stationary state.

[0012] Optionally, collecting data from a geomagnetic sensor and determining the user's hand posture based on the data from the geomagnetic sensor includes:

[0013] Determining a composite vector of each axis of the wearable device based on data from the geomagnetic sensor;

[0014] By converting the coordinate axes, the rotation angle of the wearable device relative to each axis of the geomagnetic field is obtained.

[0015] Optionally, if the wearable device is not in a stationary state, filtering the data of the acceleration sensor and collecting the data of the gyroscope;

[0016] The data of the acceleration sensor is double-integrated according to the data of the gyroscope to determine the spatial trajectory of the wearable device.

[0017] Optionally, the evaluating the user's movement process and posture based on the postures of the wearable device in two adjacent static states includes:

[0018] Determining the direction of movement of the wearable device based on the spatial trajectory of the wearable device during movement;

[0019] Based on the postures of the wearable device in two adjacent static states, data from the acceleration sensor, and data from the geomagnetic sensor, it is evaluated whether the posture of the user is in place during the movement, and movement information of the user's movement is displayed.

[0020] In a second aspect, an embodiment of the present invention provides a device for multi-sensor based motion recognition, comprising:

[0021] an acquisition unit, configured to acquire data from the acceleration sensor when it is determined that the user starts exercising;

[0022] The processing unit is used to determine whether the wearable device is in a stationary state based on the data of the acceleration sensor; if so, collect data from the geomagnetic sensor and determine the user's hand posture based on the data of the geomagnetic sensor; and evaluate the user's movement process and posture based on the postures of the wearable device in two adjacent stationary states.

[0023] Optionally, the processing unit is specifically configured to:

[0024] Determine whether the data of the acceleration sensor is less than an acceleration threshold; if so, determine that the wearable device is in a stationary state; otherwise, determine that the wearable device is not in a stationary state.

[0025] Optionally, the processing unit is specifically configured to:

[0026] Determining a composite vector of each axis of the wearable device based on data from the geomagnetic sensor;

[0027] By converting the coordinate axes, the rotation angle of the wearable device relative to each axis of the geomagnetic field is obtained.

[0028] Optionally, the processing unit is further configured to:

[0029] If the wearable device is not in a stationary state, filtering the data of the acceleration sensor and collecting the data of the gyroscope;

[0030] The data of the acceleration sensor is double-integrated according to the data of the gyroscope to determine the spatial trajectory of the wearable device.

[0031] Optionally, the processing unit is specifically configured to:

[0032] Determining the direction of movement of the wearable device based on the spatial trajectory of the wearable device during movement;

[0033] Based on the postures of the wearable device in two adjacent static states, data from the acceleration sensor, and data from the geomagnetic sensor, it is evaluated whether the posture of the user is in place during the movement, and movement information of the user's movement is displayed.

[0034] In a third aspect, an embodiment of the present invention further provides a computing device, including:

[0035] a memory for storing program instructions;

[0036] The processor is configured to call the program instructions stored in the memory and execute the above-mentioned method for multi-sensor-based motion recognition according to the obtained program.

[0037] In a fourth aspect, an embodiment of the present invention further provides a computer-readable non-volatile storage medium, comprising computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the above-mentioned method of multi-sensor-based motion recognition.

[0038] In an embodiment of the present invention, when it is determined that a user has begun exercising, data from the accelerometer is acquired. Based on this data, it is determined whether the wearable device is stationary. If so, data from the geomagnetic sensor is collected and the user's hand posture is determined based on this data. Based on the postures of two consecutive periods of stationary state, the user's movement process and posture are evaluated. By combining data from multiple sensors, various actions and postures can be recognized on the wearable device, assisting in determining whether the results of the exercise are valid. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1A schematic diagram of a system architecture provided by an embodiment of the present invention;

[0041] Figure 2 A flowchart of a method for multi-sensor motion recognition provided by an embodiment of the present invention;

[0042] Figure 3 A flowchart of a method for multi-sensor motion recognition provided by an embodiment of the present invention;

[0043] Figure 4 A schematic structural diagram of a multi-sensor based motion recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0045] Figure 1 A system architecture is provided for an embodiment of the present invention. Figure 1 As shown, the system architecture may be a server 100 , which may include a processor 110 , a communication interface 120 , and a memory 130 .

[0046] The communication interface 120 is used to communicate with the wearable device, send and receive information transmitted by the wearable device, and realize communication.

[0047] The processor 110 is the control center of the server 100. It connects various parts of the server 100 using various interfaces and lines. It executes various functions of the server 100 and processes data by running or executing software programs and / or modules stored in the memory 130 and calling data stored in the memory 130. Optionally, the processor 110 may include one or more processing units.

[0048] Memory 130 can be used to store software programs and modules. Processor 110 executes various functional applications and data processing by running the software programs and modules stored in memory 130. Memory 130 may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the data storage area may store data generated based on business processing. Memory 130 may also include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state memory device.

[0049] It should be noted that the above Figure 1 The structure shown is only an example and is not limited in the embodiment of the present invention.

[0050] Figure 2 A process of multi-sensor based motion recognition provided by an embodiment of the present invention is exemplarily shown. The process can be executed by a multi-sensor based motion recognition device, which can be a server or located in a server.

[0051] like Figure 2 As shown in the figure, the process specifically includes:

[0052] Step 201: When it is determined that the user starts exercising, data from the acceleration sensor is obtained.

[0053] In an embodiment of the present invention, when a user exercises, a corresponding exercise mode can be activated, such as pull-ups, running, sit-ups, swimming, etc., and the user selects the exercise mode. Once the user selects the exercise mode, the user can be confirmed to have started exercising. At this time, data from the acceleration sensor, namely the three-axis acceleration, can be collected to form an acceleration curve.

[0054] Step 202: Determine whether the wearable device is in a stationary state based on the data from the acceleration sensor.

[0055] Specifically, it is determined whether the data of the acceleration sensor is less than an acceleration threshold. If so, it is determined that the wearable device is in a stationary state. Otherwise, it is determined that the wearable device is not in a stationary state. The acceleration threshold can be set based on experience.

[0056] That is, when the acceleration sensor data is less than the acceleration threshold, the user is in a stationary state and not moving. When the acceleration sensor data is greater than or equal to the acceleration threshold, the user is not in a stationary state and is still in a stationary state.

[0057] Step 203: If yes, collect data from the geomagnetic sensor, and determine the hand posture of the user based on the data from the geomagnetic sensor.

[0058] Specifically, based on the data from the geomagnetic sensor, the composite vector of each axis of the wearable device is determined; and by converting the coordinate axes, the rotation angle of the wearable device relative to each axis of the geomagnetic field is obtained.

[0059] The three-axis geomagnetic data from the geomagnetic sensor is used to determine whether the values ​​of each axis coincide with those of the Earth's axis when the wearable device is stationary. Because the combined magnetic field vectors of the Earth's magnetic field in the vertical, east-west, and north-south directions are relatively fixed, by calculating the combined vectors of each axis of the wearable device, the rotation angle of the wearable device relative to each axis of the Earth's magnetic field can be determined through coordinate axis transformation.

[0060] In addition, if the wearable device is not in a stationary state, the data from the acceleration sensor is filtered and the data from the gyroscope is collected; the data from the acceleration sensor is double-integrated based on the data from the gyroscope to determine the spatial trajectory of the wearable device.

[0061] This method can be used to draw the spatial trajectory of wearable devices. Among them, there are very mature algorithms for solving spatial postures, such as rotation matrices, quaternions, and Euler angles.

[0062] For example, the calculation of the spatial trajectory of a wearable device is: s = ∫∫(a0+a0*t)dt 2 , where s represents the motion displacement, a0 represents the initial acceleration, t represents the duration, and dt represents the differential with respect to the duration.

[0063] Step 204 : evaluating the user's movement process and posture based on the postures of the wearable device in two adjacent static states.

[0064] Specifically, the direction of the wearable device's movement can be determined based on the wearable device's spatial trajectory during movement. Furthermore, the user's posture during movement can be assessed based on the wearable device's two consecutive static positions, data from the wearable device's accelerometer, and data from its geomagnetic sensor, and the user's movement information can be displayed.

[0065] In this step, the main purpose is to obtain information such as the direction (referenced to the ground), speed, and frequency of the user's movement, and to evaluate whether the user's movement process and posture are in place. Since the strength of the geomagnetic field in the east-west and perpendicular directions to the ground is almost zero, during the movement, the north-south direction can be determined to determine whether the movement has rotated in the north-south direction, and thus the direction of movement. That is, when exercising, look at the compass to determine whether the movement is north-south or east-west, or whether there is an angle.

[0066] The strength of the Earth's magnetic field is equal to the sum of the three-axis vectors of the wearable device: A = x + y + z.

[0067] For example, the local geomagnetic field strength is 0.5 and the direction is north-south.

[0068] The x-axis of the watch is 0.3, the y-axis is 0.4, and the z-axis is 0. The wearable device is placed horizontally, and the angle between the x-axis and the north and south is arctan(4 / 3).

[0069] When evaluating the user's posture during exercise, the focus is on assessing whether the user's movements are correct in each exercise mode. For example, when doing sit-ups with the head held high, if the user's movement speed is not zero, that is, when the user is stationary, the watch's z-axis geomagnetic data is not zero, indicating that the exercise did not start from a flat position and the posture is not correct. For example, when punching, if the speed does not meet the requirements, the posture is judged to be substandard.

[0070] In order to better explain the embodiment of the present invention, the process of the above-mentioned multi-sensor based action recognition will be described below in a specific implementation scenario. Figure 3 Shown, including:

[0071] Step 301: Acquire data from an acceleration sensor.

[0072] Step 302 , determining whether the user is in a stationary state, if so, proceeding to step 303 , otherwise proceeding to step 306 .

[0073] It mainly judges whether it is in a stationary state based on the data of the acceleration sensor. If the data of the acceleration sensor is less than the acceleration threshold, it indicates that it is in a stationary state, otherwise it is not in a stationary state.

[0074] Step 303: Analyze the user's hand posture based on the data from the geomagnetic sensor.

[0075] Data from the geomagnetic sensor is collected and used to determine whether the values ​​of the Earth's axis and the wearable device's three-axis geomagnetic data coincide when the wearable device is stationary. Because the combined magnetic field vectors for each of the Earth's vertical, east-west, and north-south directions are relatively fixed, by calculating the combined vectors for each axis of the wearable device, the wearable device's rotation angle relative to each axis of the Earth's magnetic field can be determined through coordinate axis transformation.

[0076] Step 304: Compare the two static postures.

[0077] Step 305: Evaluate the user's movement process and posture.

[0078] The direction of the wearable device's movement can be determined based on the wearable device's spatial trajectory during movement. Furthermore, the user's posture during movement can be assessed based on the wearable device's two consecutive static positions, data from the wearable device's accelerometer, and data from the geomagnetic sensor, and the user's movement information can be displayed.

[0079] Step 306: filtering and noise reduction.

[0080] The data of the acceleration sensor is filtered and denoised, mainly the waveform of the acceleration curve is filtered and denoised.

[0081] Step 307: Acquire gyroscope data.

[0082] Step 308: Double integration of acceleration.

[0083] The data from the acceleration sensor is double-integrated based on the data from the gyroscope.

[0084] Step 309: draw the spatial trajectory.

[0085] The above embodiment demonstrates that when determining that a user has begun exercising, data from the accelerometer is acquired; based on this data, it is determined whether the wearable device is stationary. If so, data from the geomagnetic sensor is collected, and the user's hand posture is determined based on this data. The user's movement process and posture are then evaluated based on the postures of the wearable device during two consecutive periods of stationary state. By combining data from multiple sensors, various actions and postures can be recognized on the wearable device, assisting in determining whether the results of the exercise are valid.

[0086] Based on the same technical concept, Figure 4 The structure of a multi-sensor based motion recognition device provided by an embodiment of the present invention is exemplarily shown. The device can execute the multi-sensor based motion recognition process.

[0087] like Figure 4 As shown, the device may include:

[0088] An acquisition unit 401 is configured to acquire data from an acceleration sensor when it is determined that the user has started exercising;

[0089] The processing unit 402 is used to determine whether the wearable device is in a stationary state based on the data of the acceleration sensor; if so, collect data from the geomagnetic sensor and determine the user's hand posture based on the data of the geomagnetic sensor; and evaluate the user's movement process and posture based on the postures of the wearable device in two adjacent stationary states.

[0090] Optionally, the processing unit 402 is specifically configured to:

[0091] Determine whether the data of the acceleration sensor is less than an acceleration threshold; if so, determine that the wearable device is in a stationary state; otherwise, determine that the wearable device is not in a stationary state.

[0092] Optionally, the processing unit 402 is specifically configured to:

[0093] Determining a composite vector of each axis of the wearable device based on data from the geomagnetic sensor;

[0094] By converting the coordinate axes, the rotation angle of the wearable device relative to each axis of the geomagnetic field is obtained.

[0095] Optionally, the processing unit 402 is further configured to:

[0096] If the wearable device is not in a stationary state, filtering the data of the acceleration sensor and collecting the data of the gyroscope;

[0097] The data of the acceleration sensor is double-integrated according to the data of the gyroscope to determine the spatial trajectory of the wearable device.

[0098] Optionally, the processing unit 402 is specifically configured to:

[0099] Determining the direction of movement of the wearable device based on the spatial trajectory of the wearable device during movement;

[0100] Based on the postures of the wearable device in two adjacent static states, data from the acceleration sensor, and data from the geomagnetic sensor, it is evaluated whether the posture of the user is in place during the movement, and movement information of the user's movement is displayed.

[0101] Based on the same technical concept, an embodiment of the present invention further provides a computing device, including:

[0102] a memory for storing program instructions;

[0103] The processor is configured to call the program instructions stored in the memory and execute the above-mentioned method for multi-sensor-based motion recognition according to the obtained program.

[0104] Based on the same technical concept, an embodiment of the present invention also provides a computer-readable non-volatile storage medium, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the above-mentioned multi-sensor-based motion recognition method.

[0105] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0108] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0109] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for motion recognition based on multiple sensors, characterized in that: include: When it is determined that the user starts to move, data from the acceleration sensor is obtained; Determining whether the wearable device is in a stationary state based on data from the acceleration sensor; If yes, collecting data from a geomagnetic sensor, and determining the hand posture of the user based on the data from the geomagnetic sensor; Evaluate the user's movement process and posture based on the postures of the wearable device in two adjacent static states; The step of evaluating the user's movement process and posture based on the postures of the wearable device in two adjacent static states includes: Determining the direction of movement of the wearable device based on the spatial trajectory of the wearable device during movement; Based on the postures of the wearable device in two adjacent static states, data from the acceleration sensor, and data from the geomagnetic sensor, it is evaluated whether the posture of the user is in place during the movement, and movement information of the user's movement is displayed.

2. The method according to claim 1, wherein The determining whether the wearable device is in a stationary state based on data from the acceleration sensor includes: Determine whether the data of the acceleration sensor is less than an acceleration threshold; if so, determine that the wearable device is in a stationary state; otherwise, determine that the wearable device is not in a stationary state.

3. The method according to claim 1, wherein The collecting data from the geomagnetic sensor and determining the hand posture of the user based on the data from the geomagnetic sensor includes: Determining a composite vector of each axis of the wearable device based on data from the geomagnetic sensor; By converting the coordinate axes, the rotation angle of the wearable device relative to each axis of the geomagnetic field is obtained.

4. The method according to claim 1, wherein If the wearable device is not in a stationary state, filtering the data of the acceleration sensor and collecting the data of the gyroscope; The data of the acceleration sensor is double-integrated according to the data of the gyroscope to determine the spatial trajectory of the wearable device.

5. A multi-sensor based motion recognition device, characterized in that: include: an acquisition unit, configured to acquire data from the acceleration sensor when it is determined that the user starts exercising; a processing unit, configured to determine whether the wearable device is in a stationary state based on data from the acceleration sensor; If yes, collect data from a geomagnetic sensor, and determine the user's hand posture based on the data from the geomagnetic sensor; and evaluate the user's movement process and posture based on the postures of the wearable device in two adjacent static states; The processing unit is specifically configured to: Determining the direction of movement of the wearable device based on the spatial trajectory of the wearable device during movement; Based on the postures of the wearable device in two adjacent static states, data from the acceleration sensor, and data from the geomagnetic sensor, it is evaluated whether the posture of the user is in place during the movement, and movement information of the user's movement is displayed.

6. The device according to claim 5, characterized in that The processing unit is specifically configured to: Determine whether the data of the acceleration sensor is less than an acceleration threshold; if so, determine that the wearable device is in a stationary state; otherwise, determine that the wearable device is not in a stationary state.

7. The device according to claim 5, characterized in that The processing unit is specifically configured to: Determining a composite vector of each axis of the wearable device based on data from the geomagnetic sensor; By converting the coordinate axes, the rotation angle of the wearable device relative to each axis of the geomagnetic field is obtained.

8. A computing device, characterized in that include: a memory for storing program instructions; A processor is configured to call the program instructions stored in the memory and execute the method according to any one of claims 1 to 4 according to the obtained program.

9. A computer-readable non-volatile storage medium, characterized in that: The method comprises computer-readable instructions, which, when read and executed by a computer, enable the computer to perform the method according to any one of claims 1 to 4.

Images