Ski pole information acquisition system based on rgb camera and mems multi-sensor fusion and control method thereof

Abstract

The application discloses a ski pole information acquisition system based on an RGB camera and MEMS multi-sensor fusion and a control method thereof, and belongs to the technical field of sports data acquisition.The problem that a traditional ski pole information acquisition system in the prior art is difficult to simultaneously monitor ski pole position, posture and ground contact state information is solved;the posture information and the ground contact state of the ski pole are acquired through MEMS inertial sensors and MEMS pressure sensors; a data processing terminal configured with an HRNet neural network and a residual neural network is adopted to extract picture data shot by an RGB camera, human posture information and human position information are calculated, the positions of the hands of a skier are combined to be the positions of the handles of the ski pole, and the position information of the ski pole is indirectly obtained; meanwhile, original sensor information reflecting the force application state of the hands of the skier is obtained through the MEMS pressure sensors.The application realizes multi-element information monitoring of the ski pole, and can be applied to monitoring the state of the ski pole of the skier.

Description

Technical Field

[0001] This invention relates to a ski pole information acquisition system and its control method, and more particularly to a ski pole information acquisition system and its control method based on RGB camera and MEMS multi-sensor fusion, belonging to the field of sports data acquisition technology. Background Technology

[0002] Skiing is limited by factors such as venue, environment, and climate, resulting in a very limited time for ice and snow sports each year. With the development of ice and snow sports, ski machines are gradually moving from professional to mass use. However, current indoor ski machines are relatively simple, with low levels of intelligence and tactile feedback. High-end, sophisticated ski training equipment can cost hundreds of thousands or even millions of yuan per set, and is only used for specialized training of national-level professional athletes to help them improve their movements and deepen muscle memory during the off-season. This cannot achieve the widespread popularization of indoor skiing. Therefore, there is an urgent need to conduct research on low-cost intelligent ski equipment.

[0003] Most devices currently on the market that capture real-world data are designed for single-information measurement, and real-time observation of dynamic information related to skiing is generally done through human motion capture. However, during skiing, the posture, position, and ground contact of ski poles also need to be monitored. The ground contact of ski poles can reflect the changes in hand strength when skiers use ski poles to support themselves and move forward each time. The posture and position information of ski poles can reflect the accuracy of skiers' movements in real time.

[0004] In the prior art, patent document CN 112697333 A discloses a wireless force measurement platform for three-dimensional force measurement of ski poles. It includes a linear guide rail fixed to a guide rail mounting platform, an L-shaped slider reciprocating along the linear guide rail, a wireless amplifier fixed to one side of the L-shaped slider, and a three-dimensional force sensor, a force plate, and a snow simulation mat stacked sequentially from bottom to top on the top of the L-shaped slider. The signal output terminal of the three-dimensional force sensor is connected to the signal input terminal of the wireless amplifier, and the signal output terminal of the wireless amplifier is wirelessly connected to an external PC. During the pole vaulting process, the tip of the ski pole can be freely raised and lowered on the snow simulation mat. When using it, the tester does not need to modify the structure of the ski poles. By analyzing the collected three-dimensional force data of the pole vaulting, data such as the tester's pole vaulting cycle characteristics, number of pole vaults, pole vaulting frequency, and the magnitude of the three-dimensional force can be obtained. However, it is not easy to promote, as it is difficult to install a force plate on indoor ski machines, resulting in high installation costs. Furthermore, it is difficult to simultaneously observe the position and posture information of the ski poles, and it cannot provide a comprehensive evaluation of the skier's movements.

[0005] In summary, to address the problem of the difficulty in easily acquiring various ski pole information, there is a need for a ski pole information acquisition system and its control method that is easy to install and can simultaneously monitor ski pole attitude, position, and ground contact status. Summary of the Invention

[0006] A brief overview of the invention is given below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0007] In view of this, in order to solve the problem that traditional ski pole information acquisition systems in the prior art are unable to simultaneously monitor ski pole position, attitude and ground contact status information, the present invention provides a ski pole information acquisition system and its control method based on RGB camera and MEMS multi-sensor fusion.

[0008] Technical solution one is as follows: a ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion, including an inertial sensor, a ground contact detection module, an RGB camera, a wireless transmission module and a data processing terminal;

[0009] The inertial sensor is mounted on the ski pole and is connected to the data processing terminal via a wireless transmission module. The inertial sensor is used to acquire ski pole attitude information and transmit the ski pole attitude information to the data processing terminal via the wireless transmission module.

[0010] The ground contact detection module includes a pressure sensor and an ADC module. The pressure sensor is connected to the ADC module and is installed on the bottom of the ski pole. The pressure sensor is used to acquire the ski pole ground contact status information. The ground contact detection module is connected to the data processing terminal through a wireless transmission module to transmit the ski pole ground contact status information to the data processing terminal.

[0011] The RGB cameras are installed around the ski resort. The RGB cameras are used to acquire the location information of skiers. The RGB cameras are connected to the data processing terminal through a wireless transmission module and transmit the skier's location information to the data processing terminal.

[0012] Furthermore, the inertial sensor is a MEMS inertial sensor, and the pressure sensor is a MEMS pressure sensor.

[0013] Technical Solution 2: The control method for the ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion as described in Technical Solution 1 includes the following steps:

[0014] S1. Read the raw sensor data from the gyroscope, accelerometer and magnetometer in the inertial sensor, substitute the raw sensor data into the attitude calculation algorithm to obtain the raw ski pole attitude information, and upload the calculated current raw ski pole attitude data to the data processing terminal through the wireless transmission module.

[0015] S2. Read the raw sensor data of the pressure sensor, compare the raw sensor data of the pressure sensor with the set threshold to determine the current original ski pole contact state, and upload the original ski pole contact state information and the raw sensor data of the pressure sensor to the data processing terminal through the wireless transmission module.

[0016] S3. Take a picture containing the complete skier using an RGB camera, upload the picture to the data processing terminal via a wireless transmission module, input the image data into the HRNet neural network, obtain human posture information and human position information based on the residual neural network, and indirectly obtain the original ski pole position information based on the human posture information and human position information.

[0017] S4. Integrate the original ski pole posture data, original ski pole ground contact state information, and original ski pole position information into original ski pole data. Establish the correspondence between the experimental scenario's manually labeled real data and the original ski pole data. Train the data correction neural network. Substitute the original ski pole data into the trained data correction neural network to filter out error information and data noise, and obtain the final ski pole information.

[0018] Furthermore, step S1 includes the following steps:

[0019] S11. Perform error modeling for the accelerometer and magnetometer, and collect some raw sensor data from the inertial sensor for fitting and correction;

[0020] S12. The rotational angular rate, linear acceleration, and geomagnetic field strength data of the ski poles during their movement are measured by the gyroscope, accelerometer, and magnetometer mounted on the inertial sensor, respectively.

[0021] S13. Normalize the data from the accelerometer and magnetometer. Based on the complementary characteristics of the gyroscope, accelerometer and magnetometer in the frequency domain, use complementary filtering to fuse the rotational angular rate data, linear acceleration data and geomagnetic field strength data measured by the inertial sensor. Correct the pitch angle and roll angle through the accelerometer and correct the azimuth angle through the magnetometer.

[0022] S14. Euler angles, i.e., the original ski pole attitude information, are obtained by calculating the corrected pitch, roll and azimuth angles. After being numbered, they are transmitted to the data processing terminal via the wireless transmission module.

[0023] Further, in step S2, the original sensor data of the pressure sensor is read by the ADC module and compared with a set threshold. If the original sensor data of the pressure sensor exceeds the set threshold, the current original ski pole contact state is determined to be a contact state. If the original sensor data of the pressure sensor does not reach the set threshold, the current original ski pole contact state is determined to be a non-contact state. The information of the current original ski pole contact state and the information of the force applied by the athlete's hand during ski support movement, i.e., the original sensor data of the pressure sensor, are uploaded to the data processing terminal through the wireless transmission module.

[0024] Furthermore, step S3 includes the following steps:

[0025] S31. Take an image containing the complete skier using an RGB camera and transmit the image to a data processing terminal configured with an HRNet neural network;

[0026] S32. Use the HRNet neural network to obtain two-dimensional key point information of the human body in the image, save it in chronological order, and form a two-dimensional human body key point time series.

[0027] S33. Input the time series of two-dimensional key points of the human body into the residual neural network to predict the three-dimensional key point information of the current skier;

[0028] S34. Combine the skier's three-dimensional key point information to infer the skier's human posture and position information, and obtain the ski pole position information based on the position of the skier's hands as the position of the ski pole handles.

[0029] The beneficial effects of this invention are as follows: This invention acquires the attitude information and ground contact state of ski poles through MEMS inertial sensors and MEMS pressure sensors; it captures images of the skier as a whole using an RGB camera, and the data processing terminal uses HRNet neural networks and residual neural networks to extract and calculate human posture information and human position information. Combined with the position of the skier's hands, which is the position of the ski pole handles, the ski pole position information is indirectly obtained; the ski pole position information obtained by installing RGB cameras around the ski resort meets the required accuracy and is acquired in real time. The ski pole attitude information and ground contact state information can be obtained simply by installing MEMS inertial sensors and MEMS pressure sensors on the ski poles and combining them with a computer, which is easy to promote and reduces the cost of ski training equipment; the data processing terminal integrates the obtained ski pole information and the raw sensor information reflecting the skier's hand force state obtained by the MEMS pressure sensor, which can be subsequently applied to the construction of virtual machine environments for indoor ski machines, skiing action evaluation, and skiing sports analysis, enriching the skier's skiing experience. Attached Figure Description

[0030] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0031] Figure 1 This is a schematic diagram of a ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion.

[0032] Figure 2 A schematic diagram showing the connection between ski poles and the ski pole information acquisition system;

[0033] Figure 3 A schematic diagram illustrating the principle of acquiring original ski pole ground contact state information;

[0034] Figure 4 A diagram illustrating the relationship between the skier's position and the ski poles' position;

[0035] Figure 5 A schematic diagram illustrating the principle of ski pole position acquisition;

[0036] Figure 6 This is a schematic diagram of the control method for a ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion.

[0037] Reference numerals: 1. Ski pole; 2. Inertial sensor; 3. Pressure sensor. Detailed Implementation

[0038] To make the technical solutions and advantages of the embodiments of the present invention clearer, the exemplary embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0039] Example 1: Reference Figure 1-5 This embodiment describes a ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion, including an inertial sensor 2, a ground contact detection module, an RGB camera, a wireless transmission module, and a data processing terminal.

[0040] The inertial sensor 2 is mounted on the ski pole 1. The inertial sensor 2 is connected to the data processing terminal through a wireless transmission module. The inertial sensor 2 is used to acquire the attitude information of the ski pole 1 and transmit the attitude information of the ski pole 1 to the data processing terminal through the wireless transmission module.

[0041] The ground contact detection module includes a pressure sensor 3 and an ADC module. The pressure sensor 3 is connected to the ADC module and is installed at the bottom of the ski pole 1. The pressure sensor 3 is used to obtain the ground contact status information of the ski pole 1. The ground contact detection module is connected to the data processing terminal through a wireless transmission module and transmits the ground contact status information of the ski pole 1 to the data processing terminal.

[0042] The RGB camera is installed around the ski resort. The RGB camera is used to acquire the location information of the skiers. The RGB camera is connected to the data processing terminal through a wireless transmission module and transmits the location information of the skiers to the data processing terminal.

[0043] Specifically, the ADC module can convert the analog signal output by the pressure sensor 3 into a digital signal and transmit it to the wireless transmission module.

[0044] Furthermore, inertial sensor 2 is a MEMS inertial sensor, and pressure sensor 3 is a MEMS pressure sensor.

[0045] Example 2: Reference Figure 1-6 This embodiment describes a control method for a ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion, specifically including the following steps:

[0046] S1. Read the raw sensor data from the gyroscope, accelerometer and magnetometer in the inertial sensor, substitute the raw sensor data into the attitude calculation algorithm to obtain the raw ski pole attitude information, and upload the calculated current raw ski pole attitude data to the data processing terminal through the wireless transmission module.

[0047] S2. Read the raw sensor data of the pressure sensor, compare the raw sensor data of the pressure sensor with the set threshold to determine the current original ski pole contact state, and upload the original ski pole contact state information and the raw sensor data of the pressure sensor to the data processing terminal through the wireless transmission module.

[0048] S3. Take a picture containing the complete skier using an RGB camera, upload the picture to the data processing terminal via a wireless transmission module, input the image data into the HRNet neural network, obtain human posture information and human position information based on the residual neural network, and indirectly obtain the original ski pole position information based on the human posture information and human position information.

[0049] S4. Integrate the original ski pole posture data, original ski pole ground contact state information, and original ski pole position information into original ski pole data. Establish the correspondence between the experimental scene's manually labeled real data and the original ski pole data. Train the data correction neural network. Substitute the original ski pole data into the trained data correction neural network to filter out error information and data noise, and obtain the final ski pole information.

[0050] Specifically, the acquired original ski pole posture, position, and ground contact information are susceptible to errors and deviations due to various factors. The data processing terminal provides computing power to the system to process the image data. The data processing terminal uses a data correction neural network to filter the information, thereby obtaining more reliable data. In this embodiment, the data processing terminal uses a Transformer data correction neural network to filter data noise, and uses an HRNet neural network and a residual neural network to extract information from photos taken by an RGB camera to obtain human posture and position information. At the same time, the data processing terminal can also analyze and evaluate skiing actions by analyzing ski pole information data, and provide professional guidance to skiers by combining with a skiing experience database.

[0051] Furthermore, step S1 includes the following steps:

[0052] S11. Perform error modeling for the accelerometer and magnetometer, and collect some raw sensor data from the inertial sensor for fitting and correction;

[0053] In step S11, the rotational angular rate data, linear acceleration data, and geomagnetic field strength data of the ski poles during movement are measured by the gyroscope, accelerometer, and magnetometer mounted on the inertial sensor, respectively. The data from the accelerometer and magnetometer are normalized. Based on the complementary characteristics of the gyroscope, accelerometer, and magnetometer in the frequency domain, the rotational angular rate data, linear acceleration data, and geomagnetic field strength data measured by the inertial sensor are fused by complementary filtering. The pitch angle and roll angle are corrected by the accelerometer, and the azimuth angle is corrected by the magnetometer.

[0054] S12. Euler angles, i.e., the original ski pole attitude information, are obtained by calculating the corrected pitch angle, roll angle and azimuth angle. After being numbered, they are transmitted to the data processing terminal via the wireless transmission module.

[0055] Specifically, by using a complementary filter to fuse the rotational angular rate data, linear acceleration data, and geomagnetic field strength data from inertial sensor 2, the measurement accuracy of inertial sensor 2 and the dynamic response performance of the system can be improved. During the fusion process, the quaternion is first calculated based on the initial state of inertial sensor 2, and then the gravity vector and magnetic field lines are deduced. The data from the accelerometer and magnetometer are obtained and normalized. The normalized data are multiplied and then summed to calculate the error. A proportional-integral controller is used to correct the gyroscope. The corrected gyroscope values, namely the corrected pitch angle, roll angle, and azimuth angle, are used to update the quaternion. Euler angles are obtained by solving the equation.

[0056] Further, in S2, the original sensor data of the pressure sensor is read by the ADC module and compared with a set threshold. If the original sensor data of the pressure sensor exceeds the set threshold, the current original ski pole contact state is determined to be a contact state. If the original sensor data of the pressure sensor does not reach the set threshold, the current original ski pole contact state is determined to be a non-contact state. The information of the current original ski pole contact state and the information of the force applied by the athlete's hand during the ski support movement, i.e., the original sensor data of the pressure sensor, are uploaded to the data processing terminal through the wireless transmission module.

[0057] Specifically, the pressure value change of the pressure sensor 3 at the bottom of the ski pole 1 is strongly positively correlated with the force exerted by the skier's wrist during the forward movement of skiing. Therefore, transmitting the raw sensor data of the pressure sensor to the data processing terminal through the wireless transmission module can serve as an important indicator in subsequent skiing evaluation.

[0058] Furthermore, step S3 includes the following steps:

[0059] S31. Take an image containing the complete skier using an RGB camera and transmit the image to a data processing terminal configured with an HRNet neural network;

[0060] S32. Use the HRNet neural network to obtain two-dimensional key point information of the human body in the image, save it in chronological order, and form a two-dimensional human body key point time series.

[0061] S33. Input the time series of two-dimensional key points of the human body into the residual neural network to predict the three-dimensional key point information of the current skier;

[0062] S34. Combine the skier's three-dimensional key point information to infer the skier's human posture and position information, and obtain the ski pole position information based on the position of the skier's hands as the position of the ski pole handles.

[0063] Specifically, directly obtaining the position information of ski pole 1 is relatively difficult, and the obtained position information of ski pole 1 has low accuracy. By recognizing the skier's posture information, the position information of ski pole 1 can be obtained indirectly. This achieves the required accuracy at a low cost while obtaining a reasonable skier posture and ski pole 1 position, which more intuitively reflects whether the skier's movements are standard. In addition, HRNet neural network and residual neural network are existing models, which are stable and efficient.

[0064] Although the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of interpreting or limiting the subject matter of the invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the invention is illustrative and not restrictive, and the scope of the invention is defined by the appended claims.

Claims

1. A control method for a ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion, characterized in that, Includes the following steps: S1. Read the raw sensor data from the gyroscope, accelerometer and magnetometer in the inertial sensor, substitute the raw sensor data into the attitude calculation algorithm to obtain the raw ski pole attitude information, and upload the calculated current raw ski pole attitude data to the data processing terminal through the wireless transmission module. S2. Read the raw sensor data of the pressure sensor, compare the raw sensor data of the pressure sensor with the set threshold to determine the current original ski pole contact state, and upload the original ski pole contact state information and the raw sensor data of the pressure sensor to the data processing terminal through the wireless transmission module. S3. Take a picture containing the complete skier using an RGB camera, upload the picture to the data processing terminal via a wireless transmission module, input the image data into the HRNet neural network, obtain human posture information and human position information based on the residual neural network, and indirectly obtain the original ski pole position information based on the human posture information and human position information. S4. Integrate the original ski pole posture data, original ski pole ground contact state information, and original ski pole position information into original ski pole data. Establish the correspondence between the experimental scene's manually labeled real data and the original ski pole data. Train the data correction neural network. Substitute the original ski pole data into the trained data correction neural network to filter out error information and data noise, and obtain the final ski pole information. The ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion includes an inertial sensor (2), a ground contact detection module, an RGB camera, a wireless transmission module and a data processing terminal; The inertial sensor (2) is installed on the ski pole (1). The inertial sensor (2) is connected to the data processing terminal through the wireless transmission module. The inertial sensor (2) is used to obtain the attitude information of the ski pole (1) and transmit the attitude information of the ski pole (1) to the data processing terminal through the wireless transmission module. The ground contact detection module includes a pressure sensor (3) and an ADC module. The pressure sensor (3) is connected to the ADC module. The pressure sensor (3) is installed at the bottom of the ski pole (1). The pressure sensor (3) is used to obtain the ground contact status information of the ski pole (1). The ground contact detection module is connected to the data processing terminal through the wireless transmission module to transmit the ground contact status information of the ski pole (1) to the data processing terminal. The RGB cameras are installed around the ski resort. The RGB cameras are used to acquire the location information of skiers. The RGB cameras are connected to the data processing terminal through a wireless transmission module and transmit the skier's location information to the data processing terminal.

2. The control method for the ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion according to claim 1, characterized in that, The inertial sensor (2) is a MEMS inertial sensor, and the pressure sensor (3) is a MEMS pressure sensor.

3. The control method for the ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion according to claim 2, characterized in that, S1 includes the following steps: S11. Perform error modeling for the accelerometer and magnetometer, and collect some raw sensor data from the inertial sensor for fitting and correction; In step S11, the rotational angular rate data, linear acceleration data, and geomagnetic field strength data of the ski poles during movement are measured by the gyroscope, accelerometer, and magnetometer mounted on the inertial sensor, respectively. The data from the accelerometer and magnetometer are normalized. Based on the complementary characteristics of the gyroscope, accelerometer, and magnetometer in the frequency domain, the rotational angular rate data, linear acceleration data, and geomagnetic field strength data measured by the inertial sensor are fused by complementary filtering. The pitch angle and roll angle are corrected by the accelerometer, and the azimuth angle is corrected by the magnetometer. S12. Euler angles, i.e., the original ski pole attitude information, are obtained by calculating the corrected pitch angle, roll angle and azimuth angle. After being numbered, they are transmitted to the data processing terminal via the wireless transmission module.

4. The control method for the ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion according to claim 3, characterized in that, In step S2, the ADC module reads the raw sensor data of the pressure sensor and compares it with a set threshold. If the raw sensor data exceeds the set threshold, the current original ski pole contact state is determined to be a contact state. If the raw sensor data does not reach the set threshold, the current original ski pole contact state is determined to be a non-contact state. The raw sensor data of the pressure sensor, which reflects the force applied by the athlete's hand during ski support movement, is uploaded to the data processing terminal through the wireless transmission module.

5. The control method for the ski pole information acquisition system based on RGB camera and MEMS multi-sensor fusion according to claim 4, characterized in that, S3 includes the following steps: S31. Take an image containing the complete skier using an RGB camera and transmit the image to a data processing terminal configured with an HRNet neural network; S32. Use the HRNet neural network to obtain two-dimensional key point information of the human body in the image, save it in chronological order, and form a two-dimensional human body key point time series. S33. Input the time series of two-dimensional key points of the human body into the residual neural network to predict the three-dimensional key point information of the current skier; S34. Combine the skier's three-dimensional key point information to infer the skier's human posture and position information, and obtain the ski pole position information based on the position of the skier's hands as the position of the ski pole handles.

Images