Swing action recognition method, electronic device storage medium, and program product

Abstract

This application provides a swing motion recognition method, an electronic device storage medium, and a program product. The method includes: acquiring motion data of a first object during its movement; continuously sampling the motion data through multiple first windows to obtain multiple candidate window data corresponding to the motion data; and performing motion recognition on the multiple candidate window data to obtain the motion recognition result corresponding to the motion data. In other words, this method of reading motion data in real time during the object's movement allows for timely determination of the object's motion content, improving the accuracy and efficiency of motion recognition and enhancing the real-time nature of the motion recognition results.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for recognizing racket swings, an electronic device storage medium, and a program product. Background Technology

[0002] In sports scenarios, it is often necessary to identify the actions performed by users during the exercise in order to determine the content of their actions. For example, during a competition, it is necessary to identify and judge the athletes' actions in order to determine the competition results.

[0003] In related technologies, cameras are used to record athletes' movements, and the content of the athletes' movements is identified and analyzed based on the footage.

[0004] However, the method of motion recognition based on the shooting results cannot identify the athlete's movements in real time, resulting in low efficiency and poor real-time performance of motion recognition. Summary of the Invention

[0005] This application provides a swing motion recognition method, an electronic device storage medium, and a program product. This solution improves motion recognition efficiency and provides good real-time performance by reading the motion data of an object in real time and performing motion recognition.

[0006] Firstly, a swing motion recognition method is provided, which is applied to an electronic device. The method includes: acquiring motion data of the first object during its movement, the motion data being used to describe the motion state of the first object; continuously sampling the motion data through multiple first windows to obtain multiple candidate window data corresponding to the motion data, the multiple candidate window data corresponding one-to-one with the multiple first windows, wherein the i-th candidate window data is obtained by sampling the motion data using the i-th first window as the sampling range, the window length of the first window is a first preset value, the sampling interval between two adjacent first windows is a second preset value, the second preset value is less than the first preset value, and i is a positive integer; performing motion recognition on the multiple candidate window data to obtain the motion recognition result corresponding to the motion data, the motion recognition result being used to indicate the action content category of the first object.

[0007] In this application's technical solution, motion data of the first object is acquired during its movement. Multiple candidate window data are obtained by continuously sampling the motion data through multiple first windows, with at least two candidate windows having overlapping data content. Action recognition is then performed on the multiple candidate window data to obtain the action corresponding to the motion data. In other words, this method of real-time motion data reading and action recognition during the object's movement can promptly determine the object's action content, improving the accuracy and efficiency of action recognition and enhancing the real-time nature of the action recognition results.

[0008] It should be understood that motion data refers to data describing the motion state of the first object during the motion process.

[0009] Optionally, the motion data includes at least one of acceleration data or angular velocity data.

[0010] Among them, acceleration data is used to describe the rate of change of the first object's motion, and angular velocity data is used to describe the rate of rotation of the first object around the axis.

[0011] To illustrate, the acceleration data is implemented as three-axis acceleration data, namely X-axis acceleration data, Y-axis acceleration data and Z-axis acceleration data; the angular velocity data is implemented as three-axis angular velocity data, namely X-axis angular velocity data, Y-axis angular velocity data and Z-axis angular velocity data.

[0012] It should be understood that motion data can be realized as motion data over a period of time. In other words, motion data is a series of data generated at various time points during the motion process. Therefore, the first window is used to indicate the single sampling range of the motion data. The time range of the single sampling is determined by the first window, and the motion data is sampled to obtain the candidate window data corresponding to the first window.

[0013] This is illustrative; the number of candidate window data corresponds to the number of data in the first window.

[0014] Indicatively, the second preset value being less than the first preset value means that there is overlap between two adjacent first windows. Therefore, the candidate window data collected by the two adjacent first windows also contain the same data.

[0015] It should be understood that the action recognition result represents the action content of the first object during the movement process. For example, when the first object is playing badminton, by performing action recognition on the corresponding movement data of playing badminton, the action content category of the first object is obtained, including: forehand swing, backhand swing and high clear.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the window transformation process includes the process of the prompt window expanding in a first direction and gradually transforming into the application window. Using the above method, during the display of the application transition animation, the prompt window is used as a reference point to display the expansion in a specified direction until the application window corresponding to the first application is displayed, thus improving the display continuity of the window expansion.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the first preset value is greater than the first threshold and less than the second threshold; the second preset value is greater than the third threshold and less than the fourth threshold, the third threshold is less than the first threshold, and the fourth threshold is less than the second threshold. By using the above method, setting the window length and sampling interval within a specified range ensures that the window length and sampling interval are neither too large nor too small during the sampling process, thereby improving the accuracy and completeness of data sampling.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, when the motion data meets the first condition, the motion data undergoes a symmetrical transformation to obtain transformed data. Transformed data refers to the data obtained after mirroring the motion data according to a preset baseline. The transformed data is then continuously sampled to obtain multiple candidate window data. Using the above method, taking a left-hand gesture as an example, since motion data generated by a right-hand gesture is more common than that generated by a left-hand gesture, the accuracy of action recognition results obtained from right-hand data is higher than that from left-hand data. Therefore, if the motion data is generated by a left-hand gesture, it is symmetrically transformed to obtain right-hand gesture motion data, which facilitates subsequent action recognition and improves the accuracy of action recognition.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the motion data includes acceleration data, and the acceleration data includes X-axis acceleration data; using a preset baseline as a reference, the X-axis acceleration data is mirrored and flipped in a preset direction to obtain X-axis converted acceleration data.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the motion data includes angular velocity data, which in turn includes Z-axis angular velocity data and Y-axis angular velocity data. Using a preset baseline as a reference, the Y-axis angular velocity data is mirrored and flipped in a preset direction to obtain Y-axis converted angular velocity data; similarly, using a preset baseline as a reference, the Z-axis angular velocity data is mirrored and flipped in a preset direction to obtain Z-axis converted angular velocity data. By flipping the motion data along a specified axis in the acceleration and angular velocity data using the above method, the rationality and accuracy of the converted data can be improved.

[0021] In conjunction with the first aspect, in certain implementations of the first aspect, the waveform peak data corresponding to the candidate window data is obtained; the number of repeated samplings corresponding to the candidate window data is determined based on the waveform peak data, window length, and sampling interval; multiple candidate window data are deduplicated based on the number of repeated samplings to obtain deduplicated window data; and action recognition is performed on multiple deduplicated window data to obtain the action recognition result corresponding to the motion data. By analyzing the waveform peak data in conjunction with the window length and sampling interval, determining the number of repeated samplings of the same candidate motion data, and deduplicating the repeatedly sampled candidate windows, the number of candidate window data required for final action recognition can be reduced, duplicate recognition can be avoided, computational overhead can be saved, and thus the efficiency of action recognition can be improved.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, an action prediction model is used to perform action detection processing on multiple candidate window data, obtaining action detection results corresponding to each candidate window data. These action detection results indicate whether the candidate window data belongs to a swing action. If the candidate window data belongs to a swing action, an action classification model is used to perform action classification processing on the candidate window data, obtaining the corresponding action recognition result. Through this method, it is first determined whether the candidate window data belongs to a swing action, and then the candidate window data belonging to a swing action are further classified to finally obtain the action recognition result. Using a staged recognition approach allows for the timely removal of candidate window data that does not belong to a swing action, reducing the recognition process and improving the efficiency and accuracy of action recognition.

[0023] In conjunction with the first aspect, in certain implementations of the first aspect, sample motion data is acquired; during the sliding sample window process, the sample motion data is continuously sampled to obtain multiple sample window data corresponding to the sample motion data; the sample window data is labeled with categories to obtain category labels corresponding to the multiple sample window data, the category labels including at least one of swing category labels and action category labels; a first dataset is generated based on the swing category labels, and a second dataset is generated based on the action category labels; a sample prediction model is trained based on the first dataset to obtain an action prediction model; a sample classification model is trained based on the second dataset to obtain an action classification model. Through the above method, by using sample windows to label category data, after sampling the sample motion data according to the sample windows, different datasets are generated based on the sampling results and category labels for training neural network models under different tasks, thus improving model training efficiency.

[0024] In conjunction with the first aspect, some implementations of the first aspect include: the swing motion during the swing process; non-swing motions during the swing process; and motions outside the swing process. By incorporating motions not part of the swing process into the model training data using the above method, the accuracy of the model training results can be further improved.

[0025] In a second aspect, a swing motion recognition device is provided, the device comprising a unit consisting of software and / or hardware for performing any of the methods of the first aspect.

[0026] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it is able to implement any of the methods of the first aspect.

[0027] Fourthly, a chip is provided, including a processor for reading and executing a computer program stored in a memory, wherein the computer program, when executed by the processor, is capable of implementing any of the methods of the first aspect.

[0028] Optionally, the chip also includes a memory electrically connected to the processor.

[0029] Optionally, the chip may also include a communication interface.

[0030] Fifthly, a computer-readable storage medium is provided that stores a computer program that, when executed by a processor, can implement any of the methods of the first aspect.

[0031] In a sixth aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, can implement any of the methods of the first aspect. Attached Figure Description

[0032] Figure 1 This is an application scenario diagram of a racket swing action recognition method according to an embodiment of this application.

[0033] Figure 2 This is a flowchart illustrating the application process of a racket swing action recognition method according to an embodiment of this application.

[0034] Figure 3 This is a schematic diagram of a symmetry transformation process according to an embodiment of this application.

[0035] Figure 4 This is a schematic diagram of window sampling according to an embodiment of this application.

[0036] Figure 5 This is a schematic diagram of a deduplication process according to an embodiment of this application.

[0037] Figure 6 This is a schematic diagram of a swing action recognition method in an application jump scenario according to an embodiment of this application.

[0038] Figure 7 This is a flowchart of a tour process for an action recognition model according to an embodiment of this application.

[0039] Figure 8 This is a flowchart of a racket swing action recognition method according to an embodiment of this application.

[0040] Figure 9 This is a schematic diagram of a software architecture according to an embodiment of this application.

[0041] Figure 10 This application describes a real-time recognition device for badminton racket swing actions.

[0042] Figure 11 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0043] The embodiments of this application will now be described in conjunction with the accompanying drawings.

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0046] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0047] The computer devices (including terminal devices) involved in the above embodiments of this application can be implemented as at least one of the following terminal device types: mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. This application does not impose any restrictions on the specific type of terminal device.

[0048] For example, a terminal device can be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, an in-vehicle device, a vehicle-to-everything (V2X) terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a set-top box (STB), customer premises equipment (CPE), and / or other devices used for communication over a wireless system, as well as next-generation communication systems, such as mobile terminals in 5G networks or mobile terminals in future evolved Public Land Mobile Network (PLMN) networks.

[0049] As an example and not a limitation, when the computer device is a wearable device, the term "wearable device" can also refer to any device that utilizes wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices worn directly on the body or integrated into a user's clothing or accessories. Wearable devices are not merely hardware devices; they achieve powerful functions through software support, data interaction, and cloud interaction. Broadly defined, wearable smart devices include those with comprehensive functions, large sizes, and the ability to perform complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those focused on a specific application function that require interaction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0050] With the rapid development of terminal technology, users have increasingly higher demands for action recognition in sports scenarios. Typically, high-speed cameras are used to record the movement of a first object in real time. The movement of the first object is then analyzed based on the recorded video content to determine the action content of the first object. For example, in a badminton match, the process of athletes playing badminton is recorded in real time by setting up a video camera off the court. The recorded video is then played back to determine the action content performed by the athletes during the badminton match.

[0051] However, the above method cannot identify the actions performed by the first object in motion in a timely manner. The motion content of the first object can only be determined by analyzing the recorded video afterward, which results in low efficiency of action recognition and reduces the real-time performance of action recognition.

[0052] The technical solution provided in this application, when the first object is wearing an electronic device, acquires motion data of the first object during its movement. This motion data describes the motion state of the first object. Multiple first windows are used to continuously sample the motion data, resulting in multiple candidate window data corresponding to the motion data. Each candidate window data corresponds one-to-one with a first window, and the candidate window data is collected by sampling the motion data within the first window's sampling range. The window length of the first window is fixed to a first preset value, and the sampling interval between two adjacent first windows is fixed to a second preset value. Setting the second preset value smaller than the first preset value ensures overlap between the window data collected by adjacent first windows, preventing omissions during data sampling. Finally, action recognition is performed on the multiple candidate window data to obtain the action recognition result corresponding to the motion data. In other words, this method of real-time motion data reading and action recognition during the object's movement can promptly determine the object's action content, improving the accuracy and efficiency of action recognition and enhancing the real-time performance of the action recognition results.

[0053] This is illustrative; please refer to it. Figure 1 The diagram illustrates an application scenario of the racket swing recognition method provided in an exemplary embodiment of this application, such as... Figure 1 As shown, currently, the first object 101 is wearing an electronic device 102. The electronic device 102 is activated, displaying a sports selection interface 103. This interface 103 is used to select the sports activity the first object 101 wants to perform. Two options are displayed: badminton and aerobics. When the electronic device 102 receives a trigger operation for badminton, it displays a left / right hand selection interface 104. This interface 104 is used to select whether the first object 101 is wearing the electronic device 102 on its left or right hand. Figure 1As shown, the first object 101 wears the electronic device 102 on its left hand. Therefore, it can select to wear it on its left hand, displaying the ready interface 105, which shows a start control. When the start control is triggered, the first object 101 can start playing badminton. During the badminton game, the electronic device 102 acquires the motion data of the first object 101 in real time and continuously samples the motion data through multiple windows to obtain multiple candidate window data. Action recognition is performed on the multiple candidate window data to obtain the action content category of the first object. Among them, the action content interface 106, action content interface 107, and action content interface 108 are three different action content display results. The action content interface 106 mainly counts the swing and smash, the action content interface 107 classifies and counts the forehand and backhand swings and the overhand and underhand swings, and the action content interface 108 classifies and counts the tactical actions.

[0054] Optionally, the action content interfaces 106, 107, and 108 can be displayed in turn, or the first object 101 can slide to display the action content interfaces 106, 107, and 108 by performing a sliding operation on the electronic device 101; or the action content interfaces 106, 107, and 108 can be switched in turn according to a preset time interval; or the first object 101 can pre-select a specific interface from the action content interfaces 106, 107, and 108, and when the movement is completed, the electronic device 102 selects the interface pre-selected by the first object 101 for display.

[0055] In one feasible scenario, taking the action content interface 106 as an example, during the process of the first object 101 playing badminton, the action content interface 106 is displayed in the electronic device 102. The total number of swings, the number of consecutive swings, and the number of smashes are updated in real time according to the swing situation of the first object 101 playing badminton. That is to say, at the fifth second, the total number of swings is 5, the number of consecutive swings is 5, and the number of smashes is 1. At the 35th second, the total number of swings is 8, the number of consecutive swings is 7, and the number of smashes is 4.

[0056] In another feasible scenario, taking the action content interface 106 as an example, when the first object 101 finishes playing badminton, the action content interface 106 immediately displays the total number of swings, consecutive swings, and smashes of the first object 101 during this badminton game. In other words, in this case, the total number of swings, consecutive swings, and smashes are not updated in real time, but are displayed immediately after the first object 101 finishes its game.

[0057] The following section provides a detailed explanation of the application and training processes of this scheme.

[0058] Application process:

[0059] This is illustrative; please refer to it. Figure 2 It illustrates a flowchart of the application process of the racket swing action recognition method provided in an exemplary embodiment of this application, such as... Figure 2 As shown, the method includes the following steps.

[0060] S210, during the movement of the first object, acquire the motion data of the first object.

[0061] Among them, motion data is used to describe the motion state of the first object.

[0062] Optionally, motion data refers to motion parameters used to describe the motion state of the first object.

[0063] Optionally, the motion data includes at least one of acceleration data or angular velocity data.

[0064] Optionally, the motion data can be acquired in at least one of the following ways:

[0065] The first method involves acquiring the acceleration and angular velocity data of the first object in real time during its motion, using this data as motion data.

[0066] The second method involves pre-setting a time interval and acquiring the acceleration and angular velocity data of the first object according to the time interval as motion data. For example, the time interval is set to 0.3 seconds, and the acceleration and angular velocity data of the first object are acquired every 0.3 seconds.

[0067] It is worth noting that the above-described methods for acquiring motion data are merely illustrative examples, and the embodiments of this application do not limit them.

[0068] In this embodiment, the motion data is implemented as acceleration data and angular velocity data of the first object at different moments during its motion, wherein the acceleration data is implemented as triaxial acceleration data A. pos =(a x ,a y ,a z ), where a x This represents the X-axis acceleration data, a y This represents the Y-axis acceleration data, a z This represents the Z-axis acceleration data; the angular velocity data is implemented as triaxial angular velocity data G. pos =(g x ,g y ,g z ), where g xRepresents the X-axis angular velocity data, g y This represents the angular velocity data along the Y-axis, g. z This represents the Z-axis angular velocity data.

[0069] In this embodiment, the maximum complete acquisition period length of the above acceleration data and angular velocity data is T. Then, for any time t, its acceleration data is represented as (a xt ,a yt ,a zt ), angular velocity data can be expressed as (g xt ,g yt ,g zt ).

[0070] In this embodiment, after obtaining the above acceleration data and angular velocity data, the acceleration data is integrated to obtain the resultant acceleration data, and the angular velocity data is integrated to obtain the resultant angular velocity data. For details, please refer to Formula 1 and Formula 2 below.

[0071]

[0072]

[0073] Among them, a rt This represents the resultant acceleration data, g. rt This represents the resultant angular velocity data.

[0074] After obtaining the resultant acceleration data and resultant angular velocity data, the resultant angular velocity data can be integrated with the aforementioned acceleration data to obtain the fused acceleration data A. t =(a xt ,a yt ,a zt ,a rt The combined angular velocity data is then integrated with the aforementioned angular velocity data to obtain the fused angular velocity data G. t =(g xt ,g yt ,g zt ,g rt ).

[0075] In this embodiment, the fused acceleration data and fused angular velocity data are used as the motion data of the first object.

[0076] S220: If the motion data meets the first condition, perform symmetric transformation on the motion data to obtain transformed data.

[0077] Among them, the transformed data refers to the data obtained after mirroring and flipping the motion data according to a preset baseline.

[0078] As an illustration, since the first object has different hand habits during movement, that is, some objects are left-handed and some are right-handed, and in most cases, the use of the right hand to perform actions is more common than the use of the left hand, when the first object uses the left hand to perform actions, it is necessary to perform a symmetrical transformation on the motion data to adjust it to the motion data generated by the right hand to improve the accuracy of subsequent action recognition.

[0079] In this embodiment, after obtaining the motion data in the above steps, if the motion data belongs to the motion data generated by the left hand of the first object, then the motion data is subjected to symmetry transformation processing to obtain the transformed data corresponding to the motion data.

[0080] In this embodiment, the motion data is waveform data generated according to a time series. That is, when the motion data is implemented as triaxial acceleration data and triaxial angular velocity data, a total of eight waveforms are generated, namely waveform A corresponding to the X-axis acceleration data, waveform B corresponding to the Y-axis acceleration data, waveform C corresponding to the Z-axis acceleration data, waveform D corresponding to the resultant acceleration data, waveform E corresponding to the X-axis angular velocity data, waveform F corresponding to the Y-axis angular velocity data, waveform G corresponding to the Z-axis angular velocity data, and waveform H corresponding to the resultant angular velocity data. A central axis (or baseline) is preset, and one or more waveforms are flipped around the central axis as the flip axis to obtain the converted waveform, which is used as the converted data. The specific execution process is as follows.

[0081] The first method is to perform symmetrical transformation of acceleration data: flip the waveform data corresponding to the X-axis acceleration data in the three-axis acceleration data along the central axis (generally, the horizontal line corresponding to the 0 value in the data axis is selected as the baseline), while the remaining Y-axis acceleration data, Z-axis data and resultant acceleration data remain unchanged.

[0082] The second method involves symmetrical transformation of angular velocity data: the waveform data corresponding to the Y-axis angular velocity data and Z-axis angular velocity data are flipped along the central axis (generally the 0-value central axis is selected), while the remaining X-axis angular velocity data and resultant angular velocity data remain unchanged.

[0083] This is illustrative; please refer to it. Figure 3 It illustrates a schematic diagram of a symmetry transformation process provided in an exemplary embodiment of this application, such as... Figure 3As shown, the current display shows the symmetry transformation process 310 corresponding to acceleration data and the symmetry transformation process 320 corresponding to angular velocity data. Symmetry transformation process 310 includes waveforms A311 corresponding to X-axis acceleration, B312 corresponding to Y-axis acceleration, C313 corresponding to Z-axis acceleration, and D314 corresponding to resultant acceleration. In the symmetry transformation process for acceleration data, only waveform A is flipped along baseline 301 to obtain flipped waveform 315. Symmetry transformation process 320 includes waveforms E321 corresponding to X-axis angular velocity, F322 corresponding to Y-axis angular velocity, G323 corresponding to Z-axis angular velocity, and H324 corresponding to resultant angular velocity. In the symmetry transformation process for angular velocity data, waveforms F322 and G323 are flipped along baseline 302 to obtain flipped waveforms 325 and 326.

[0084] In this embodiment, after obtaining the converted data, it is also necessary to preprocess the converted data to obtain preprocessed data.

[0085] Optionally, preprocessing includes at least one of data filtering, data denoising, time alignment, or time correction.

[0086] Among them, data filtering refers to removing interference signals generated in motion data through filtering techniques in digital signal processing; data denoising refers to removing noise signals in motion data, for example, by using the exponential moving average method to weight and sum the values ​​corresponding to the data at a certain moment and the data at several previous moments to eliminate noise signals in motion data, while amplifying the trend in the data; time alignment or time correction refers to adjusting motion data at different time points to the same time reference.

[0087] In this embodiment, mean filtering is used to filter the motion data, achieving the goal of reducing noise and smoothing the data while considering the overall power consumption of the device. Taking acceleration data A as an example, the window length of the mean filter is set to k. For the acceleration data A in the time series (acceleration data A can be realized as a set of acceleration data over a period of time), the mean filtering method for the acceleration at each time t can be referred to the following formula 3.

[0088]

[0089] Among them, A t t This represents the acceleration at time t.

[0090] In this embodiment, standardized operations are used to eliminate the dimensional differences corresponding to different sensor data installed in different electronic devices. That is to say, for the same electronic device, the types of multiple sensors installed in it may be different. For example, an electronic device may have a gyroscope and a magnetometer sensor installed. The magnetometer sensor is used to acquire acceleration data, and the gyroscope is used to acquire angular velocity data. Therefore, there will be obvious differences between acceleration data and angular velocity data in the same set of data. Therefore, the dimensional differences are removed by means of the following method, which can be referred to Formula 4.

[0091]

[0092] Where μ represents the mean vector of acceleration and angular velocity data for each axis (X-axis, Y-axis, and Z-axis), and σ represents the standard deviation vector of acceleration and angular velocity data for each axis.

[0093] S230: The preprocessed data is continuously sampled through the first window to obtain multiple candidate window data.

[0094] Among them, multiple candidate window data correspond one-to-one with multiple first windows. The i-th candidate window data is obtained by collecting motion data with the i-th first window as the sampling range. The window length of the first window is a first preset value, and the sampling interval between two adjacent first windows is a second preset value. The second preset value is less than the first preset value, and i is a positive integer.

[0095] Indicatively, the preprocessed data is a set of data corresponding to different moments in a time series; therefore, the preprocessed data can also be called motion data. As for the first window, it is used to determine the time range corresponding to a single sample during the sampling and preprocessing of the data.

[0096] In a schematic manner, during the continuous sampling of preprocessed data through the first window, the preprocessed data is sampled for the first time with the window length corresponding to the first window as the time range to obtain the first candidate window data. After moving the first window according to the preset sampling interval, a new first window is generated. The preprocessed data is sampled for the second time using the new first window to obtain the second candidate window data, and so on, until multiple candidate window data are finally obtained.

[0097] This is illustrative; please refer to it. Figure 4 This illustrates a window sampling diagram provided in an exemplary embodiment of this application, such as... Figure 4As shown, according to the above content, the preprocessed data includes X-axis acceleration data, Y-axis acceleration data, Z-axis acceleration data, resultant acceleration data, X-axis angular velocity data, Y-axis angular velocity data, Z-axis angular velocity data, and resultant angular velocity data. Therefore, these eight data correspond to different waveforms in the same time series, which represent the values ​​at different times. Thus, the process of continuously sampling the preprocessed data through the first window is to align the eight waveforms according to the time series and then synchronously sample the eight waveforms according to the first window. That is, each first window contains the waveform values ​​corresponding to the eight waveforms respectively.

[0098] from Figure 4 It can be seen that windows 401 and 402 are two adjacent windows, and windows 403 and 404 are two adjacent windows. Taking windows 401 and 402 as examples, window 402 is a new window generated after moving the sampling interval of window 401. That is to say, windows 401 and 402 are two windows of the same length but corresponding to different time ranges. Taking the motion process as playing badminton, according to windows 401 and 402, it can be seen that both windows 401 and 402 sampled the peak data corresponding to the eight waveforms respectively. Therefore, the result corresponding to windows 401 and 402 is: the action at this moment is a swing action. According to windows 403 and 404, it can be seen that neither window 403 nor window 404 sampled the peak data in the waveform. Therefore, the result corresponding to windows 403 and 404 is: the action at this moment is a non-swing action.

[0099] In this embodiment, assuming the length of the first window is L and the length of each sliding step (i.e., the length of each sampling interval) is S, the continuous sensor data generated by the athlete during badminton can be continuously sampled through the sliding window. By inputting the sampled motion data of a fixed length into the subsequent recognizer model, the continuous detection and classification of the athlete's swing action can be achieved.

[0100] The most critical aspect of this implementation is determining the window length L and the window sliding step length S. If the window length L is too large, it may contain motion data corresponding to two or even more swing actions at the same time, thus failing to decouple the swing detection and classification results. If the window length L is too small, the range of motion data contained in a single window may be too small to reflect the complete information of a single swing action.

[0101] Similarly, an excessively large window sliding step length S not only leads to a low sampling rate, making it easy to miss some swing action information, but also increases the overall time delay of the device in detecting and classifying swing actions; while an excessively small window sliding step length S increases the sampling frequency of single swing action information, increasing model complexity and theoretical power consumption.

[0102] This is illustrative; please refer to it. Figure 5 It illustrates a schematic diagram of a window determination process provided in an exemplary embodiment of this application, such as... Figure 5 As shown, after statistically analyzing the time intervals of standard right-handed swings from multiple pairs of athletes, it can be found that the frequency changes significantly from the two-second mark to the four-second mark, and the number of occurrences is greater than in other time periods. Therefore, it can be inferred that the approximate range of the time interval between two adjacent swings is between 2 and 4 seconds. Thus, the theoretical length of the window should be less than 2 seconds.

[0103] In this embodiment, the sensor sampling frequency is set to 100 Hz. To minimize the output delay of the subsequent swing action detection and classification recognition model, the window length can be set between 100 (corresponding to the first threshold) and 120 (corresponding to the second threshold). For example, the window length is set to 100 (i.e., 1 second), and the window sliding step length (i.e., sampling interval) can be set between 40 (corresponding to the third threshold) and 60 (corresponding to the fourth threshold), for example, 50 (i.e., 0.5 seconds).

[0104] S240: The action detection model is used to perform action detection processing on multiple candidate window data to obtain the action detection results corresponding to the multiple candidate window data respectively.

[0105] In a schematic manner, an action detection model is used to detect actions on multiple candidate window data, and action detection results corresponding to the multiple candidate window data are obtained. The action detection results include two types: those belonging to a swing action and those not belonging to a swing action. The multiple candidate window data belonging to a swing action are taken as the action detection results.

[0106] S250: Obtain the waveform peak data corresponding to the motion detection result.

[0107] In illustrative terms, waveform peak data refers to the data within a single candidate window that corresponds to the waveform peak. In this embodiment, it can also be referred to as the waveform peak position index.

[0108] Optionally, the peak value corresponding to the same peak in eight waveforms can be used as a single data point. In other words, a single candidate window data includes one or more peaks, and each peak corresponds to different peak data in eight waveform diagrams.

[0109] S260 determines the number of repeated samplings corresponding to the motion detection result based on waveform peak data, window length, and sampling interval.

[0110] As an illustration, since the sampling interval between two adjacent first windows is less than the window length corresponding to a single first window, there may be repeated detection for the same waveform peak data. For example, waveform peak data 1 is collected in window 1 and waveform peak data 1 is also collected in window 2.

[0111] After employing the above strategy to perform continuous window segmentation on the sensor waveform data, the sensor waveform data of a single swing action will be repeatedly sampled, and then repeatedly detected and classified by the swing action detection model and the swing action classification model. In this embodiment, the relationship between the sliding window length L, the sliding step length S, and the number of times the waveform corresponding to any swing action is repeatedly sampled is first modeled. The left boundary index position of any first window i (that is, the value corresponding to the left window boundary of the first window i) can be referred to Formula 5.

[0112] Formula 5:

[0113] left i =1+(i-1)·S for i=1,2,3,…,k

[0114] Where k is the total number of window data segments that can be obtained by dividing the motion data of total length T into multiple windows of length L and stride length S. Therefore, the index range (i.e., the window range) of each window data segment can be determined by the following formula 6.

[0115] Formula Six:

[0116] [left i ,left i +1,…,left i +L-1]

[0117] In other words, as can be seen from Formula 6, the total number of window data segments is obtained by dividing the time series corresponding to the motion data based on the left boundary index position, according to the window length and stride length.

[0118] In this embodiment, for the waveform peak position index p corresponding to a certain swing action, if the waveform peak is in the sliding window i, the following condition must be met, which can be referred to Formula 7 for details.

[0119] Formula 7:

[0120] left i ≤p≤left i +L-1

[0121] The left corresponding to formula five above i Substituting the definition into the above formula, we can obtain the following formula eight.

[0122] Formula 8:

[0123] 1+(i-1)·S≤p≤1+(i-1)·S+L-1

[0124] After further simplification, we can obtain the following formula nine.

[0125]

[0126] Therefore, the number of times the peak position index p of any waveform is repeatedly sampled can be referred to the following formula ten.

[0127]

[0128] The above formula can be used to calculate the number of times the waveform peak corresponding to any swing action is repeatedly sampled and repeatedly predicted under different combinations of window length and window sliding step length.

[0129] S270 performs deduplication processing on multiple action detection results based on the number of repeated samplings to obtain deduplication window data.

[0130] Taking the swing motion detection task in this application as an example, with a window length of 100 and a step length of 50, it can be obtained through Formula 10 that under this combination of conditions, the waveform peak corresponding to any swing motion will be predicted twice.

[0131] This is illustrative; please refer to it. Figure 6 It illustrates a deduplication process provided in an exemplary embodiment of this application, such as... Figure 6 As shown in this example, during the detection of motion data through multiple first windows along the detection direction, taking data a corresponding to 11 as the true label (i.e., the motion data corresponding to 11 belongs to the swing action) as an example, as long as the result of any first window detecting data a is determined to be a swing action (i.e., category 1, belonging to the swing action), it means that a valid swing action has been detected at this time. That is, when the detection result of data a is 11, 10 or 01, it is considered to have a valid swing action, and since a total of three detections are performed, two of the motion data are removed.

[0132] The same principle applies to other tasks such as forehand and backhand classification; and by setting a minimum time interval for continuous swings, it is possible to prevent the first window within a similar time distance from being repeatedly judged as a valid swing action.

[0133] S280 performs action recognition on multiple deduplication window data to obtain the action recognition results corresponding to the motion data.

[0134] The action recognition result is used to indicate the category of the action content of the first object.

[0135] In this embodiment, the model classifies the deduplication window data into actions using a pre-trained action classification model, and obtains the probability of the deduplication window data belonging to the forehand / backhand, uphand / downhand, and detailed action categories. In this process, deduplication can also be performed. The processing flow can be referred to the same as S270 above. Finally, the action recognition result of the first object in the motion process is obtained.

[0136] Training phase:

[0137] This is illustrative; please refer to it. Figure 7 The diagram illustrates a flowchart of a motion recognition model provided in an exemplary embodiment of this application, which includes the following steps.

[0138] S710, acquire sample motion data.

[0139] In the offline training phase, the first step is to acquire the sample motion data X and corresponding label data Y required for model training. Using wristband electronic devices equipped with accelerometers and gyroscopes, standard right-handed racket swing data is collected from N right-handed athletes. This includes three-axis acceleration data A for effective swing actions such as forehand, backhand, overhead, underhand, clear, smash, and block. pos =(a x ,a y ,a z ) and triaxial angular velocity data G pos =(g x ,g y ,g z The three-axis acceleration data A includes the data for actions such as picking up the shuttlecock, catching the shuttlecock, and slapping the thigh (used to represent swinging motions not part of the badminton game). neg =(a x ,a y ,a z ) and triaxial angular velocity data G neg =(g x ,g y ,g z ).

[0140] Assuming the maximum complete acquisition period length of the above acceleration and angular velocity data is T, then for any time t, the acceleration and angular velocity data can be expressed as (a xt ,a yt ,a zt ) and (gxt ,g yt ,g zt Furthermore, the resultant acceleration value a at any time t can be calculated using formulas one and two above. rt Resultant angular velocity value g rt Then the acceleration data at any time t is A. t =(a xt ,a yt ,a zt ,a rt ), angular velocity data is G t =(g xt ,g yt ,g zt ,g rt The acceleration and angular velocity data obtained above are used as sample motion data, and the superscripts of the forward and reverse scenes are ignored here.

[0141] The S720 continuously samples sample motion data through multiple sample windows to obtain multiple sample window data corresponding to the sample motion data.

[0142] In illustration, motion data is a collection of data corresponding to different moments in a time series; therefore, motion data can also be called motion data. As for the sample window, it is used to determine the time range corresponding to a single sample during the sampling of motion data.

[0143] In a schematic manner, during the continuous sampling of motion data through a sample window, the motion data is sampled for the first time with the window length corresponding to the sample window as the time range, and the first candidate window data is obtained. After the sample window is moved according to the preset sampling interval, a new sample window is generated. The preprocessed data is sampled for the second time using the new sample window to obtain the second candidate window data, and so on, until multiple candidate window data are obtained.

[0144] In this embodiment, assuming the sample window length is L and the length of each sliding step is S, the continuous sensor data generated by the athlete during badminton can be continuously sampled through this sliding window. By inputting the sampled sensor data of a fixed length into the subsequent recognizer model, the continuous detection and classification of the athlete's swing action can be achieved.

[0145] In this embodiment, the most critical aspect is determining the window length L and the window sliding step length S. If the window length L is too large, it may contain motion data corresponding to two or even more swing actions at the same time, thus failing to decouple the swing detection and classification results. If the window length L is too small, the range of motion data contained in a single window may be too small to reflect the complete information of a single swing action.

[0146] Similarly, an excessively large window sliding step length S not only leads to a low sampling rate, making it easy to miss some swing action information, but also increases the overall time delay of the device in detecting and classifying swing actions; while an excessively small window sliding step length S increases the sampling frequency of single swing action information, increasing model complexity and theoretical power consumption.

[0147] S730 performs category labeling on the sample window data, obtaining the category labels corresponding to multiple sample window data respectively.

[0148] The category labels include at least one of the swing category labels and the action category labels.

[0149] During the offline training phase, it is also necessary to specify the label category for each segmented window. As mentioned above... Figure 3 As shown, the vertical dashed line marks the position of the waveform peak corresponding to the motion data of a certain swing. Assuming the index position of this data in the time series is p, then if the left boundary index position of the segmented window i is left... i and right boundary index position i If the conditions in Formula 11 below are met, the window data segment is labeled as a swing category (i.e., category 0); otherwise, it is labeled as a non-swing category (i.e., category 1). For the window data labeled as a swing category, its corresponding forehand / backhand, overhand / underhand, and detailed swing action categories are further labeled. Specifically, the reverse scene multivariate motion data X... neg All window data labels processed by the sliding window segmentation module should be marked as non-swing category (i.e., category 1).

[0150] Formula 11:

[0151] left i ≤p <right i

[0152] After processing by the aforementioned sliding window segmentation module, a fixed-length swing action window data and the corresponding swing action label can be obtained.

[0153] S740 generates a first dataset based on swing category labels, and generates a second dataset based on action category labels.

[0154] To detect valid swing motions and classify them into forehand / backhand, overhand / underhand, and detailed motion categories, separate swing motion detection datasets need to be constructed. Racket Swing Classification Dataset Among them, the racket swing detection dataset The data comes from two sources: positive scene multivariate motion data Xpos and reverse scene multivariate motion data X neg The latter is added to improve the robustness of the subsequent swing action detection model under reverse scenario conditions.

[0155] Therefore, the racket swing detection dataset can be represented as The racket swing classification dataset can be represented as Where N represents the number of sliding window samples in the dataset, T represents the length of the sliding window data, and D represents the number of time series variables. Let represent the label of sample i on the k-th task. Since there are a total of 3 classification tasks here, namely forward / backward hand classification, up / down hand classification, and detailed action category classification, the maximum value of k is 3.

[0156] S750 trains the sample prediction model based on the first dataset to obtain the action prediction model.

[0157] S760, based on the second dataset, trains the sample classification model to obtain the action classification model.

[0158] Furthermore, based on the aforementioned racket swing detection dataset Racket Swing Classification Dataset A racket swing detection model and a racket swing classification model are constructed and trained separately. In this patent, considering the limitations of the computing power of wearable devices and the high accuracy requirements for badminton racket swing recognition, the main structure of both models adopts a two-layer gated recurrent unit (GRU) network with 64 hidden layer units. The structure of a single-layer GRU network is described by Formulas 12 to 15 below.

[0159] Formula 12:

[0160] z t =σ(W z ·[h t-1 ,x t ]+b z )

[0161] Formula Thirteen:

[0162] r t =σ(W r ·[h t-1 ,x t ]+b r )

[0163]

[0164] The swing action detection model consists of a two-layer GRU network followed by a Dropout layer and a fully connected layer. The output vector of the fully connected layer has a dimension of (N,1). Finally, the activation function (sigmoid) is used to map the output to the interval [0,1], which represents the probability that the model will determine the current swing action as a valid swing action.

[0165] In this embodiment, the swing action classification model adopts a multi-task learning architecture. After two layers of GRU network, there is a Dropout layer and three task-specific layers. The output h(x) of the GRU layer is mapped to the output layer of each task through the three task-specific layers f1(·), f2(·), and f3(·). The output shape of the forehand / backhand classification task layer f1(·) is (N,2), the output shape of the overhand / underhand classification task layer f2(·) is (N,2), and the output shape of the detailed action classification task layer f3(·) is (N,8). The activation function (softmax) is used to map the above outputs to the interval [0,1], which represents the probability that the model will classify the current valid swing action as forehand / backhand, overhand / underhand, or detailed action.

[0166] All the above models use the cross-entropy loss function and Adam optimizer to calculate model loss and update parameters. Among them, the swing action classification model adopts a multi-task learning network architecture, so its loss function can be expressed as Equation 16.

[0167]

[0168] Where L1, L2, and L3 represent the loss functions for the forehand / backhand classification task, the uphand / downhand classification task, and the detailed action classification task, respectively, and α, β, and γ represent the weight coefficients of the three tasks. After completing the offline training process of the above models, the optimal parameters of each model are saved for use in the online real-time application phase.

[0169] This is illustrative; please refer to it. Figure 8 It illustrates a flowchart of a swing motion recognition method provided in an exemplary embodiment of this application, including a training process and an application process, such as... Figure 8 As shown, the method includes the following steps.

[0170] Step 801: Collect data on the athlete's right-hand swing.

[0171] In the offline training phase of the recognition model, the first step is to acquire the motion data X and corresponding label data Y required for model training. Using wristband electronic devices equipped with accelerometers and gyroscopes, standard right-handed racket swing data is collected from N right-handed athletes. This includes three-axis acceleration data A for effective swing actions such as forehand, backhand, overhead, underhand, high clear, smash, and block. pos=(a x ,a y ,a z ) and triaxial angular velocity data G pos =(g x ,g y ,g z ).

[0172] Assuming the maximum complete acquisition period length of the above acceleration and angular velocity data is T, then for any time t, the acceleration and angular velocity data can be expressed as (a xt ,a yt ,a zt ) and (g xt ,g yt ,g zt ).

[0173] Furthermore, the resultant acceleration value a at any time t can be calculated using formulas one and two above. rt Resultant angular velocity value g rt Then the acceleration data at any time t is A. t =(a xt ,a yt ,a zt ,a rt ), angular velocity data is G t =(g xt ,g yt ,g zt ,g rt (Note: The superscripts for forward and reverse scenarios are omitted here.)

[0174] Step 802: Collect the reverse scene data of the athlete's right hand.

[0175] Acquire three-axis acceleration data A for reverse scene actions such as picking up a ball, catching a ball, and slapping one's thigh. neg =(a x ,a y ,a z ) and triaxial angular velocity data G neg =(g x ,g y ,g z ).

[0176] Step 803, data preprocessing.

[0177] In this embodiment, after obtaining the converted data, it is also necessary to preprocess the converted data to obtain preprocessed data.

[0178] Optionally, preprocessing includes at least one of data filtering, data denoising, time alignment, or time correction.

[0179] Among them, data filtering refers to removing interference signals generated in motion data through filtering techniques in digital signal processing; data denoising refers to removing noise signals in motion data, for example, by using the exponential moving average method to weight and sum the values ​​corresponding to the data at a certain moment and the data at several previous moments to eliminate noise signals in motion data, while amplifying the trend in the data; time alignment or time correction refers to adjusting motion data at different time points to the same time reference.

[0180] In this embodiment, mean filtering is used to filter the motion data, which reduces noise and smooths the data while taking into account the overall power consumption of the device.

[0181] Step 804, slide window splitting.

[0182] In illustration, preprocessed data is a set of data corresponding to different moments in a time series; therefore, preprocessed data can also be called motion data. As for the sample window, it is used to determine the time range corresponding to a single sample during the sampling and preprocessing of the data.

[0183] In a schematic manner, during the continuous sampling of preprocessed data through a sample window, the preprocessed data is sampled for the first time with the window length corresponding to the sample window as the time range, resulting in the first candidate window data. After moving the sample window according to the preset sampling interval, a new sample window is generated. The preprocessed data is then sampled for the second time using the new sample window to obtain the second candidate window data, and so on, until multiple candidate window data are finally obtained.

[0184] Step 805: Define window labels.

[0185] During the offline training phase, a label category needs to be assigned to each segmented window. As shown in Figure 3 above, the vertical dashed line marks the peak position of the waveform corresponding to a certain swing action. Assuming its time series index position is p, then if the left boundary index position of the segmented window i is left... i and right boundary index position i If the conditions in Formula 11 below are met, the window data segment is labeled as a swing category (i.e., category 0); otherwise, it is labeled as a non-swing category (i.e., category 1). For the window data labeled as a swing category, its corresponding forehand / backhand, overhand / underhand, and detailed swing action categories are further labeled. Specifically, the reverse scene multivariate motion data X... neg All window data labels processed by the sliding window segmentation module should be marked as non-swing category (i.e., category 1).

[0186] Step 806, racket swing action classification dataset.

[0187] Step 807, racket swing action detection dataset.

[0188] Step 808: Reverse the scene dataset.

[0189] To detect valid swing motions and classify them into forehand / backhand, overhand / underhand, and detailed motion categories, separate swing motion detection datasets need to be constructed. Racket Swing Classification Dataset Among them, the racket swing detection dataset The data comes from two sources: positive scene multivariate motion data X pos and reverse scene multivariate motion data X neg The latter is added to improve the robustness of the subsequent swing action detection model under reverse scenario conditions.

[0190] Therefore, the racket swing detection dataset can be represented as The racket swing classification dataset can be represented as Where N represents the number of sliding window samples in the dataset, T represents the length of the sliding window data, and D represents the number of time series variables. Let represent the label of sample i on the k-th task. Since there are a total of 3 classification tasks here, namely forward / backward hand classification, up / down hand classification, and detailed action category classification, the maximum value of k is 3.

[0191] Step 809: Train the action prediction model.

[0192] Step 810: Train the action classification model.

[0193] Furthermore, based on the aforementioned swing action detection dataset D and swing action classification dataset c, a swing action detection model and a swing action classification model are constructed and trained, respectively. In this patent, considering the limitations of the computing power of wearable devices and the high accuracy requirements for badminton swing action recognition, the main structure of both models adopts a two-layer GRU network with 64 hidden layer units.

[0194] The swing action detection model consists of a two-layer GRU network followed by a Dropout layer and a fully connected layer. The output vector of the fully connected layer has a dimension of (N,1). Finally, the sigmoid activation function is used to map the output to the interval [0,1], which represents the probability that the model will determine the current swing action as a valid swing action. In this patent, the swing action classification model adopts a multi-task learning architecture. A Dropout layer and three task-specific layers follow a two-layer GRU network. The output h(x) of the GRU layer is mapped to the output layer of each task through the three task-specific layers f1(·), f2(·), and f3(·). Specifically, the output shape of the forehand / backhand classification task layer f1(·) is (N,2), the output shape of the overhand / underhand classification task layer f2(·) is (N,2), and the output shape of the detailed action classification task layer f3(·) is (N,8). All of these outputs are mapped to the [0,1] interval using the softmax activation function, representing the probability that the model classifies the current valid swing action as a forehand / backhand, overhand / underhand, or detailed action. All of the above models use the cross-entropy loss function and the Adam optimizer for model loss calculation and parameter updates. The swing action classification model, due to its multi-task learning network architecture, is particularly noteworthy.

[0195] Step 811: Obtain real-time data such as the athlete's racket swing acceleration and angular velocity.

[0196] You can refer to step 801 above.

[0197] Step 812, symmetry transformation.

[0198] As an illustration, since the first object has different hand habits during movement, that is, some objects are left-handed and some are right-handed, and in most cases, the use of the right hand to perform actions is more common than the use of the left hand, when the first object uses the left hand to perform actions, it is necessary to perform a symmetrical transformation on the motion data to adjust it to the motion data generated by the right hand to improve the accuracy of subsequent action recognition.

[0199] In this embodiment, after obtaining the motion data in the above steps, if the motion data belongs to the motion data generated by the left hand of the first object, then the motion data is subjected to symmetry transformation processing to obtain the transformed data corresponding to the motion data.

[0200] In this embodiment, the motion data is waveform data generated according to a time series. That is, when the motion data is implemented as triaxial acceleration data and triaxial angular velocity data, a total of eight waveforms are generated, namely waveform A corresponding to the X-axis acceleration data, waveform B corresponding to the Y-axis acceleration data, waveform C corresponding to the Z-axis acceleration data, waveform D corresponding to the resultant acceleration data, waveform E corresponding to the X-axis angular velocity data, waveform F corresponding to the Y-axis angular velocity data, waveform G corresponding to the Z-axis angular velocity data, and waveform H corresponding to the resultant angular velocity data. A central axis (or baseline) is preset, and one or more waveforms are flipped around the central axis as the flip axis to obtain the converted waveform, which is used as the converted data. The specific execution process is as follows.

[0201] The first method is to perform symmetrical transformation of acceleration data: flip the waveform data corresponding to the X-axis acceleration data in the three-axis acceleration data according to the central axis (generally the central axis corresponding to the 0 value is selected), while the remaining Y-axis acceleration data, Z-axis data and resultant acceleration data remain unchanged.

[0202] The second method involves symmetrical transformation of angular velocity data: the waveform data corresponding to the Y-axis angular velocity data and Z-axis angular velocity data are flipped along the central axis (generally the 0-value central axis is selected), while the remaining X-axis angular velocity data and resultant angular velocity data remain unchanged.

[0203] Step 813, slide window splitting.

[0204] You can refer to step 805 above.

[0205] Step 814, swing motion detection.

[0206] To illustrate, a pre-trained action detection model is used to perform action detection on multiple candidate window data to obtain multiple candidate window data belonging to the swing action, which are then used as the action detection results.

[0207] Step 815, post-processing of racket swing detection.

[0208] In illustrative terms, waveform peak data refers to the data within a single candidate window that corresponds to the waveform peak. In this embodiment, it can also be referred to as the waveform peak position index.

[0209] After employing the aforementioned strategy to segment the sensor waveform data into continuous windows, the sensor waveform data of a single swing action will be repeatedly sampled, and subsequently repeatedly detected and classified by the swing action detection model and the swing action classification model. To address this issue, this patent first models the relationship between the aforementioned sliding window length L, sliding step length S, and the number of times the waveform corresponding to any swing action is repeatedly sampled.

[0210] The number of times the waveform peak corresponding to any swing action is repeatedly sampled and predicted under different combinations of window length and window sliding step length is calculated. Taking the swing action detection task in this patent as an example, it can be calculated that under this combination of conditions, the waveform peak corresponding to any swing action will be repeatedly predicted twice. Step 816, swing action classification.

[0211] Step 817: Post-processing of swing motion classification.

[0212] Step 818: Swing motion recognition output display.

[0213] In this embodiment, the deduplication window data is further input into the swing action classification model. At the same time, the probability of the effective swing action belonging to the forehand / backhand, overhand / underhand, and detailed action categories is determined. After deduplication after swing action classification, the specific category to which the swing action belongs is output as the action recognition result.

[0214] This is illustrative; please refer to it. Figure 9 This illustration shows a schematic diagram of a software architecture provided by an exemplary embodiment of this application. It includes a hardware layer, a driver layer, a hardware abstraction layer (HAL), also known as an abstraction layer, framework layer, and an application layer.

[0215] The abstraction layer is used to provide interfaces to the upper layers, shielding the implementation details of the underlying hardware.

[0216] After the badminton task management initialization is completed, control parameters will be passed down through the corresponding HAL layer to control the capture of acceleration and angular velocity data.

[0217] After receiving the control parameters, the accelerometer and gyroscope begin data acquisition. Once data is acquired, an interrupt is triggered to upload the data, which is then stored in a data buffer. Subsequently, the data in the buffer is transmitted sequentially to the data preprocessing algorithm and the swing action recognition algorithm through the function interface according to a certain buffer length. This outputs the detection and classification results of the swing actions contained in the current buffer data and reports them to the device interface to complete the display of the real-time recognition results of badminton swing actions.

[0218] This is illustrative; please refer to it. Figure 10 The diagram illustrates the structure of a real-time badminton racket swing action recognition device provided in an exemplary embodiment of this application, which mainly includes:

[0219] Sensor data acquisition module: used to collect in real time the motion trajectory data of the wrist of the badminton player wearing electronic device during the game, including three-axis acceleration data and three-axis angular velocity data, as well as the resultant acceleration data after data integration of the three-axis acceleration data and the resultant angular velocity data after data integration of the three-axis angular velocity data;

[0220] Data preprocessing module: used to filter, denoise, align, and standardize the motion data acquired by the sensor data acquisition module to obtain preprocessed data;

[0221] Sliding window data sampling module: This module is used to continuously slide sample motion data obtained by the data preprocessing module based on a first window with a preset window length and sampling interval, so as to obtain multiple candidate window data of a fixed length.

[0222] The motion recognition algorithm module mainly includes a symmetry transformation algorithm for left-hand swing data (which converts the motion data generated by the left wrist into right-hand motion data through symmetry), a swing motion detection algorithm (which identifies whether it is a swing motion), a swing motion classification algorithm (which identifies the specific swing motion category), and their respective post-processing deduplication algorithms.

[0223] The real-time display module is used to visualize the output results of the aforementioned racket swing recognition algorithm module.

[0224] Figure 11 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Figure 11 As shown, the electronic device 1100 may include a processor 1110, an external memory interface 1120, an internal memory 1121, a universal serial bus (USB) interface 1130, a charging management module 1140, a power management module 1141, a battery 1142, an antenna 1, an antenna 2, a mobile communication module 1150, a wireless communication module 1160, an audio module 1170, a speaker 1170A, a receiver 1170B, a microphone 1170C, a headphone jack 1170D, a sensor module 1180, buttons 1190, a motor 1191, an indicator 1192, a camera 1193, a display screen 1194, and a subscriber identification module (SIM) card interface 1195, etc.

[0225] The sensor module 1180 may include a pressure sensor 1180A, a gyroscope sensor 1180B, a barometric pressure sensor 1180C, a magnetic sensor 1180D, an accelerometer sensor 1180E, a distance sensor 1180F, a proximity sensor 1180G, a fingerprint sensor 1180H, a temperature sensor 1180J, a touch sensor 1180K, an ambient light sensor 1180L, a bone conduction sensor 1180M, etc.

[0226] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1100. In other embodiments of this application, the electronic device 1100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0227] For example, Figure 11 The processor 1110 shown may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0228] The controller can serve as the nerve center and command center of the electronic device 1100. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0229] In this embodiment of the application, the processor 1110 is mainly used to launch the application loading page and respond to user input interaction operations.

[0230] The processor 1110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1110 is a cache memory. This memory can store instructions or data that the processor 1110 has just used or that are used repeatedly. If the processor 1110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 1110, and thus improves the efficiency of the system.

[0231] Electronic device 1100 implements display functions through a GPU, a display screen 11114, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 11114 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 1110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0232] The display screen 1194 is used to display images, videos, etc. The display screen 1194 includes a display panel. In some embodiments, the electronic device 1100 may include one or N display screens 1194, where N is a positive integer greater than 1.

[0233] In this embodiment, the various interfaces are mainly presented to the user through the display screen 1194, for example... Figure 1 The various interfaces within.

[0234] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0235] In this embodiment, the NPU can be used to identify the display content and instruct the processor 1110 to generate the display content, for example, it can instruct the processor 1110 to generate the display content according to... Figure 1 The content is generated and displayed in the form shown.

[0236] Pressure sensor 1180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 1180A can be disposed on display screen 1194. There are many types of pressure sensors 1180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 1180A, the capacitance between the electrodes changes. Electronic device 1100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 1194, electronic device 1100 detects the touch operation intensity based on pressure sensor 1180A. Electronic device 1100 can also calculate the touch position based on the detection signal from pressure sensor 1180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with a touch operation intensity less than a first pressure threshold is applied to the SMS application icon, a command to view SMS messages is executed. When a touch operation with a strength greater than or equal to the first pressure threshold is applied to the SMS application icon, the instruction to create a new SMS message is executed.

[0237] In this embodiment, the pressure sensor 1180A is mainly used to collect user interaction operations, such as swiping operations and clicking operations.

[0238] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0239] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0240] This application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, it can implement the steps in any of the above methods.

[0241] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0242] This application provides a computer program product, which includes a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0243] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0244] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0245] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0246] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0247] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0248] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0249] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0250] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0251] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0252] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0253] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for recognizing racket swing motions, characterized in that, The method is applied to an electronic device worn by a first object, and the method includes: During the movement of the first object, motion data of the first object is acquired, and the motion data is used to describe the motion state of the first object; The motion data is continuously sampled through multiple first windows to obtain multiple candidate window data corresponding to the motion data. The multiple candidate window data corresponds one-to-one with the multiple first windows. The i-th candidate window data is obtained by collecting the motion data with the i-th first window as the sampling range. The window length of the first window is a first preset value, the sampling interval between two adjacent first windows is a second preset value, the second preset value is less than the first preset value, and i is a positive integer. Action recognition is performed on the multiple candidate window data to obtain the action recognition result corresponding to the motion data. The action recognition result is used to indicate the action content category of the first object.

2. The method according to claim 1, characterized in that, The first preset value is greater than the first threshold and less than the second threshold; the second preset value is greater than the third threshold and less than the fourth threshold, wherein the third threshold is less than the first threshold and the fourth threshold is less than the second threshold.

3. The method according to claim 1 or 2, characterized in that, Before continuously sampling the motion data to obtain multiple candidate window data corresponding to the motion data during the sliding of the first window, the method further includes: If the motion data meets the first condition, the motion data is subjected to symmetric transformation to obtain transformed data. The transformed data refers to the data obtained after the motion data is mirrored and flipped according to a preset baseline. The step of continuously sampling the motion data to obtain multiple candidate window data corresponding to the motion data includes: The transformed data is continuously sampled to obtain the multiple candidate window data.

4. The method according to claim 3, characterized in that, The motion data includes acceleration data, and the acceleration data includes X-axis acceleration data; The step of performing a symmetric transformation on the motion data to obtain transformed data includes: Using the preset baseline as a reference, the X-axis acceleration data is mirrored and flipped in a preset direction to obtain the X-axis converted acceleration data.

5. The method according to claim 3, characterized in that, The motion data includes angular velocity data, which includes Z-axis angular velocity data and Y-axis angular velocity data; The step of performing a symmetric transformation on the motion data to obtain transformed data includes: Using the preset baseline as a reference, the Y-axis angular velocity data is mirrored and flipped in a preset direction to obtain the Y-axis converted angular velocity data; Using the preset baseline as a reference, the Z-axis angular velocity data is mirrored and flipped according to the preset direction to obtain the Z-axis converted angular velocity data.

6. The method according to claim 1 or 2, characterized in that, Before performing action recognition on the multiple candidate window data to obtain the action recognition result corresponding to the motion data, the method further includes: Obtain the waveform peak data corresponding to the candidate window data; The number of repeated samplings corresponding to the candidate window data is determined based on the waveform peak data, the window length, and the sampling interval. Based on the number of repeated samplings, the multiple candidate window data are deduplicated to obtain deduplicated window data; The step of performing action recognition on the multiple candidate window data to obtain the action recognition result corresponding to the motion data includes: Action recognition is performed on the multiple deduplicated window data to obtain the action recognition result corresponding to the motion data.

7. The method according to claim 1 or 2, characterized in that, The step of performing action recognition on the multiple candidate window data to obtain the action recognition result corresponding to the motion data includes: The multiple candidate window data are processed by an action prediction model to obtain action detection results corresponding to the multiple candidate window data respectively. The action detection results are used to indicate whether the candidate window data belongs to a swing action. If the candidate window data belongs to the swing action, the candidate window data is classified by an action classification model to obtain the action recognition result corresponding to the candidate window data.

8. The method according to claim 7, characterized in that, Before performing action detection processing on the multiple candidate window data using an action prediction model to obtain the action detection results corresponding to the multiple candidate window data respectively, the method further includes: Acquire sample motion data; The sample motion data is continuously sampled through multiple sample windows to obtain multiple sample window data corresponding to the sample motion data. The sample window data is categorized to obtain category labels corresponding to the multiple sample window data, and the category labels include at least one of swing category labels and action category labels; A first dataset is generated based on the swing category labels, and a second dataset is generated based on the action category labels; The action prediction model is obtained by training the sample prediction model based on the first dataset; The action classification model is obtained by training the sample classification model based on the second dataset.

9. The method according to claim 8, characterized in that, The motion content corresponding to the sample motion data includes at least one of the following: The swing motion during the swing process; Non-swinging movements during the swing; Actions outside of the swinging motion.

10. An electronic device, characterized in that, The electronic device includes a memory, one or more processors, and a computer program stored in the memory and executable on the processor, wherein the one or more processors, when executing the computer program, cause the electronic device to perform the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by an electronic device, implements the method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1 to 9 to be performed.

Images