Skiing behavior identification method and system based on multi-position multi-branch network

Abstract

The invention belongs to the technical field of behavior recognition, and provides a skiing behavior recognition method and system based on a multi-position multi-branch network. Comprising the following steps: processing original data acquired by different body position sensors to obtain a feature data block corresponding to each position; constructing a multi-branch network model, wherein the multi-branch network model comprises sub-networks corresponding to the position sensors; each sub-network inputs a feature data block corresponding to a position and outputs a feature representation of the position, on one hand, the feature representation is used for executing a position identification task, and on the other hand, the feature representation is used for feature fusion; fusing the feature representations output by the sub-networks in a time dimension and a space dimension to obtain time feature data and space feature data; and inputting the time feature data and the spatial feature data into a behavior recognition network to execute a behavior recognition task, and outputting a skiing behavior recognition result. The problem that in the prior art, different position information cannot be fully utilized to contribute to an identification task is solved.

Description

Technical Field

[0001] This invention belongs to the field of behavior recognition technology, and in particular relates to a skiing behavior recognition method and system based on a multi-location, multi-branch network. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] As people's living standards improve, they are increasingly pursuing a richer spiritual life. Sports, which offer both competitive enjoyment and improved physical fitness, have become an important leisure activity. Among these, skiing has gained popularity in recent years, attracting numerous enthusiasts to challenge themselves and improve their skills.

[0004] Skiing is divided into different disciplines, each containing different behaviors such as turning, jumping, and gliding. Each behavior can be further subdivided according to skill, such as snowplowing and parallel gliding. Different skiing behaviors play different roles in different disciplines and can even determine the level of performance in a skiing event. Therefore, identifying these key behaviors helps to locate and analyze them, evaluate the mastery of these behaviors, and ultimately optimize skiing skills.

[0005] Based on data type, existing research on skiing behavior recognition can be mainly divided into two categories: visual information-based and sensor information-based. Visual information-based research relies on cameras, which are complex to deploy, costly, and easily affected by the environment, hindering large-scale application. Sensor information-based research, on the other hand, is easy to install and low-cost, allowing for convenient integration into wearable devices with minimal discomfort to skiers. Therefore, its research results hold greater promise for practical application. However, most of this type of research focuses on single-point information and primarily uses sensors installed on the upper body, neglecting the contribution of other location information to behavior recognition. Some studies also install sensors in multiple locations, but during information fusion, they generally use simple splicing, failing to fully utilize the contribution of information from different locations to the task. Summary of the Invention

[0006] The purpose of this invention is to provide a skiing behavior recognition method and system based on multi-location multi-branch networks. The method constructs a multi-branch network model, extracts features with higher semantic meaning, and then fuses them to solve the technical problem in the prior art that it cannot fully utilize the contribution of information from different locations to the recognition task.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of this invention provides a skiing behavior recognition method based on a multi-location, multi-branch network, comprising: Raw data is collected from sensors fixed at at least two different body positions of the skier; The raw data is processed to obtain the feature data block corresponding to each location; A multi-branch network model is constructed, which includes sub-networks corresponding to each location sensor. Each sub-network takes a feature data block corresponding to a location as input and outputs the feature representation of that location. Each sub-network is set up with a location recognition task, which takes the feature representation as input and is used to perform the location recognition task and feature fusion. The feature representations output by each sub-network are fused in the time and spatial dimensions respectively to obtain temporal feature data and spatial feature data; The temporal and spatial feature data are input into the behavior recognition network to perform the behavior recognition task. The temporal and spatial feature data are fused based on an attention mechanism before the classification layer, and the skiing behavior recognition result is output through the classification layer.

[0008] As a further technical solution, the processing of the original data to obtain the feature data block corresponding to each position specifically includes: Attitude calculations are performed on the raw data from the sensors to extract attitude features; the raw data from the sensors includes triaxial acceleration, triaxial angular velocity, and triaxial magnetic field data. Standardize the pose features; The standardized data is divided into multiple feature data blocks of equal length using a sliding window.

[0009] As a further technical solution, the sub-network includes a Bi-LSTM network layer, a Relu+Dropout layer, and a fully connected layer connected in sequence.

[0010] As a further technical solution, the behavior recognition network includes: The first branch includes a one-dimensional convolutional layer, an attention mechanism layer, a bidirectional long short-term memory network layer, a ReLU+Dropout layer, and a fully connected layer. The second branch includes at least one two-dimensional convolutional layer, at least one BN+ReLU layer, and one fully connected layer.

[0011] As a further technical solution, the output features of the two branches are fused based on the attention mechanism and then input into the classification layer, which includes two fully connected layers, to obtain the final behavior recognition network output for behavior recognition tasks; specifically, spatial features / temporal features are respectively used as Q in the attention mechanism, and temporal features / spatial features are respectively used as K and V.

[0012] Where Q is the query, K is the key, and V is the value; Q and K are multiplied to obtain the similarity score between each query vector and all key vectors. The similarity score is the weight of each key vector. After normalization, it is multiplied by V to obtain the weighted feature representation.

[0013] As a further technical solution, before inputting the feature data block into the multi-branch network model, data augmentation is also included: The sequence of feature data blocks from the same batch, from different location sensors, but which are time-corresponding, is randomly adjusted so that the adjusted individual feature data blocks maintain a temporal correspondence and simultaneously contain data from different locations.

[0014] As a further technical solution, the fusion of the time dimension specifically involves: splicing the feature representations output by each sub-network according to time to form a time feature data that integrates different location information; the fusion of the spatial dimension specifically involves: splicing the feature representations output by each sub-network according to location to form a spatial feature data that integrates different location information.

[0015] A second aspect of the present invention provides a skiing behavior recognition system based on a multi-location, multi-branch network, comprising: The data acquisition module is configured to acquire raw data from sensors fixed at at least two different body positions of the skier; The data processing module is configured to process the raw data to obtain feature data blocks corresponding to each location; The data acquisition module is configured to: construct a multi-branch network model, which includes sub-networks corresponding to each location sensor; The multi-branch network model building module is configured such that each sub-network takes a feature data block corresponding to a location as input and outputs the feature representation of that location; each sub-network is set up with a location recognition task, which takes the feature representation as input, and is used to perform the location recognition task on the one hand, and feature fusion on the other hand. The feature fusion module is configured to fuse the feature representations output by each sub-network in the time dimension and the spatial dimension, respectively, to obtain temporal feature data and spatial feature data. The behavior recognition module is configured to: input the temporal feature data and spatial feature data into the behavior recognition network to perform the behavior recognition task; fuse the temporal feature data and spatial feature data based on an attention mechanism before the classification layer; and output the skiing behavior recognition result through the classification layer.

[0016] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a skiing behavior recognition method based on a multi-location, multi-branch network as described in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a skiing behavior recognition method based on a multi-location multi-branch network as described in the first aspect of the present invention.

[0018] The technical solution of the present invention has the following beneficial effects: This invention provides a method for skiing behavior recognition based on multi-location information fusion. Unlike existing research that directly splices together different location information, this method designs sub-networks for different location information and extracts high-level features for fusion.

[0019] Based on multi-location information, this invention proposes a multi-branch network, which specifically introduces a location recognition task to highlight the information differences between different locations, thereby obtaining better feature representation.

[0020] This invention is based on a multi-location, multi-branch network that makes full use of the contribution of temporal and spatial information to task learning, enabling the model to achieve better performance.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of the method in the first embodiment; Figure 2 This is a schematic diagram of the subnetwork in the first embodiment; Figure 3 This is a schematic diagram of the sensor installation location in the first embodiment. Detailed Implementation

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0027] Example 1 like Figure 1 As shown, this embodiment discloses a skiing behavior recognition method based on a multi-location, multi-branch network; including: S1: Collect raw data from sensors fixed at at least two different body positions of the skier; S2: Process the raw data to obtain the feature data block corresponding to each position; S3: Construct a multi-branch network model, which includes sub-networks corresponding to each location sensor; each sub-network takes a feature data block corresponding to a location as input and outputs the feature representation of that location; each sub-network is set up with a location recognition task, taking the feature representation as input, which is used to perform the location recognition task on the one hand and for feature fusion on the other. S4: The feature representations output by each sub-network are fused in the time and spatial dimensions to obtain temporal feature data and spatial feature data. S5: Input temporal and spatial feature data into the behavior recognition network to perform the behavior recognition task. Before the classification layer, the temporal and spatial feature data are fused based on the attention mechanism, and the skiing behavior recognition result is output through the classification layer.

[0028] In this embodiment, the inertial measurement unit is a sensor. "Multiple positions" means that the sensor is installed in different locations, but it is necessary to ensure that there is at least one sensor on the upper and lower body respectively. As for scalability, other installation positions can be added on this basis, such as... Figure 3 Examples 1 and 2 are shown, with the sensor installed in the interscapular region and the leg. The sensor is typically an inertial measurement unit (IMU), but other commonly used measurement sensors such as GPS can also be integrated. In this embodiment, the sensor includes both an IMU and a magnetometer.

[0029] In S1, the specific process of acquiring data includes: S1.1: Data Collection. In this embodiment, sensors are fixed to the interscapular region and legs of the test personnel to collect various behavioral data. The raw data collected by the sensors includes triaxial acceleration, triaxial angular velocity, and triaxial magnetic field data. To obtain annotation information, methods such as deploying cameras, having staff follow and film, or using drones can be selected to record the test personnel's movement process, or recording and data collection can be completed according to specified actions.

[0030] In S2, the raw data is processed as follows: S2.1: Calculate attitude features. The attitude is calculated based on Kalman filtering of the raw sensor data to obtain attitude features.

[0031] S2.2: Data standardization, using the Z-Score standardization method to process the data.

[0032] In the formula, The mean of the data. The standard deviation is denoted as .

[0033] The above steps yield the characteristics of a single-point location, including acceleration, angular velocity, magnetic field, and attitude information. Further features such as velocity, maximum value, minimum value, mean, and standard deviation can be derived from these. These features are all derived from sensor data. In this embodiment, only acceleration, angular velocity, attitude information, and velocity are used as example features. For scalability, other derived features can be added. The characteristics of a single-point location are represented by a 12-dimensional data set. The subscript S indicates the interscapular region, and leg features are represented by... express.

[0034] S2.3: Data segmentation. Use a window of size 100 to segment the standardized data at different locations with a step size of 50, forming multiple feature data blocks of equal length to construct the dataset.

[0035] S2.4: Data Labeling. Using the labeling information obtained during data collection, the above data is labeled, including turning, jumping, gliding, walking, riding the cable car or magic carpet, falling, and other behaviors. As an expandable feature, other skiing behaviors can also be added.

[0036] Although data from different sensors is collected under the same action, theoretically, the features exhibited at different positions under the same action will differ and have different emphases. For example, when turning, the leg movement will be greater than the upper body movement. Therefore, simply stitching together features from different positions will result in the loss of some useful information. To effectively utilize feature information, a position recognition task is added to the action recognition task, namely, determining whether the current input data comes from the interscapular region or other positions; the position information serves as the annotation information. In this embodiment, the input data for the multi-branch network model can be obtained through the above steps. To enhance the model's generalization performance, data augmentation is also included before inputting the feature data blocks into the multi-branch network model. Specifically: As described above, the data at each location will be divided into N window sizes of L (L=100), corresponding to N labeled data. At this point, the N data points at each location are temporally corresponding. The first step in shuffling the data is to ensure that the data at each location still maintains a temporal correspondence, but to change the original order of the data at the same location. The resulting dataset is then divided in a 4:1 ratio to obtain training and testing sets. The training set is further divided in a 9:1 ratio to obtain training and validation sets. The batch size of the network input is set to B. Therefore, the input data at each location when inputting into the network is... The second step in shuffling the data is to randomly rearrange the data at different positions in the current batch before inputting it into the network. The data originates from either the interscapular region or the legs. After being randomly swapped at corresponding times, the original data will now include data from both the interscapular region and the legs, while maintaining a one-to-one correspondence in time.

[0037] In S3, the construction of a multi-branch network model specifically includes: Two sub-networks are constructed, each consisting of a Bi-LSTM network layer, a Relu+Dropout layer, and a fully connected layer connected sequentially.

[0038] The two subnetworks have input dimensions of... The feature data is given by B, where B is the batch size, L is the feature data length (i.e., the time length), and d is the feature dimension of the feature data at each time step. The data is then passed sequentially through a Bi-LSTM network layer, a ReLU+Dropout layer, and a fully connected layer to obtain the feature representation at that location. This feature indicates that it will be used for both location recognition tasks and feature fusion.

[0039] In S4, further, such as Figure 2 As shown, the above feature fusion is divided into two steps: The feature representations output by each subnetwork are concatenated according to time to form a temporal feature data that integrates information from different locations. In short, time characteristics It is the dimension after fusion.

[0040] The feature representations output by each subnetwork are concatenated according to their location to form a spatial feature data that integrates information from different locations. , or spatial feature for short, where n represents n different positions, which is 2 in this embodiment.

[0041] In S5, fused temporal feature data and fused spatial feature data are input into the behavior recognition network to perform the behavior recognition task, specifically as follows: First, construct the behavior recognition network: The first branch includes a one-dimensional convolutional layer, an attention mechanism layer, a bidirectional long short-term memory network layer, a ReLU+Dropout layer, and a fully connected layer. The second branch includes two two-dimensional convolutional layers, two BN+ReLU layers, and one fully connected layer.

[0042] Then, input the temporal features into the first branch and the spatial features into the second branch.

[0043] Finally, the output features of the two branches are fused and then passed through two fully connected layers to obtain the final behavior recognition network output, which is used for behavior recognition tasks.

[0044] To further improve model performance, the fusion of the output features of the two branches is not a simple direct concatenation and addition, but based on the attention mechanism, spatial features and temporal features are respectively used as Q in the attention mechanism, and correspondingly, temporal features and spatial features are respectively used as K and V.

[0045] Where Q is the query, K is the key, and V is the value. Q and K are multiplied to obtain the similarity score between each query vector and all key vectors. The similarity score is the weight of each query vector. After normalization, it is multiplied by V to obtain the weighted feature representation.

[0046] The above methods obtain spatially relevant temporal information from a spatial perspective and, similarly, temporally relevant spatial information from a temporal perspective, achieving a symmetrical, bidirectional information flow. This integrates temporal features with spatial information, and vice versa. To avoid semantic loss, a residual structure is introduced, adding the output of the attention mechanism to the input separately, addressing gradient vanishing, exploding, and network degradation issues in deep learning. Finally, the two features processed by the residual connection are added together to obtain the fused feature.

[0047] The loss function for different tasks is calculated, and the network is fed back for learning. This process is repeated multiple times until the model reaches its optimal state. In this embodiment, cross-entropy loss is used, as detailed below:

[0048] in The total number of categories, For real labels, For prediction The probability of the category.

[0049] As a scalable technical solution, behavior recognition networks also include Batch Normalization (BN) layers, Dropout layers, and activation function layers.

[0050] Compared to ordinary fusion methods, this method greatly leverages the feature representation of different information sources, thereby significantly enriching feature representation.

[0051] Example 2 This embodiment discloses a skiing behavior recognition system based on a multi-location, multi-branch network, including: The data acquisition module is configured to acquire raw data from sensors fixed at at least two different body positions of the skier; The data processing module is configured to process the raw data to obtain feature data blocks corresponding to each location; The data acquisition module is configured to: construct a multi-branch network model, which includes sub-networks corresponding to each location sensor; The multi-branch network model building module is configured such that each sub-network takes a feature data block corresponding to a location as input and outputs the feature representation of that location; each sub-network is set up with a location recognition task, which takes the feature representation as input, and is used to perform the location recognition task on the one hand, and feature fusion on the other hand. The feature fusion module is configured to cross-fuse the feature representations output by each sub-network in the time and spatial dimensions to obtain fused temporal feature data and fused spatial feature data. The behavior recognition module is configured to input the fused temporal feature data and fused spatial feature data into the behavior recognition network to perform the behavior recognition task and output the skiing behavior recognition result.

[0052] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium. A computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of a skiing behavior recognition method based on a multi-location, multi-branch network as described in Embodiment 1 of this disclosure.

[0053] Example 4 The purpose of this embodiment is to provide an electronic device. An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in a skiing behavior recognition method based on a multi-location, multi-branch network as described in Embodiment 1 of this disclosure.

[0054] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0055] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0056] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A skiing behavior recognition method based on a multi-location, multi-branch network, characterized in that, include: Raw data is collected from sensors fixed at at least two different body positions of the skier; The raw data is processed to obtain the feature data block corresponding to each location; A multi-branch network model is constructed, which includes sub-networks corresponding to each location sensor. Each sub-network takes a feature data block corresponding to a location as input and outputs the feature representation of that location. Each sub-network is set up with a location recognition task, which takes the feature representation as input and is used to perform the location recognition task and feature fusion. The feature representations output by each sub-network are fused in the time and spatial dimensions respectively to obtain temporal feature data and spatial feature data; The temporal and spatial feature data are input into the behavior recognition network to perform the behavior recognition task. The temporal and spatial feature data are fused based on an attention mechanism before the classification layer, and the skiing behavior recognition result is output through the classification layer.

2. The skiing behavior recognition method based on a multi-location, multi-branch network as described in claim 1, characterized in that, The process of processing the original data to obtain the feature data block corresponding to each location specifically includes: Attitude calculations are performed on the raw data from the sensors to extract attitude features; the raw data from the sensors includes triaxial acceleration, triaxial angular velocity, and triaxial magnetic field data. Standardize the pose features; The standardized data is divided into multiple feature data blocks of equal length using a sliding window.

3. The skiing behavior recognition method based on a multi-location, multi-branch network as described in claim 1, characterized in that, The sub-network comprises a Bi-LSTM network layer, a Relu+Dropout layer, and a fully connected layer connected in sequence.

4. The skiing behavior recognition method based on a multi-location, multi-branch network as described in claim 1, characterized in that, The behavior recognition network includes: The first branch includes a one-dimensional convolutional layer, an attention mechanism layer, a bidirectional long short-term memory network layer, a ReLU+Dropout layer, and a fully connected layer. The second branch includes at least one two-dimensional convolutional layer, at least one BN+ReLU layer, and one fully connected layer.

5. The skiing behavior recognition method based on a multi-location, multi-branch network as described in claim 4, characterized in that, The output features of the two branches are fused based on the attention mechanism and then input into the classification layer, which includes two fully connected layers, to obtain the final behavior recognition network output for behavior recognition tasks. Specifically, spatial features and temporal features are respectively used as Q in the attention mechanism, and temporal features and spatial features are respectively used as K and V. Where Q is the query, K is the key, and V is the value; Q and K are multiplied to obtain the similarity score between each query vector and all key vectors. The similarity score is the weight of each key vector. After normalization, it is multiplied by V to obtain the weighted feature representation.

6. The skiing behavior recognition method based on a multi-location, multi-branch network as described in claim 1, characterized in that, Before inputting the feature data blocks into the multi-branch network model, data augmentation is also included: The sequence of feature data blocks from the same batch, from different location sensors, but which are time-corresponding, is randomly adjusted so that the adjusted individual feature data blocks maintain a temporal correspondence and simultaneously contain data from different locations.

7. The skiing behavior recognition method based on a multi-location, multi-branch network as described in claim 1, characterized in that, The cross-fusion of the time dimension specifically involves concatenating the feature representations output by each sub-network according to time to form a fused time feature data that integrates information from different locations; the cross-fusion of the spatial dimension specifically involves concatenating the feature representations output by each sub-network according to location to form a fused spatial feature data that integrates information from different locations.

8. A skiing behavior recognition system based on a multi-location, multi-branch network, characterized in that, include: The data acquisition module is configured to acquire raw data from sensors fixed at at least two different body positions of the skier; The data processing module is configured to process the raw data to obtain feature data blocks corresponding to each location; The data acquisition module is configured to: construct a multi-branch network model, which includes sub-networks corresponding to each location sensor; The multi-branch network model building module is configured such that each sub-network takes a feature data block corresponding to a location as input and outputs the feature representation of that location; each sub-network is set up with a location recognition task, which takes the feature representation as input, and is used to perform the location recognition task on the one hand, and feature fusion on the other hand. The feature fusion module is configured to fuse the feature representations output by each sub-network in the time dimension and the spatial dimension, respectively, to obtain temporal feature data and spatial feature data. The behavior recognition module is configured to: input the temporal feature data and spatial feature data into the behavior recognition network to perform the behavior recognition task; fuse the temporal feature data and spatial feature data based on an attention mechanism before the classification layer; and output the skiing behavior recognition result through the classification layer.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the skiing behavior recognition method based on a multi-location multi-branch network as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the skiing behavior recognition method based on a multi-location multi-branch network as described in any one of claims 1-7.

Images