Foot treading recognition method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a foot treading recognition method and device, equipment and a storage medium. The method comprises the steps that the treading states of at least two target parts of the feet of a user are determined, and the target parts are the parts, capable of making contact with the ground, of the feet when the feet fall to the ground; and according to the trampling state change of each target part, determining a trampling event of the target part. According to the embodiment of the invention, the treading events of at least two target parts of the foot can be accurately recognized, various treading actions executed by the foot through different target parts can be accurately recognized, and the diversity of foot treading recognition is ensured.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a method, device, equipment and storage medium for identifying foot stepping. Background Art

[0002] In various virtual reality scenarios supported by Extended Reality (XR) technology, corresponding information interaction can usually be achieved by performing corresponding stepping movements with the user's feet. For example, the user can experience corresponding dance machines, music-based stepping games, etc. by performing stepping movements with the feet in a virtual reality scenario.

[0003] At present, the existing foot stepping recognition methods usually directly analyze the stepping states represented by various stepping actions performed by the entire foot as a whole, which makes the foot stepping recognition have certain limitations. Summary of the invention

[0004] The embodiments of the present application provide a foot stepping recognition method, device, equipment and storage medium to achieve accurate recognition of stepping events at at least two target parts of the foot and ensure the diversity of foot stepping recognition.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying foot stepping, the method comprising:

[0006] Determining the pedaling status of at least two target parts of the user's foot according to the pedaling motion data of the user's foot, wherein the target parts are parts that can be contacted with the ground when the foot lands;

[0007] According to the change of the trampling state of each target part, the trampling event of the target part is determined.

[0008] In a second aspect, an embodiment of the present application provides a foot stepping recognition device, the device comprising:

[0009] A pedaling state determination module, used to determine the pedaling state of at least two target parts of the user's foot according to the pedaling motion data of the user's foot, wherein the target parts are parts that can be contacted with the ground when the foot lands;

[0010] The foot stepping determination module is used to determine the stepping event of each target part according to the stepping state change of the target part.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0012] A processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the foot stepping recognition method provided in the first aspect of the present application.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, which enables a computer to execute the foot stepping recognition method provided in the first aspect of the present application.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, which enable a computer to execute the foot stepping recognition method provided in the first aspect of the present application.

[0015] Through the technical solution of the present application, for at least two target parts of the contactable parts between the user's feet and the ground when the user lands on the ground, the trampling state of each target part can be determined. Then, according to the trampling state change of each target part, the trampling event of the target part is determined, thereby achieving accurate recognition of the trampling events of at least two target parts of the foot, supporting accurate recognition of various trampling actions performed by the foot through different target parts, and ensuring the diversity of foot trampling recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of a foot stepping recognition method provided in an embodiment of the present application;

[0018] Figure 2 An exemplary schematic diagram of a target part of a foot provided in an embodiment of the present application;

[0019] Figure 3 An exemplary schematic diagram of a stampede event at a target part provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of the principle of a process for determining an overall foot stepping event provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of the structure of a stepping identification system provided in an embodiment of the present application;

[0022] Figure 6a , Figure 6b , Figure 6c and Figure 6d They are respectively exemplary model structure diagrams of four different structures of the stepping recognition model provided in the embodiments of the present application;

[0023] Figure 7 A schematic diagram of the structure of a training data acquisition system for a stepping recognition model provided in an embodiment of the present application;

[0024] Figure 8 A principle block diagram of a foot stepping recognition device provided in an embodiment of the present application;

[0025] Fig. 9 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0029] Before introducing the specific technical solution of the present application, the application scenario of the present application is first described accordingly:

[0030] The implementation scheme of the present application can be applied to any scenario that can support the user's feet to trigger corresponding information interaction by performing various stepping actions. Among them, the application scenario may include triggering information interaction supported by a relevant device in the real environment in which the user is currently located through the stepping action performed by the user's feet, such as a dance machine in a real environment. The application scenario may also include the user entering a virtual reality environment through an XR device, and triggering corresponding information interaction in the virtual reality environment through the stepping action performed by the user's feet, such as a dance machine in a virtual reality environment, a music stepping game, etc. This application does not limit the specific application scenario, and can support the user's feet to trigger corresponding information interaction by performing various stepping actions.

[0031] Taking the virtual reality environment as an example, the application scenario of the present application can be exemplified as follows:

[0032] In a virtual reality environment, users can perform various stepping movements with their feet to trigger corresponding information interactions, such as dance machines and music stepping games in a virtual reality environment. In order to achieve foot stepping interactions in a virtual reality environment, the present application can display the corresponding virtual reality environment through the display of any electronic device that communicates with a display and one or more input devices. The electronic device can be any extended reality (Extended Reality, abbreviated as XR) device, which can specifically include virtual reality (Virtual Reality, abbreviated as VR) devices, augmented reality (Augmented Reality, abbreviated as AR) devices, and mixed reality (Mixed Reality, abbreviated as MR) devices, etc., and the present application does not limit this.

[0033] The display may be any display screen that has a communication connection with the electronic device. For example, the display may be a display screen configured by a head mounted display device on a VR device, an AR device, or an MR device, which is not limited in this application.

[0034] Moreover, in order to realize normal interaction of users in the virtual reality environment, the present application can initiate corresponding interactive operations on the virtual reality environment displayed by the display through one or more input devices that communicate with the above-mentioned electronic devices, thereby supporting users to perform various interactions in the virtual reality environment.

[0035] Among them, one or more input devices can be any control device and information collection device that has a communication connection with the above electronic device. For example, one or more input devices can be a handle configured on a VR device, an AR device, or an MR device, or a collection module for detecting hand operation and eye movement information, or a voice collector for collecting user voice information, etc., and this application does not limit this.

[0036] In the present application, in order to accurately identify various pedaling actions performed by the user's feet, one or more input devices may be a collection module for detecting pedaling motion data when the user's feet perform various pedaling actions.

[0037] At present, in order to solve the problem of certain limitations in foot stepping recognition when directly analyzing the overall stepping action of the entire foot through a binary classification model to identify the stepping state of the foot, the inventive concept of the present application is: according to the contactable part between the user's foot and the ground when the user's foot lands, at least two target parts can be distinguished on the user's foot, and the stepping state of each target part can be determined. According to the change in the stepping state of each target part, the stepping event of the target part can be determined. Then, by comprehensively referring to the stepping events of at least two target parts of the foot, the overall stepping event of the foot can be determined, thereby achieving accurate identification of the foot stepping event. By comprehensively referring to the stepping events of multiple target parts of the foot, it supports accurate identification of various stepping actions performed by the foot through different target parts, ensuring the comprehensiveness of foot stepping recognition.

[0038] Figure 1 A flowchart of a foot stepping recognition method provided in an embodiment of the present application, which can be applied to XR devices, but is not limited thereto. The method can be executed by a foot stepping recognition device provided in the present application, wherein the foot stepping recognition device can be implemented by any software and / or hardware. Exemplarily, the foot stepping recognition device can be configured in an electronic device such as AR / VR / MR that can simulate a virtual reality scene, and the present application does not impose any restrictions on the specific type of electronic device.

[0039] Specifically, Figure 1 As shown, the method may include the following steps:

[0040] S110, determining the pedaling states of at least two target parts of the user's foot according to the pedaling motion data of the user's foot.

[0041] In a virtual reality environment, corresponding information interaction can usually be achieved by performing corresponding stepping actions with the user's feet. Therefore, in order to ensure accurate interaction of users in the virtual reality environment, it is usually necessary to accurately determine whether the user's feet are stepping on the ground in the virtual reality environment as a stepping recognition task of the user's feet to achieve various interactions in the virtual reality environment.

[0042] Considering that the user's foot is not a simple point, but includes multiple different parts. Then, for the foot stepping recognition task, the user's foot can step on the ground through multiple different parts that can touch the ground to perform the corresponding stepping action. Since when stepping on the ground through a certain part that can touch the ground when the foot lands, there is no need to consider the state between other parts of the foot and the ground, which shows that the stepping actions performed by the user's foot are diverse.

[0043] Exemplarily, the stepping actions performed by the user's feet may include the following: stepping on the ground with the toes when the heels touch the ground, stepping on the ground with the heels when the toes touch the ground, stepping on the ground with the toes when the heels are suspended in the air, stepping on the ground with the heels when the toes are suspended in the air, stepping straight up and down with the toes and heels when the sole of the foot is parallel to the ground, etc.

[0044] Therefore, in order to ensure the comprehensiveness of foot stepping recognition, the present application can determine at least two target parts of the foot by analyzing the contactable parts between the user's foot and the ground when the foot lands, so as to support various stepping actions performed through the above target parts of the foot. Among them, the contactable parts between the user's foot and the ground when the foot lands can include but are not limited to the toes, heels, toes, forefoot, side soles, etc.

[0045] For example, in order to support the convenient execution of the user's foot pedaling action, Figure 2 As shown, the target parts in the present application may be the toes and heels of the foot.

[0046] It is understandable that the user's foot can be a single foot or two feet, and this application does not limit this. Therefore, for a single foot or two feet, each single foot can include at least two target parts.

[0047] Then, when the user's foot performs the corresponding pedaling action, the present application can use the corresponding data acquisition equipment to obtain various pedaling motion data of the foot in real time. Among them, the pedaling motion data may include but is not limited to at least one of the information of the movement position, direction posture, acceleration, angular velocity, etc. of the user's foot when performing various pedaling actions.

[0048] Then, for each target part of the user's foot, by performing corresponding motion analysis on various pedaling motion data of the foot, it is possible to judge whether each target part is in contact with the ground or has left the ground, thereby determining the pedaling state of each target part.

[0049] It can be seen that the stepping state of each target part can include two states: a stepping state and a lifting state. Among them, the distinguishing criterion between the stepping state and the lifting state can usually be the height of the target part of the foot from the ground. If the height of a certain target part from the ground is less than a preset threshold (for example, 5 cm), it means that the target part is approximately in contact with the ground, and it can be determined that the target part is in the stepping state. If the height of a certain target part from the ground is greater than or equal to a preset threshold (for example, 5 cm), it means that the target part is approximately far away from the ground, and it can be determined that the target part is in the lifting state.

[0050] S120, determining a trampling event of each target part according to a trampling state change of the target part.

[0051] Since the pedaling action performed by the user's foot is mainly the movement of the user's foot stepping on and off the ground, that is, the pedaling state of the user's foot is constantly changing. Then, in order to accurately identify each pedaling action performed by the user's foot, the present application can characterize the change of the foot pedaling state through a pedaling event.

[0052] Therefore, after determining the pedaling states of at least two target parts of the user's foot, the present application can determine the pedaling event represented by each target part when the pedaling state changes by analyzing the change of the pedaling state of the target part. Thus, in the same manner as above, the pedaling event of each target part can be determined.

[0053] As an optional implementation scheme in the present application, since the stepping state of each target part of the user's foot may include a stepping state and a lifting state, then Figure 3 As shown in FIG. 1 , for each target part of the foot, the stepping event of the target part can be divided into the following two cases:

[0054] Case 1: If the target part changes from the ground-pressing state to the lifted state, a lift event of the target part is generated.

[0055] That is to say, during the process of the user performing a corresponding stepping action through a certain target part of the foot, if the target part changes from the stepping state to the lifting state, such as Figure 3 The state "1" represents the stepping on the ground, which is changed to the state "0" represents the lifting up state, which indicates that the target part leaves the ground at the current transformation moment, thereby generating a lifting event of the target part.

[0056] Case 2: If the target part changes from the lifted state to the stepped-on state, a stepped-on event of the target part is generated.

[0057] That is to say, during the process of the user performing a corresponding stepping action through a certain target part of the foot, if the target part changes from a lifted state to a stepped state, such as Figure 3 The change from the lifted state represented by state "0" to the stepped state represented by state "1" indicates that the target part touches the ground at the current transformation moment, thereby generating a stepped event of the target part.

[0058] According to the above content, after determining the stepping events of at least two target parts of the user's foot, the present application can use the stepping events of any one or more of the target parts as the triggering condition of a certain interactive operation in the virtual reality environment. Then, after detecting the stepping events of one or more target parts, the corresponding interactive operation can be performed in the virtual reality environment.

[0059] The technical solution provided by the embodiment of the present application can determine the trampling state of each target part of at least two target parts of the contactable parts between the user's foot and the ground when the user lands on the ground. Then, according to the trampling state change of each target part, the trampling event of the target part is determined, thereby achieving accurate recognition of the trampling events of at least two target parts of the foot, supporting accurate recognition of various trampling actions performed by the foot through different target parts, and ensuring the diversity of foot trampling recognition.

[0060] In the present application, in order to ensure comprehensive identification of foot stepping, after determining the stepping events of at least two target parts of the foot, the present application can also determine the overall stepping event of the foot based on the stepping events of at least two target parts of the foot.

[0061] That is, after determining the trampling event of each target part of the foot, since the trampling event of each target part of the foot can represent a trampling event caused by the foot performing a trampling action. Therefore, in order to ensure the comprehensive identification of the user's foot trampling event, the present application can comprehensively analyze the trampling events of at least two target parts of the foot to obtain a complete trampling event as the overall trampling event of the foot in the present application.

[0062] In some possible implementations, in order to avoid repetition and redundancy of the overall pedaling event of the foot, the present application can adopt the following steps to determine the overall pedaling event of the foot: merge the pedaling events of each target part of the foot to obtain a pedaling event combination of the foot; filter the pedaling event combination according to the preset pedaling logic of the foot to obtain the overall pedaling event of the foot.

[0063] After determining the trampling events of each target part, the present application can combine the trampling events of each target part according to the execution time sequence of the trampling events of each target part to obtain a trampling event combination of the foot.

[0064] Since there are different recognition requirements for foot stepping tasks in different foot stepping application scenarios, the present application can pre-set a suitable foot stepping logic in each foot stepping application scenario according to the different foot stepping recognition requirements in different foot stepping application scenarios, as the preset foot stepping logic in the present application.

[0065] Therefore, for the foot stepping event combination, the present application can adopt a preset stepping logic suitable for the current foot stepping application scenario, filter out some stepping events in the foot stepping event combination that do not conform to the preset stepping logic, and delete the filtered some stepping events from the stepping event combination to obtain the overall foot stepping event.

[0066] For example, at least two target parts of the foot are the toes and the heels, such as Figure 4 As shown, the present application can determine the toe stepping state and the heel stepping state respectively, thereby determining the toe stepping event and the heel stepping event. Figure 4 The upward arrow in the middle can represent a lift event, and the downward arrow can represent a stepping event.

[0067] Then, according to the execution time sequence of each toe stamping event and each heel stamping event, the present application can merge each toe stamping event and each heel stamping event together to obtain a foot stamping event combination.

[0068] Assume that the preset pedaling logic of the foot is "the pedaling events and the lifting events of the foot appear alternately in sequence" and "the time interval between adjacent pedaling events is not less than the preset interval threshold (for example, 1s)". Then, according to the preset pedaling logic of "the pedaling events and the lifting events of the foot appear alternately in sequence", in the pedaling event combination of the foot, for the same pedaling event (such as the pedaling event or the lifting event) that appears continuously, this application can only retain the first pedaling event that appears, and delete the subsequent repeated pedaling events that appear continuously, for example Figure 4 The stamped events and lifted events marked in the stampede event combination are obtained to obtain the stampede event combination after preliminary filtering.

[0069] Then, in the stampede event combination after preliminary filtering, according to the preset stampede logic that "the time interval between adjacent stampede events is not less than the preset interval threshold (for example, 1s)", the present application can continue to delete some stampede events whose interval time with the previous stampede event is less than the preset interval threshold, for example Figure 4 The stepping events and lifting events marked in the stepping event combination after preliminary filtering are used to obtain the overall stepping event of the foot.

[0070] According to the above content, after determining the overall stepping event of the user's foot, the present application can use the overall stepping event of the foot as a trigger condition for a certain interactive operation in the virtual reality environment. Then, after detecting the overall stepping event of the foot, the corresponding interactive operation can be performed in the virtual reality environment.

[0071] The technical solution provided in the embodiment of the present application determines the overall stepping event of the foot through a comprehensive analysis of the stepping events of at least two target parts of the foot, thereby achieving accurate identification of the foot stepping event; and supports accurate identification of various stepping actions performed by the foot through different target parts by comprehensively referring to the stepping events of multiple target parts of the foot, thereby ensuring the comprehensiveness of the foot stepping identification.

[0072] As an optional implementation scheme in the present application, in order to ensure accurate identification of the pedaling state of each target part of the user's foot, the present application can pre-train a pedaling recognition model, which can be used to accurately predict the pedaling state of each target part of the foot.

[0073] Then, for the pedaling state of each target part of the foot, the present application can input the pedaling motion data of the user's foot into a pre-built pedaling recognition model, and output the pedaling states of at least two target parts of the foot.

[0074] In the present application, when the user's foot performs various pedaling actions, the present application can use corresponding data acquisition equipment to obtain various pedaling motion data of the foot in real time. The pedaling motion data may include but is not limited to at least one of the information of the movement position, direction, posture, acceleration, angular velocity, etc. of the user's foot when performing various pedaling actions.

[0075] In some possible implementations, the present application may pre-build a stepping recognition system to obtain the stepping motion data of the foot through the stepping recognition system.

[0076] like Figure 5 As shown, the stepping recognition system in the present application may include an electronic device 510 and at least one inertial sensor 520 communicatively connected to the electronic device 510 .

[0077] Among them, the electronic device 510 can be an XR device, specifically including an AR device, a VR device, an MR device, etc., so as to present a corresponding virtual reality environment to the user.

[0078] Specifically, the number of inertial sensors 520 may be equal to the number of the user's feet, and each inertial sensor 520 may be worn at a relevant part of the user's foot. Since the foot can mainly be represented by the movement of the center of the sole when performing various pedaling actions, it can also drive the corresponding calf of the foot to perform the same movement. Therefore, the relevant part of the foot can be the center of the sole of the foot, or the corresponding calf of the foot.

[0079] Then, each inertial sensor 520 can collect inertial data when the worn foot performs a specified stepping action in real time. The inertial data may include, but is not limited to, relevant direction posture data and angular velocity information of the foot when performing various stepping actions. Then, the stepping motion data of the user's foot may include the inertial data collected by each inertial sensor 520 worn on the associated parts of each foot. Then, the stepping motion data is transmitted to the electronic device 510, so that the electronic device 510 can obtain the stepping motion data of each foot. Then, the electronic device 510 can determine the stepping state of at least two target parts of the foot according to the stepping motion data of the foot.

[0080] Among them, as for the number of inertial sensors, assuming that this application mainly focuses on the stepping events of one of the left and right feet, then an inertial sensor can be configured and worn on the related parts of the foot that is focused on this time, so as to collect inertial data when the foot performs various stepping actions as the stepping motion data of the foot.

[0081] Assuming that the present application mainly focuses on the pedaling events of both feet consisting of the left and right feet, two inertial sensors can be configured and worn on the corresponding parts of the left and right feet respectively, so that the inertial data of the left and right feet when performing various pedaling actions can be collected respectively by the two inertial sensors as the pedaling motion data of the left and right feet.

[0082] It should be noted that, in addition to wearing the inertial sensor 520, the user's foot-related parts may also wear electronic components with certain computing capabilities to independently analyze the trampling events of multiple target parts of the user's foot and / or the overall trampling events of the foot. It can be seen that the electronic device 510 and the inertial sensor 520 in this application may not be a one-to-many relationship, but a one-to-one relationship.

[0083] Moreover, for the stepping recognition model in the present application, the stepping recognition model may include a skeleton network (backbone network) and multiple head networks (head networks), and the total number of head networks is equal to the total number of target parts of the feet, and the target parts corresponding to each head network are different.

[0084] On the other hand, the present application can also set at least two stepping recognition models, each stepping recognition model can include a skeleton network with the same structure and at least one head network, and the total number of head networks in each stepping recognition model is equal to the total number of target parts of the foot, and each head network corresponds to a different target part.

[0085] In the above two pedaling recognition models, the skeleton network can be composed of multiple basic blocks (basic_block) with the same structure, which are used to predict the time series fusion features of the pedaling motion data of the user's feet, and the time series fusion features are respectively input into each head network connected to the skeleton network, so that each head network in each pedaling recognition model can obtain the time series fusion features of the pedaling motion data of the user's feet. And each head network can be used to predict the pedaling state of a target part of the foot according to the time series fusion features.

[0086] That is, by setting the sum of the number of head networks in each stepping recognition model to be equal to the sum of the number of target parts of the foot, all the head networks in each stepping recognition model can be matched one-to-one with the target parts of the foot, so that one head network can be used to predict the stepping state of one of the target parts. Then, through all the head networks in each stepping recognition model, the stepping state of each target part of the foot can be predicted.

[0087] Moreover, each basic block in the backbone network in each stepping recognition model may include fully connected layers (FC), a normalization layer (Layer Normalization, LayerNorm) and a recurrent neural network (RNN) connected in sequence. The head network in each stepping recognition model may be a multilayer perceptron (MLP).

[0088] Taking the foot in this application as a pair of feet consisting of a left and a right foot, and the target parts of the foot as the toes and the heels, this application ultimately needs to output four pieces of information through the stepping recognition model: the stepping state of the left toe, the stepping state of the left heel, the stepping state of the right toe, and the stepping state of the right heel. Therefore, the total number of head networks in each stepping recognition model can be 4.

[0089] Therefore, depending on the number of stepping recognition models set, the number of head networks in each stepping recognition model constructed in this application may be different. Figure 6a , Figure 6b, Figure 6c and Figure 6d They can be four different exemplary model structure diagrams of the stepping recognition model in the present application.

[0090] In the present application, after obtaining the pedaling motion data of the user's foot, the pedaling motion data can be directly input into the above-constructed pedaling recognition model. Then, the pedaling motion data of the foot is processed with corresponding features in turn through multiple basic blocks (basic_block) with the same structure in the skeleton network (backbone network) of the pedaling recognition model, so as to obtain corresponding time series fusion features. Then, each head network (head network) in the pedaling recognition model continues to perform corresponding feature mapping processing on the time series fusion features to determine the pedaling state of each target part of the foot.

[0091] When there are at least two pedaling recognition models, the present application can input the pedaling motion data of the user's feet into at least two pre-built pedaling recognition models respectively, and output the pedaling status of at least two target parts of the foot. Thus, for the pedaling motion data input into each pedaling recognition model, the pedaling motion data of the foot can be sequentially subjected to corresponding feature fusion processing through multiple basic blocks (basic_block) with the same structure in the skeleton network (backbone network) of each pedaling recognition model, thereby obtaining corresponding time series fusion features. Then, each head network (head network) in each pedaling recognition model continues to perform corresponding feature mapping processing on the time series fusion features to determine the pedaling status of each target part of the foot.

[0092] In some implementable methods, in order to ensure the accuracy of identifying the pedaling state of each target part of the foot, the present application outputs the pedaling state of each target part of the foot through a pedaling recognition model, which can be specifically: preprocessing the pedaling motion data of the user's foot to obtain a pedaling feature vector of the foot; inputting the pedaling feature vector into a pre-built pedaling recognition model, and outputting the pedaling state of at least two target parts of the foot.

[0093] That is to say, after obtaining the pedaling motion data of the user's foot, since the pedaling motion data has multiple structures, it is not convenient for the pedaling recognition model to process efficiently. Therefore, the present application can pre-process the pedaling motion data of the user's foot and transform it into a one-dimensional feature vector as the pedaling feature vector of the foot in the present application.

[0094] For example, when the user's foot performs various pedaling actions, the present application can collect the user's foot pedaling motion data in real time, indicating that the user's foot pedaling motion data is a set of time series data, which can be expressed as: X = {x1, x2, L, x T}, indicating that it contains T frames of inertial data collected by each inertial sensor worn on the foot. Among them, the pedaling motion data x of each frame t It may include but is not limited to: the rotation matrix corresponding to the direction and posture information collected by each inertial sensor worn on the user's foot, and the angular velocity information.

[0095] Then, for each frame of pedaling motion data x t , the present application can convert the rotation matrix corresponding to the direction and attitude information collected by each inertial sensor into a corresponding one-dimensional vector. Moreover, using the pedaling motion data x of the previous frame t-1 The inverse of the rotation matrix in is multiplied by the pedaling motion data x of the frame t The corresponding angular velocity information can be obtained by using the rotation matrix in , and converted into another one-dimensional vector. Then, the above two one-dimensional vectors and the angular velocity information collected by each inertial sensor are combined to obtain the pedaling motion data x of the frame. t The corresponding trampling feature vector f t .

[0096] After determining the stepping feature vector of the foot, the present application can input the stepping feature vector into the pre-built stepping recognition model. Then, the stepping feature vector is processed accordingly through the skeleton network and each head network in the stepping recognition model to predict the stepping state of each target part of the foot.

[0097] In one or more embodiments of the present application, in order to ensure that the pedaling recognition model accurately recognizes the pedaling state of each target part of the foot, the present application needs to specify a large number of user feet performing various pedaling actions in advance to obtain a large amount of training data. Then, a corresponding loss function can be set in the pedaling recognition model to calculate the loss between the predicted value of the pedaling state of each target part of the user's foot after a large amount of training data is input and the true value of the pedaling state represented by the sample label, so as to continuously update the pedaling recognition model until the loss of the pedaling recognition model reaches convergence.

[0098] Among them, the loss function in the stepping recognition model can be a binary cross entropy loss (Binary CrossEntropy Loss, referred to as BCE loss).

[0099] As for the training data of the stepping recognition model, the present application can pre-build a training data acquisition system to obtain the corresponding training data through the training data acquisition system.

[0100] like Figure 7As shown, the training data collection system may include a computer device 710 , at least one inertial sensor 720 communicatively connected to the computer device 710 , at least two rigid bodies 730 of different shapes, and an optical motion capture device 740 .

[0101] Specifically, the number of inertial sensors 720 is equal to the number of user's feet, and each inertial sensor 720 can be worn at a corresponding part of the user's foot. Then, each inertial sensor 720 can be used to collect inertial data when the foot performs a specified stepping action, and transmit the inertial data to the computer device 710 as a corresponding training sample.

[0102] The number of rigid bodies 730 is equal to the total number of target parts of the user's feet, and each rigid body 730 is fixed one-to-one at each target part of the user's feet, so that rigid bodies of different shapes are used to intuitively represent different target parts, so that the optical motion capture device 740 can easily distinguish the actual stepping state of each target part by capturing rigid body images of different shapes of each target part.

[0103] The optical motion capture device 740 may include a surround camera array to capture rigid body images of different shapes worn by each target part of the user's foot when performing various pedaling actions from all angles, so as to easily distinguish the true pedaling state of each target part. Then, the optical motion capture device 740 can obtain the rigid body image of each target part of the foot when the foot performs a specified pedaling action through the surround camera array, thereby determining the true value of the pedaling state of the target part, and transmit the true value of the pedaling state of each target part as the corresponding sample label to the computer device 710.

[0104] It is understandable that the computer device 710 in the training data acquisition system and the electronic device 510 in the stepping recognition system can be the same device or different devices, such as a head-mounted display device, and this application does not limit this.

[0105] Therefore, during the training data acquisition stage, according to the stepping events of different feet that the stepping recognition model terminal focuses on, a large number of users can be required to fix rigid bodies 730 of different shapes on each target part of the foot that is of focus, and wear corresponding inertial sensors 720 at the associated parts of the foot that are of focus.

[0106] Then, when the user's foot performs various designated pedaling movements, each inertial sensor 720 worn at the associated part of the user's foot can collect inertial data of the worn foot when performing the designated pedaling movement in real time. The inertial data may include but is not limited to relevant direction posture data and angular velocity information of the foot when performing various pedaling movements. Then, the present application can use the inertial data collected by each inertial sensor 720 as the historical pedaling motion data of the corresponding foot on which the inertial sensor 720 is worn, and as the corresponding training sample. Then, the above training sample is transmitted to the computer device 710 together with the timestamp represented by the collection time of the training sample.

[0107] Moreover, at the same time point, the optical motion capture device 740 can use a surround camera array to shoot a rigid body of different shapes fixed to each target part of the foot when each inertial sensor 720 is worn on the corresponding foot and perform a specified stepping action, so as to obtain a rigid body image corresponding to each target part. Then, the optical motion capture device 740 determines the height of the rigid body in the rigid body image from the ground by performing corresponding feature positioning on the rigid body image of each target part, so as to judge whether each target part is in contact with the ground, thereby obtaining the true value of the stepping state of each target part. Furthermore, the true value of the stepping state of each target part can be used as the corresponding sample label and transmitted to the computer device 710 together with the timestamp represented by the shooting time of the rigid body image.

[0108] Then, the computer device 710 can obtain a large number of training samples and sample labels. By combining the training samples and sample labels under the same timestamp, the corresponding training data can be obtained, so as to accurately train the stepping recognition model in this application.

[0109] Figure 8 This is a principle block diagram of a foot stepping recognition device provided in an embodiment of the present application. Figure 8 As shown, the foot stepping recognition device 800 may include:

[0110] A pedaling state determination module 810 is used to determine the pedaling state of at least two target parts of the user's foot according to the pedaling motion data of the user's foot, wherein the target parts are parts that can contact the ground when the foot lands;

[0111] The foot stepping determination module 820 is used to determine the stepping event of each target part according to the stepping state change of the target part.

[0112] In some implementations, the pedaling state determination module 810 may be specifically used to:

[0113] The pedaling motion data of the user's foot is input into a pre-built pedaling recognition model, and the pedaling states of at least two target parts of the foot are output.

[0114] In some implementations, the pedaling state determination module 810 may be specifically used to:

[0115] Preprocessing the pedaling motion data of the user's foot to obtain a pedaling feature vector of the foot;

[0116] The stepping feature vector is input into a pre-built stepping recognition model, and the stepping status of at least two target parts of the foot is output.

[0117] In some possible implementations, the stepping recognition model includes a skeleton network and multiple head networks, and the total number of the head networks is equal to the total number of the target parts of the foot, and the target parts corresponding to each of the head networks are different.

[0118] In some possible implementations, there are at least two stepping recognition models, each of which includes a skeleton network and at least one head network of the same structure, and the total number of head networks in each stepping recognition model is equal to the total number of target parts of the foot, and each head network corresponds to a different target part;

[0119] Accordingly, the pedaling state determination module 810 may be specifically used for:

[0120] The pedaling motion data of the user's foot are respectively input into at least two pre-built pedaling recognition models, and the pedaling states of at least two target parts of the foot are output.

[0121] In some possible implementations, the training data of the stepping recognition model is obtained through a pre-built training data acquisition system; the training data acquisition system includes a computer device, at least one inertial sensor communicatively connected to the computer device, at least two rigid bodies of different shapes, and an optical motion capture device; wherein,

[0122] The number of the inertial sensors is equal to the number of the feet, and the inertial sensors are worn at associated parts of the feet to collect inertial data when the feet perform a specified stepping action, and transmit the inertial data as corresponding training samples to the computer device;

[0123] The number of the rigid bodies is equal to the sum of the number of the target parts of the foot, and each rigid body is fixed at each target part of the foot in a one-to-one correspondence;

[0124] The optical motion capture device includes a surround camera array, which is used to obtain a rigid body image of each target part of the foot when the foot performs a specified stepping action, so as to determine the true value of the stepping state of the target part, and transmit the true value of the stepping state of each target part as a corresponding sample label to the computer device;

[0125] The computer device combines the training samples and sample labels at the same timestamp to obtain corresponding training data.

[0126] In some possible implementations, the pedaling motion data of the foot is obtained through a pre-built pedaling recognition system; the pedaling recognition system includes an electronic device and at least one inertial sensor that is communicatively connected to the electronic device; wherein,

[0127] The number of the inertial sensors is equal to the number of the feet, and the inertial sensors are worn at the relevant parts of the feet, and are used to collect inertial data when the feet perform a specified stepping action, and transmit the inertial data as the stepping motion data of the feet to the electronic device;

[0128] The electronic device is used to determine the pedaling status of at least two target parts of the foot according to the pedaling motion data of the foot.

[0129] In some implementations, the stepping state includes a stepping state and a lifting state. The foot stepping determination module 820 may be specifically used to:

[0130] For each target part of the foot, if the target part changes from a ground-pressing state to a lifted state, a lift event of the target part is generated;

[0131] If the target part changes from the lifted state to the stepped-on state, a stepped-on event of the target part is generated.

[0132] In some possible implementations, the foot stepping recognition device 800 may further include:

[0133] The whole foot stepping identification module is used to determine the whole foot stepping event according to the stepping events of at least two target parts of the foot.

[0134] In some implementations, the whole foot stepping recognition module can be specifically used for:

[0135] Merging the trampling events of each target part of the foot to obtain a trampling event combination of the foot;

[0136] According to the preset stepping logic of the foot, the stepping event combination is filtered to obtain the overall stepping event of the foot.

[0137] In the embodiment of the present application, for at least two target parts of the contactable parts between the user's foot and the ground when the user lands on the ground, the trampling state of each target part can be determined. Then, according to the trampling state change of each target part, the trampling event of the target part is determined, thereby achieving accurate recognition of the trampling events of at least two target parts of the foot, supporting accurate recognition of various trampling actions performed by the foot through different target parts, and ensuring the diversity of foot trampling recognition.

[0138] It should be understood that the device embodiment and the method embodiment in the present application may correspond to each other, and similar descriptions may refer to the method embodiment in the present application. To avoid repetition, they will not be described here.

[0139] Specifically, Figure 8 The device 800 shown can execute any method embodiment provided in this application, and Figure 8 The aforementioned and other operations and / or functions of each module in the illustrated device 800 are respectively for implementing the corresponding processes of the aforementioned method embodiment, and for the sake of brevity, they are not described in detail herein.

[0140] The above method embodiment of the embodiment of the present application is described from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0141] Fig. 9 A schematic block diagram of an electronic device provided in an embodiment of the present application.

[0142] like Fig. 9 As shown, the electronic device 900 may include:

[0143] The memory 910 and the processor 920, the memory 910 is used to store the computer program and transmit the program code to the processor 920. In other words, the processor 920 can call and run the computer program from the memory 910 to implement the method in the embodiment of the present application.

[0144] For example, the processor 920 may be configured to execute the above method embodiments according to instructions in the computer program.

[0145] In some embodiments of the present application, the processor 920 may include but is not limited to:

[0146] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0147] In some embodiments of the present application, the memory 910 includes but is not limited to:

[0148] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0149] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 910 and executed by the processor 920 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 900.

[0150] like Fig. 9 As shown, the electronic device may also include:

[0151] The transceiver 930 may be connected to the processor 920 or the memory 910 .

[0152] The processor 920 may control the transceiver 930 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 930 may include a transmitter and a receiver. The transceiver 930 may further include an antenna, and the number of antennas may be one or more.

[0153] It should be understood that the various components in the electronic device 900 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0154] The present application also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to execute the method of the above method embodiment.

[0155] The embodiment of the present application also provides a computer program product including a computer program / instruction, which, when executed by a computer, enables the computer to execute the method of the above method embodiment.

[0156] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0157] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for identifying foot stepping, characterized in that: include: Determining the pedaling status of at least two target parts of the user's foot according to the pedaling motion data of the user's foot, wherein the target parts are parts that can be contacted with the ground when the foot lands; According to the change of the trampling state of each target part, the trampling event of the target part is determined.

2. The method according to claim 1, characterized in that The step of determining the pedaling states of at least two target parts of the user's foot according to the pedaling motion data of the user's foot comprises: The pedaling motion data of the user's foot is input into a pre-built pedaling recognition model, and the pedaling states of at least two target parts of the foot are output.

3. The method according to claim 2, characterized in that The step of inputting the pedaling motion data of the user's foot into a pre-built pedaling recognition model and outputting the pedaling states of at least two target parts of the foot comprises: Preprocessing the pedaling motion data of the user's foot to obtain a pedaling feature vector of the foot; The stepping feature vector is input into a pre-built stepping recognition model, and the stepping status of at least two target parts of the foot is output.

4. The method according to claim 2, characterized in that: The stepping recognition model includes a skeleton network and multiple head networks, and the total number of the head networks is equal to the total number of the target parts of the feet, and the target parts corresponding to each of the head networks are different.

5. The method according to claim 2, characterized in that: There are at least two stepping recognition models, each of which includes a skeleton network and at least one head network of the same structure, and the total number of head networks in each stepping recognition model is equal to the total number of target parts of the foot, and each head network corresponds to a different target part; Accordingly, the step of inputting the pedaling motion data of the user's foot into the pre-built pedaling recognition model and outputting the pedaling states of at least two target parts of the foot includes: The pedaling motion data of the user's foot are respectively input into at least two pre-built pedaling recognition models, and the pedaling states of at least two target parts of the foot are output.

6. The method according to claim 2, characterized in that The training data of the stepping recognition model is obtained through a pre-built training data acquisition system; the training data acquisition system includes a computer device, at least one inertial sensor connected to the computer device for communication, at least two rigid bodies of different shapes and an optical motion capture device; wherein, The number of the inertial sensors is equal to the number of the feet, and the inertial sensors are worn at associated parts of the feet to collect inertial data when the feet perform a specified stepping action, and transmit the inertial data as corresponding training samples to the computer device; The number of the rigid bodies is equal to the sum of the number of the target parts of the foot, and each rigid body is fixed at each target part of the foot in a one-to-one correspondence; The optical motion capture device includes a surround camera array, which is used to obtain a rigid body image of each target part of the foot when the foot performs a specified stepping action, so as to determine the true value of the stepping state of the target part, and transmit the true value of the stepping state of each target part as a corresponding sample label to the computer device; The computer device combines the training samples and sample labels at the same timestamp to obtain corresponding training data.

7. The method according to claim 2, characterized in that The pedaling motion data of the foot is obtained through a pre-built pedaling recognition system; the pedaling recognition system includes an electronic device and at least one inertial sensor that is communicatively connected to the electronic device; wherein, The number of the inertial sensors is equal to the number of the feet, and the inertial sensors are worn at the relevant parts of the feet, and are used to collect inertial data when the feet perform a specified stepping action, and transmit the inertial data as the stepping motion data of the feet to the electronic device; The electronic device is used to determine the pedaling status of at least two target parts of the foot according to the pedaling motion data of the foot.

8. The method according to claim 1, characterized in that The stepping state includes a stepping state and a lifting state. The stepping event of each target part is determined according to the stepping state change of the target part, including: For each target part of the foot, if the target part changes from a ground-pressing state to a lifted state, a lift event of the target part is generated; If the target part changes from the lifted state to the stepped-on state, a stepped-on event of the target part is generated.

9. The method according to claim 1, characterized in that: The method further comprises: The overall stepping event of the foot is determined according to the stepping events of at least two target parts of the foot.

10. The method according to claim 9, characterized in that The step of determining the overall stepping event of the foot according to the stepping events of at least two target parts of the foot comprises: Merging the trampling events of each target part of the foot to obtain a trampling event combination of the foot; According to the preset stepping logic of the foot, the stepping event combination is filtered to obtain the overall stepping event of the foot.

11. A foot stepping recognition device, characterized in that: include: A pedaling state determination module, used to determine the pedaling state of at least two target parts of the user's foot according to the pedaling motion data of the user's foot, wherein the target parts are parts that can be contacted with the ground when the foot lands; The foot stepping determination module is used to determine the stepping event of each target part according to the stepping state change of the target part.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the foot stepping recognition method described in any one of claims 1-10 by executing the executable instructions.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the foot stepping recognition method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions contained in the computer program product are run on an electronic device, the electronic device executes the foot stepping recognition method according to any one of claims 1 to 10.

Images