Computer-implemented method for selecting a category of shoe

Abstract

The invention relates to a computer-implemented method and a data processing system for selecting a category of shoe for a wearer. The method comprises the steps of: obtaining (181) a video (V) of the wearer (1), wherein in the video (V) the wearer (1) walks several steps forwards facing towards the recording camera, or away therefrom facing away from it; determining (182) key points (31–37) of the skeleton of the wearer (1) and their movement by analysing the video (V), wherein the key points comprise key points (31–37) at the head, the femoral head, the knee and the ankle joint; determining (183) a stance phase (4) of at least one foot from at least some of the key points (31–37); determining (184) a knee movement pattern (K) during the stance phase (4) from at least some of the key points (31-37), which pattern indicates the medio-lateral change in position of a knee key point (33, 34) in the frontal plane; and using (185) the determined knee movement pattern (K) to select a category of shoe from a plurality of predefined categories of shoe, wherein the knee movement pattern (K) is used directly or indirectly as a parameter in a predefined formula for determining the category of shoe.

Description

[0001] COMPUTER-IMPLEMENTED PROCESS FOR SELECTING A SHOE CATEGORY

[0002] Description

[0003] The invention relates to a computer-implemented method for selecting a shoe category.

[0004] It's challenging for a person to find the perfect shoe, including the perfect insole or insole. Both buying the right shoe and buying an insole require a lot of consultation. Necessary information (e.g., information about foot type) is unknown to most people. Accordingly, potential users primarily seek advice in brick-and-mortar stores.

[0005] Due to the increasing digitalization in the industry and the rapidly growing relevance of e-commerce, there is a need to be able to provide advice and a selection of shoes to both retailers and end users in an effective way online.

[0006] The present invention is therefore based on the object of providing a computer-implemented method, a computer program and a data processing system that effectively enable the selection of a shoe in the context of an eCommerce purchase.

[0007] This object is achieved according to the invention by a computer-implemented method having the features of claim 1, a computer program having the features of claim 21, and a data processing system having the features of claim 22. Embodiments of the invention are specified in the dependent claims. A first aspect of the invention provides a computer-implemented method for selecting a shoe category for a user. The method comprises the steps:

[0008] - Receiving a video of the user, where the video is received via a communication connection. In the video, the user walks several steps toward the recording camera or away from it. The video is transmitted via the communication connection, for example, from a communication device assigned to the user.

[0009] - Determining key points of the user's skeleton and their movement by evaluating the video, whereby the key points include key points on the head, femoral head, knee and ankle.

[0010] - Determining a stance phase of at least one foot from at least some of the key points, wherein the stance phase is defined as the period between initial contact of the foot with the ground and lifting of the foot from the ground.

[0011] - Determining a knee movement pattern during the stance phase from at least some of the key points. The knee movement pattern is defined as the medio-lateral position change of the knee key point in the frontal plane. Specifically, the extent of medio-lateral movement of a knee key point (e.g., the patella or the center of the patella) between initial foot contact and the moment of maximum foot loading is determined. Optionally, the direction of the medio-lateral movement (inward or outward) can also be recorded.

[0012] - Using the determined knee movement pattern to select a shoe category from a plurality of predefined shoe categories, whereby the knee movement pattern is used directly or indirectly as a parameter in a predefined formula for determining the shoe category.

[0013] The invention is thus based on the idea of ​​recording and evaluating the user's medio-lateral knee movement via a video provided by the user and incorporating this parameter into a predefined formula for determining a suitable shoe category for the user. The predefined formula for determining a shoe category can include additional parameters, as will be explained below. The formula can be heuristic and / or empirically based. The inventive solution performs an analysis based on key points of the skeleton, which are determined, for example, via a trained neural network or an algorithm. The determined key points are used to determine a stance phase of the user's foot.Determining the stance phase is important because the user's skeleton, especially the ankle, knee, and hip, are subjected to stress during the stance phase and experience force and deformation. The stance phase is determined separately for each foot, although determining the stance phase for just one foot is generally sufficient.

[0014] The solution according to the invention further provides for determining the knee movement pattern within the stance phase, i.e. determining the movement of the knee between the initial contact of the foot and a suitable later point in time during the stance phase, for example the moment of maximum foot load.

[0015] A medio-lateral movement in the sense of the present invention is a movement on the frontal plane which is alternately directed away from the median plane (lateral) or opposite to the median plane (medial), whereby the median describes the center of the body.

[0016] The inventive solution is based on the finding that the knee movement pattern provides significant information about the stability of the locomotor system, i.e. the musculoskeletal system that drives the body, so that this parameter is essential for selecting a shoe category. The different shoe categories differ in particular in the medio-lateral stability that the shoe provides to the user or runner. The shoe categories represent stability classes. The medio-lateral stability of the shoe can be adjusted, for example, via parameters such as the material of the midsole, the geometry of the midsole, the shoe construction, the upper material of the shoe shaft, the cushioning properties, heel reinforcement, or the like. Such parameters can be used to divide the shoes into shoe categories or stability classes.

[0017] The shoe category is, for example, a running shoe category, without the present invention being limited thereto. One embodiment of the invention provides that the beginning and end of the stance phase are determined by evaluating the movement of at least one key point on the head, whereby the vertical movement of the key point is evaluated. Since the head performs a characteristic periodic vertical movement when running, the beginning and end of the stance phase, i.e., the time of initial contact (so-called IC - "Initial Contact") and the time at which the foot lifts off the ground again (so-called "Toe Off"), can be detected with the aid of the head height. Alternatively or additionally, the movement of a key point on the ankle joint can be evaluated to determine the beginning and end of the stance phase.

[0018] One variant involves evaluating the knee movement pattern by comparing skeletal points at the time of initial contact and at the time of maximum foot loading during the stance phase. At the time of maximum foot loading, the force acting on the skeleton and muscles is also at its maximum. Studies show that within the stance phase (i.e., between initial contact and foot liftoff), maximum loading occurs after approximately 40% of the time within this time interval has elapsed. The knee movement pattern is thus determined, for example, by comparing key skeletal points between initial contact and the 40% time within the stance phase.

[0019] In one embodiment, the knee movement pattern is determined by evaluating the dynamic leg axis, whereby the dynamic leg axis comprises the imaginary line between the ankle and knee, the imaginary line between the knee and hip joint, and the dynamic angle between these two lines. The imaginary lines between the ankle and knee, and between the knee and hip joint, can be determined using the key points. This allows the angle between these two imaginary lines to be easily determined. This angle is "dynamic" in that it changes depending on the vertical force acting on the knee joint. The knee movement pattern can, in turn, be determined from the change in the dynamic angle, for example, using conventional trigonometric calculations.The smallest angle and the largest angle are each assigned a specific height of the ankle-knee-hip triangle, with the height difference being equal to the knee movement pattern.

[0020] However, this is only an example. There are other ways to determine the knee movement pattern. Another embodiment of this involves determining the knee movement pattern by evaluating the medio-lateral movement of a key knee point (e.g., the position of the center of the kneecap) during the stance phase. In this way, the knee movement pattern can be determined directly as the maximum difference in the frontal plane of the key point at two different times during the stance phase (in particular, the difference between the position at initial contact and the position at maximum load (after the 40% time).

[0021] The inventive solution can evaluate a variety of skeleton keypoints from the video that are suitable for analysis. Advantageous embodiments provide for the following keypoints to be evaluated:

[0022] - as a key point associated with the head, the position of the nose: the nose key point is particularly suitable for determining the change in the vertical position of the head:

[0023] - as the key point associated with the femoral head, the center of the femoral head;

[0024] - as the key point associated with the knee, the center of the knee or the center of the kneecap; and

[0025] - as the key point associated with the ankle joint, the center of the ankle joint.

[0026] In one advantageous embodiment, such or other determination and recognition of skeletal keypoints is carried out using a trained neural network. Such trained networks are known and are available for commercial use. One example is the "BlazePose" convolutional neural network (CNN) from Google, LLC. BlazePose analyzes a person's posture based on a video displaying skeletal keypoints. Determining skeletal keypoints based on the video submitted by the user represents only the starting point for further calculations (determining the stride phase and the knee movement pattern, as well as using this information in a formula to determine a shoe category).

[0027] One embodiment of the invention provides that the determination of the stance phase and / or the knee movement pattern is also carried out using the trained neural network or another trained neural network, or alternatively, using an algorithm. When using a neural network to detect the stance phase, the times of initial contact and lift-off from the ground are trained in a training phase. The training of the neural network can be implemented as supervised learning, with the neural network learning from given pairs of inputs and outputs, and a "teacher" providing the correct function value for an input. The correct function value can be determined by analyzing the video images.To recognize the knee movement pattern, the neural network or another defined neural network can be trained in the same way using supervised learning, or an algorithm calculates the knee movement pattern from the comparison of the positions of knee key points at different times in the stance phase, for example at the time of initial contact and at the time the foot lifts off the ground.

[0028] The use of a neural network is essentially optional. In general, the specified analyses can be performed using a computer program or algorithm, for example, through traditional segmentation and pattern recognition in the individual video images.

[0029] In a further embodiment, the invention comprises the following further steps:

[0030] - Obtaining a photo of a foot of the user, wherein the photo is sent and then received via the communication connection by a communication device assigned to the user, wherein the photo represents an inner side view of one of the user's feet, - Determining a foot type of the user by evaluating the photo, wherein the specific foot type belongs to a number of predefined foot types that differ in the shape of the inner longitudinal arch of the foot, and

[0031] - Use of the specific foot type to determine the shoe category, whereby the foot type is directly or indirectly included as an additional parameter in the predefined formula for determining the shoe category.

[0032] According to this embodiment, a photo of the foot and a walking video are sent to a data processing system on which the computer-implemented method is executed, where they are evaluated. The photo is evaluated with regard to the foot type, which is determined by the development of the inner longitudinal arch of the foot. For example, between three and five different foot types can be defined, which differ in the arch of the foot. Examples of different arches are hollow foot, neutral foot, fallen arches, and flat foot. If no foot type can be identified during the evaluation of the photo (for example, due to poor photo quality or because the user accidentally sent the wrong photo), no assignment to a foot type is made and the user is informed accordingly.

[0033] The foot type is described by analyzing the photo. A further algorithm or another trained neural network can be used for this. Alternatively, classic image recognition can be performed, which recognizes shapes based on color and brightness gradients. If the photo is analyzed using a neural network, appropriate neural networks exist for this purpose. For example, the so-called COCO dataset from Microsoft (COCO = "Common Objects in Context") is well-known; it serves as a database for training neural networks to recognize human bodies.

[0034] A further embodiment of the invention provides that the provided video is returned to the user in an edited form, with the video additionally displaying the determined key points. A submitted photo can also be returned to the user after editing, for example, with isolines displayed on the foot. This provides feedback to the user, allowing them to review the quality of the submitted video or photo and, if necessary, send an improved video or photo to the data processing system.

[0035] A further embodiment of the invention provides that the predefined formula for determining the shoe category has as parameters:

[0036] - the knee movement pattern for one knee or the average of the knee movement patterns of both knees, and

[0037] - an insole profile, whereby the insole profile is determined by the combination of the specific foot type with a static leg axis type of the user.

[0038] A further parameter is therefore considered, which results from the combination of a specific foot type with a specific static leg axis type of the user. The static leg axis type results from the leg axis in a static state, i.e. when the user is standing motionless. Examples of leg axis types are the straight leg axis, the knock-kneed leg axis, and the bowleg leg axis. A user identifies their static leg axis type by comparing it with information provided on the communication unit in the user interface. In principle, however, it is alternatively possible to determine the static leg axis type by evaluating the video, whereby a video image is evaluated at the beginning of the analysis phase. Alternatively, a user can take another photo.

[0039] One embodiment of this approach provides for the insole profile determined by combining the foot type with the static leg axis type to be communicated separately to the user. This allows the user to identify and purchase a suitable insole profile.

[0040] A further embodiment provides that the predefined formula for determining the shoe category assigns points to the knee movement pattern parameter and the insole profile parameter, which are then added together. Depending on the total number of points, the shoe category is selected from the plurality of predefined shoe categories. This allows for a simple determination of the shoe category.

[0041] The parameters mentioned and the associated points can also be negative if necessary. For example, it can be provided that a large knee movement pattern with a knock-kneed leg axis type leads to the addition of a point value, and that a large knee movement pattern with a bow-kneed leg axis type leads to the subtraction of a point value. It can be provided that the knee movement pattern is recorded as a vector, i.e., the direction of the medio-lateral deflection is also recorded, and from this, an O-kneed leg axis or a knock-knee leg axis is inferred. Alternatively, the static leg axis type is entered by the user via a user interface.

[0042] The insole profiles are also assigned points, with a high profile (with a pronounced metatarsal bridge), for example, receiving a low score and a flat profile (with a less pronounced metatarsal bridge) receiving a high score.

[0043] Further refinements provide for the predefined formula for determining the shoe category to include additional parameters that are taken into account in the formula. Such additional parameters include, for example:

[0044] - information on the user's running profile provided by the user, which information includes at least whether the user is a beginner, an advanced runner or an experienced runner,

[0045] - information about the gender of the user provided by the user.

[0046] The predefined formula for determining the shoe category also assigns points to these additional parameters (running profile and gender), which are added to the points for the knee movement pattern parameter and the insole profile parameter. For example, a beginner may be assigned a comparatively high score, while an experienced runner may be assigned a comparatively low score or even a negative score. For example, a male user may be assigned little or no score. A female user, on the other hand, may be assigned a positive score. The differences in the scoring are due to different connective tissue structures in men and women. The total number of points then determines the selection of a specific shoe category from the majority of predefined shoe categories, with each shoe category assigned a different point range.

[0047] One embodiment provides for at least three, for example five, shoe categories to be predefined, from which one shoe category is selected in each case, whereby the shoe categories correspond to different stability classes of shoes. The number mentioned, for example five shoe categories, is only to be understood as an example and alternatively a different number of shoe categories can be predefined. In one embodiment, a lowest stability class or a first shoe category results from a low number of points. Higher stability classes or the further shoe categories result from correspondingly higher number of points. The fact that a shoe belongs to the lowest stability class means that the shoe offers the user the least stability, which is why the user must have sufficient stability on their own.The fact that a shoe belongs to the highest stability class means that the shoe itself provides the most stability to the wearer, making it suitable for a user with lower locomotor stability. The stability of the shoe can be influenced by parameters such as the midsole material, the midsole geometry, the shoe construction, the upper material of the shoe shaft, the cushioning properties, heel reinforcement, and the like.

[0048] A further embodiment provides that information about at least one shoe that corresponds to the specific shoe category is sent from the data processing system to the user. The user can use this information to make a purchase decision.

[0049] In a further aspect of the invention, the present invention relates to a computer program with program code for carrying out the method steps according to one of claims 1 to 20 when the computer program is executed in a computer.

[0050] A further aspect of the invention relates to a data processing system for selecting a shoe category for a user, comprising: - at least one processor;

[0051] - a memory operatively coupled to the at least one processor and in which program instructions are stored, wherein the at least one processor is configured to execute the program instructions, and wherein the program instructions comprise steps according to the features of claim 1.

[0052] The embodiments of claims 1 to 20 apply accordingly to the data processing system according to the invention.

[0053] Thus, the shoe category is determined from a plurality of predefined shoe categories using the knee movement pattern, whereby the knee movement pattern is used as a parameter in a predefined formula for determining the shoe category.

[0054] The data processing system comprises at least one processor for executing instructions and a memory coupled to the processor in which instructions are stored that, when executed by the processor, cause the processor to perform the specified steps. In other words, the data processing system is implemented by software in combination with a processor that executes the software. Multiple processors can interact. The data processing system can be implemented on an internet server or in the cloud.

[0055] The invention is explained in more detail below with reference to the figures of the drawing using several exemplary embodiments. They show:

[0056] Figure 1 schematically shows a communication infrastructure for carrying out a method according to the invention;

[0057] Figure 2 shows an example of a video image of a user showing the user from the front;

[0058] Figure 3 shows key points of the human skeleton;

[0059] Figure 4 shows the video image of Figure 2, in which key points of the skeleton are additionally shown; Figure 5 shows a schematic representation of the dynamic leg axis of a user for a knock-kneed leg axis and a bow-kneed leg axis, with the dynamic leg axis shown for two loading situations, as well as a corresponding knee movement pattern for each;

[0060] Figure 6 shows schematically a stance phase of a foot during a running situation, wherein the stance phase is limited in time by an initial contact of the foot with the ground and a lifting of the foot from the ground;

[0061] Figure 7 shows an example of the evaluation of the vertical movement of a user's nose to determine a stance phase within a running movement;

[0062] Figure 8 shows an example of the evaluation of the vertical movement of the nose and the vertical movement of the ankle joint to determine a stance phase within a running movement;

[0063] Figure 9 shows schematically the right leg of a user showing key points and a knee movement pattern;

[0064] Figure 10 schematically shows process steps for detecting key points, a stance phase and a movement pattern;

[0065] Figure 11 shows start and end values ​​for the stance phases in a large number of videos entered by a teacher during the training of a neural network;

[0066] Figure 12 schematically shows the process of analyzing a running video for the automatic detection of relevant parameters for selecting a suitable shoe category;

[0067] Figure 13 shows an example of a photograph showing a user's foot in an internal side view;

[0068] Figure 14 schematically illustrates the evaluation of a photo according to Figure 14 for determining a user's foot type; Figure 15 illustrates various foot types of a user;

[0069] Figure 16 shows examples of different static leg axes of a user;

[0070] Figure 17 shows an example of the assignment of different combinations of foot type and static leg axis to different insole profiles; and

[0071] Figure 18 shows a flow chart of a computer-implemented method for selecting a shoe category.

[0072] Figure 1 schematically shows a communication infrastructure suitable for carrying out a method according to the invention. A user 1 is assigned a communication device 11 with which they can communicate via a user interface. The communication device 11 is, for example, a smartphone, a tablet computer, or a personal computer. The communication device 11 comprises a camera (not separately shown) via which the user 1 can take photos and videos. Alternatively, photos or videos taken via a separate camera can be transmitted to the communication device 11. As explained below, the user 1 firstly records a video V in which they walk towards the camera from the front (or alternatively walk away from the camera from the back), and secondly records a photo P which shows one of their feet in an inside side view.

[0073] The video V and the photo P are transmitted to a data processing system 2 via a communications network 12. The communications network 12 is, for example, the Internet. However, this is only intended as an example. The communications network 12 can also consist of various subnetworks, for example, a WLAN, via which the communications device 11 communicates with an Internet access point, as well as the Internet or another packet-switched network.

[0074] The recipient of the video V and the photo P is the data processing system 2. This can comprise one or more servers, data storage devices and other computer resources arranged on the Internet. In one embodiment, the data processing system 2 is integrated into a cloud computing structure. However, this is not necessarily the case. The data processing system 2 comprises at least one processor and a memory which is operatively coupled to the at least one processor and in which program instructions are stored. The at least one processor is provided and designed to execute the program instructions. The program instructions implement a method for selecting a shoe category. After completion of the method, the selected shoe category can be sent to the user 1 or the communication device 11 assigned to the user via the communication network 12.This information can be sent to user 1 along with additional information. For example, the additional information can contain examples of shoes belonging to the determined shoe category. A determined insole profile associated with the user can also be sent to the user. Examples of implementing such a method are explained in the following figures.

[0075] Figure 2 shows a video image F of a video V showing a user 1 walking head-on toward a recording camera. The video image F is also referred to as a frame. The video V shows several steps of the user 1's walk. For example, the video V shows between four and ten steps of the user 1.

[0076] According to Figure 1, the video V is transmitted to and received by the data processing system 2 via the communications network 12. In the data processing system 2, key points of the user's skeleton are analyzed by evaluating the video V using a neural network. This is done, for example, using the "BlazePose" neural network from Google, LLC. Figure 3 shows corresponding key points of the skeleton. For example, 32 key points of the skeleton can be identified. The exemplary embodiments of the invention discussed below require only 7 of these key points, namely a nose key point 37, which indicates the position of the nose, two femoral head key points 31, 32, which indicate the center of the right and left femoral heads, and two knee key points 33, 34, which indicate the center of the knee and left femoral heads, respectively.indicate the center of the kneecap, as well as two ankle key points 35 and 36, which indicate the center of the right and left ankle joints. These key points are intended as examples. For example, a different key point can be assigned to the head, such as the mouth or the crown of the head.

[0077] In other embodiments, additional key points of the skeleton are evaluated to refine the analysis, for example, key points on the shoulders or additional key points on the foot. The key points are determined for each video frame of the video.

[0078] Figure 4 shows video image F of Figure 2 with additional representation of the key points determined by the neural network, including key points 31-37. Additionally, line 61 between the knee and the hip joint and line 62 between the ankle and the knee are shown.

[0079] Figure 5 shows two examples of the so-called dynamic leg axis for the right leg, which includes the imaginary line 61 between the knee and hip joint, the imaginary line 62 between the ankle and knee, and a dynamic angle α between these two lines 61 and 62. The left illustration in Figure 5 shows the dynamic leg axis for a knock-kneed leg axis. The right illustration in Figure 5 shows the dynamic leg axis for a bow-legged leg axis (for the left leg and knee, the illustrations are symmetrically opposite).

[0080] Figure 5 also shows a change in the dynamic leg axis when the knee is loaded. Such a load occurs while walking, when the user touches the ground with one foot (so-called "initial contact") and during the subsequent stance phase, which extends until the foot is lifted off the ground. Depending on the stability of the locomotor, the angle α between the two lines 61, 62 changes when a vertical force is applied. The dashed line indicates the leg axis at maximum load (for example, at time 44 in Figure 6 below). In a knock-kneed user (left illustration), the knee key point 33 moves medially inward. In a bow-legged user (right illustration), the knee key point 33 moves laterally. The maximum lateral movement that occurs during this process is referred to as knee movement pattern K.The knee movement pattern K represents a parameter for selecting a suitable shoe category for the user. As can be seen from Figure 5, the knee movement pattern K can be determined, for example, using the angle α in conjunction with the length of lines 61, 62. The angle α changes depending on the lateral position of the knee key point 33 (thus, it is different for the solid lines than for the dashed lines 61, 62). In both representations of Figure 5, the knee movement pattern K is equal to the difference in the height of the solid and dashed triangle with the vertices 31, 33, 35.

[0081] However, the determination of the knee movement pattern K can also be performed in another way. For example, the horizontal movement of the position of the knee key point 33 is recorded and evaluated.

[0082] The movement of the knee key point 33 occurs either inward (left image of Figure 5) or outward (right image of Figure 5). Accordingly, the knee movement pattern K also includes a direction. It can be specified as a vector or as a number with a positive or negative sign (corresponding to the inward or outward direction).

[0083] Figure 6 schematically shows a section of a human's walking cycle, with the so-called stance phase 4 being of importance in this case. Stance phase 4 extends between time 41, at which the user's foot makes initial contact with the ground, and time 43, at which the foot lifts off the ground again. During the time period 40 between these two times 41 and 43, the user's foot is in contact with the ground. The load on the foot changes. Shortly before a central time point 42, approximately 40% of the total length of time point 40 at time point 44, maximum foot load occurs. Time point 44 is also referred to as "midstance."

[0084] In the stance phase 4 under consideration, the running video comprises a large number of video images or frames Fm to Fo, each of which is evaluated.

[0085] Figure 6 also shows an analysis start point 45 and an analysis end point 46, which play a role in training a neural network for recognizing the running cycle, as will be explained later. The knee movement pattern K shown in Figure 5 is determined during stance phase 4. For example, the determined key points 31-37 (in particular the respective dynamic leg axes and / or the knee key point 33, 34) are compared with each other at the time of initial contact 41 and at the time 44 of maximum foot load (shortly before time 42), and the knee movement pattern K is determined from the difference between the medio-lateral positions of the knee key point 33, 34 at these two times (whereby the knee movement patterns for the two knee key points 33, 34 can be averaged).As explained, the time of maximum foot load 44 is typically 40% of the total length 40 of the stance phase 4, so that the video image Fk is evaluated after 40% of the stance phase 4.

[0086] The stance phase is determined, for example, by evaluating the movement of the nose key point 37 or another head key point, whereby the vertical movement of the key point is evaluated, as explained by way of example with reference to Figures 7 and 8.

[0087] Figure 7 shows the vertical movement z of the nose key point 37 as a function of time or as a function of the individual video frames F of the video. For this purpose, all video frames F of the video are analyzed in an analysis phase. The goal is to identify those video frames that are within the stance phase and are therefore relevant for analysis. The relevant time interval 40 of the stance phase to be analyzed can be determined from the course of curve 71 between the start point of the analysis phase (time 41) and the end of the analysis phase (time 43).

[0088] Figure 8 shows a further embodiment in which, as in Figure 7, the vertical movement of the nose key point 37 is recorded and evaluated in curve 81. Furthermore, the vertical movement of the ankle key points is evaluated according to curve 82. The curves are opposite, since when moving toward the camera, the head visually moves upward and the ankle downward. Using both curves 81 and 82 together, the times 41 and 43 and the time interval 40 can be more accurately determined.

[0089] Figures 7 and 8 merely provide examples for determining the time interval of stance phase 4. The time interval of stance phase 4 can alternatively be determined in a corresponding manner by evaluating other of the determined key points.

[0090] Figure 9 shows, as shown in Figure 5, the key points 31, 33, 35 in a video image F of the right leg as well as a determined knee movement pattern K.

[0091] As already noted, the evaluation of video V in data processing system 2 is carried out, for example, using at least one trained neural network. Figure 10 schematically illustrates the individual processing steps. In step 101, the video V created by the user is read in. A three-stage process then takes place. In the first stage 102, the key points are identified. Existing neural networks such as "BlazePose" from Google, LLC can be used for this purpose.

[0092] In the second stage 103, stance phase 4 is detected. In the third stage 104, the knee movement pattern within a selected interval of stance phase 4 is detected.

[0093] For the second stage 103, a neural network was trained by several people ("teachers") labeling a number of videos to create a training and validation dataset. The following time points or video frames were marked (tagged), referring to Figure 6: the analysis start point 45, each initial contact (time point 41), each time point 43 at which the foot lifts off the ground, and the analysis end point 46. A multitude of steps can lie between the analysis start point 45 and the analysis end point 46. Additionally or alternatively, time points or video frames relating to the vertical movement of key points can also be marked, as shown in Figures 7 and 8. The neural network is trained using the videos marked in this way.

[0094] In a currently imported video, the individual step phases, including the aforementioned times 41 and 43, are recognized by the trained neural network. Knowing that the time of maximum foot load 44 is at 40% of stance phase 4, this can now be determined. This defines the time interval between times 41 and 44, during which the knee movement pattern is evaluated in step 104. Alternatively, time 44 can be directly marked and trained during training of the neural network.

[0095] Alternatively, in the second stage 103, the stance phase 40 can be determined by an algorithm that analyses the individual video images and tracks and analyses the change in defined key points.

[0096] The knee movement pattern is determined, for example, according to Figure 5, by evaluating the corresponding angle parameters and movements of the key points, in particular by comparing the angle parameters or key points at times 41 and 44. The knee movement pattern can be calculated using an algorithm that compares the angle parameters or key points at the different times. Alternatively, a neural network can also be trained and used for this purpose.

[0097] The determination of the correct value, for example of initial contact 41 and lifting of the foot from the ground 43, is carried out, as explained, for example by manual analysis of the video. This is shown as an example in Figure 11. The left-hand diagram shows the frame corresponding to the initial contact for a large number of videos and was individually determined by a teacher. The right-hand diagram shows the frame corresponding to the final contact (i.e., the lifting of the foot from the ground) for a large number of videos and was also determined by a teacher. Figure 11 shows these values ​​for a large number of videos. For a single specific video, the frame of initial contact and the frame of final contact were determined by individual evaluation or manually, and the neural network was trained using these values.

[0098] Figure 12 schematically shows the sequence of processing a video V. Steps 121-124 correspond to steps 101-104 of Figure 10, so that reference is made to the relevant description of Figure 10.

[0099] Finally, in step 125, the individual video images are reassembled with the corresponding additional information to form a complete video. In embodiments of the invention, this video can be sent to user 1 for information via the communications network 12 (see Figure 1). According to Figure 1, it can be provided that user 1 also sends a photo P of the foot to data processing system 2 via their communications device 11. This is explained in more detail in Figures 13 and 14. According to Figure 13, photo P is an inner side view of the user's foot 15. The foot has a specifically pronounced inner arch 150.

[0100] According to Figure 14, a specific foot type is determined based on the obtained photo P. This is done using another neural network. Well-known neural networks can be used for this, for example, the neural network and the COCO dataset from Microsoft. In step 141 of Figure 14, the photo P is acquired. In step 142, the photo P is preprocessed. During preprocessing, pattern recognition and background removal occur. The correspondingly processed photo P1 is input into a neural network in step 143 and classified in the neural network. In this process, the photo P is classified into one of four foot types, FT1 to FT4. The different foot types correspond to different characteristics of the inner longitudinal arch of the foot. For example, foot type FT1 has a very high inner longitudinal arch, while type FT4 has a flat inner longitudinal arch. The other foot types lie in between.

[0101] The neural network can be trained using supervised learning, whereby a value of the inner longitudinal arch of the foot determined by the neural network is compared with a correct value of the longitudinal arch provided by a teacher, and the neural network is trained in this way. The correct value is determined by an individual analysis of the photos by a teacher. Alternatively, the foot type can be identified using other image processing and pattern recognition methods, for example, using a corresponding algorithm.

[0102] The determined foot type FT, as well as the determined knee movement pattern K, are entered into a predefined formula for selecting the shoe category. It can be provided that additional parameters are included in this predefined formula. Numerous models can be developed as to how the aforementioned and additional parameters can be mapped into a recommendation for specific shoe categories. The following describes an exemplary embodiment in which additional parameters include information on the static leg axis type SB, information on the user's running profile LP, and information on the user's gender G. These additional parameters or information are entered by the user 1 via their communication device 11, as shown in Figure 1, and sent to the data processing system 2; they do not need to be calculated or analyzed.

[0103] The static leg axis type SB, for example, is classified into three types: a straight leg axis, a knock-kneed leg axis, and a bowlegged leg axis. The static leg axis should not be confused with the dynamic leg axis. It is defined by the shape of the legs when standing. Information about the static leg axis type SB is typically known to the user. Alternatively, this information can be determined by evaluating the video (for example, by averaging the determined dynamic leg axis types according to Figure 5).

[0104] For example, the running profile of user LP includes the three types beginner (LP1), advanced (LP2) and experienced runner (LP3).

[0105] For example, the user’s gender G includes the types male (Gl) and female (G2).

[0106] According to the example considered, the following parameters are included in the final formula:

[0107] Knee movement patterns

[0108] Foot type FT static leg axis SB

[0109] Running profile of user LP

[0110] Gender of user G

[0111] One embodiment provides for the foot type FT and the static leg axis SB to be combined into a single parameter that specifies a suitable insole profile SP. This is explained using Figures 15-17. According to Figure 15, the foot type FT comprises four classes of foot types FT1 to FT4, which differ in the development of the inner longitudinal arch of the foot. According to Figure 16, the static leg axis SB comprises three classes of static leg axis types SB1, SB2, and SB3, which correspond to an O-legged leg axis (SB1), a straight leg axis (SB2), and an X-legged leg axis (SB3).

[0112] The insole profile SP suitable for the user is determined by a combination of foot type and static leg axis type. In the illustrated example, three different insole profiles SP1, SP2, and SP3 result. For example, the combination of foot type FT1 with leg axis type SB1 or leg axis type SB2 results in the insole profile SP1. The different insole profiles SP1, SP2, and SP3 correspond to a different formation of the metatarsal bridge in terms of shape and thickness. For example, the metatarsal bridge is most pronounced in the insole profile SP1.

[0113] One embodiment provides that the determined insole profile SP1, SP2, SP3 is sent separately to the user 1 via the communication network 12 as shown in Figure 1. If the user is only interested in purchasing an insole with a profile suitable for him, this information is already sufficient for him.

[0114] By combining the foot type FT with the static leg axis SB to the parameter of the insole profile SP, the following parameters are included in the final formula for selecting a shoe category in the example considered:

[0115] Knee movement pattern K

[0116] Insole profile SP (depending on FT and SB) Walking profile of the user LP Gender of the user G

[0117] The parameters SP, LP, and G encompass several types or classes. For example, the user's running profile LP, as explained above, comprises the three classes beginner (LP1), advanced (LP2), and professional (LP3). The insole profile comprises the types or classes SP1, SP2, and SP3. The gender G comprises the types or classes male (Gl) and female (G2). The knee movement pattern K is a number or a vector (if the direction of the knee movement is also specified) that corresponds to the extent of the mediolateral movement of the knee according to Figure 5. If a vector is considered, it can point either inwards or outwards. The direction of the vector can be specified as the sign of the number.

[0118] Each of these four parameters is assigned a specific score using a predefined function. The SP, LP, and G parameters are assigned based on the class that was determined or entered. The K parameter is assigned based on the value (number) that was determined.

[0119] If the function is denoted as "f", the total score P is as follows:

[0120] P = f(K) + f(SP) + f(LP) + f(G).

[0121] The shoe category LK to be selected is ultimately determined from the determined score P. Specifically, it is intended that different shoe categories LK1 to LKn are determined depending on the determined scores. For example, five shoe categories LK1 to LK5 are assigned, with shoe category LK1 being selected if the total score P is within a first value range, shoe category LK2 being selected if the total score P is within a second value range, etc.

[0122] Some examples are given below.

[0123] For example, if the knee movement pattern for a large number of users is empirically determined to lie between the values ​​-a and -ib, this interval can be divided into four subintervals [-a, -a / 2[, [-a / 2, 0[, [0, b / 2[, [b / 2, b]. These four intervals can be assigned the scores -7, -3, 3, 7, for example, using the function f(K).

[0124] For example, the function f(SP) assigns five points to the insole profile SP1, 15 points to the insole profile SP2, and 35 points to the insole profile SP3. For example, the function f(LP) assigns five points to the running profile LP1, zero points to the running profile LP2, and minus 5 points to the running profile LP3.

[0125] For example, the function f(G) assigns zero points to gender Gl (male) and five points to gender G2 (female).

[0126] In these examples, a female advanced user with insole profile SP2 and a knee movement pattern in the interval [-a, -a / 2[ would receive the following total score:

[0127] P = f(K) + f(SP) + f(LP) + f(G) = -7 + 15 + 0 + 5 = 13

[0128] A score of 13, for example, falls within a range corresponding to the second shoe category, LK2. The determined shoe category is associated with a stability class that the shoes in this shoe category meet.

[0129] The determined shoe category is communicated to user 1 via the communications network 12. Additionally, user 1 can be provided with information about shoes that fall into the determined shoe category. If necessary, user 1 can select from these shoes and purchase them as part of an online purchase.

[0130] It should be noted that the functions f(K), f(SP), f(LP) and f(G) mentioned above, or the mappings to point values ​​they make, can be determined through empirical knowledge or heuristically, if necessary in cooperation with medical professionals and running experts.

[0131] Figure 18 summarizes the essential steps of the described method. According to step 181, a video of a user is received by the data processing system, wherein the video is received via a communications connection, and wherein the user in the video takes several steps toward the recording camera or runs away from it from behind. In step 182, key points of the skeleton are determined. This is done by evaluating the video or the individual video images. At least key points on the head, femoral head, knee, and ankle are determined. In step 183, a stance phase of at least one foot of the user or runner is determined by evaluating at least some of the key points. The stance phase is defined as the period between the initial contact of the foot with the ground and the time the foot leaves the ground.

[0132] Furthermore, with regard to the video images relating to the stance phase, the knee movement pattern is determined from at least some of the key points. The knee movement pattern is the range over which the user's knee moves during the stance phase due to varying loads. It indicates the medio-lateral position change of a knee key point in the frontal plane during the stance phase. The knee movement pattern can be a number or a vector, with the latter being inward or outward.

[0133] In step 185, the determined knee movement pattern is selected from a plurality of predefined shoe categories for selecting a shoe category. The determined parameter is incorporated directly or indirectly into a predefined formula for determining the shoe category. Additional parameters can be considered in the formula. The shoe category is selected, for example, using a score derived from the parameter values.

[0134] It is understood that the invention is not limited to the embodiments described above, and various modifications and improvements may be made without departing from the concepts described herein. It is further understood that any of the described features may be used separately or in combination with any other features, provided they are not mutually exclusive. The disclosure extends to and encompasses all combinations and subcombinations of one or more features described herein. Where ranges are defined, these include all values ​​within these ranges, as well as all subranges that fall within a range.

Claims

COMPUTER-IMPLEMENTED PROCESS FOR SELECTING A SHOE CATEGORY Patent claims 1. A computer-implemented method for selecting a shoe category for a user, the method comprising: Receiving (181) a video (V) of the user (1), wherein the video (V) is received via a communication connection, and wherein the user (1) in the video (V) walks with several steps towards the recording camera or walks away from it from behind, determining (182) key points (31-37) of the skeleton of the user (1) and their movement by evaluating the video (V), wherein the key points include key points (31-37) on the head, the femoral head, the knee and the ankle joint, determining (183) a stance phase (4) of at least one foot from at least some of the key points (31-37), wherein the stance phase (4) is defined as the period (40) between an initial contact (41) of the foot with the ground and the lifting (43) of the foot from the ground, determining (184) a knee movement pattern (K) during the stance phase (4) from at least some of the key points (31-37), which medio-lateral change in position of a knee key point (33,34) in the frontal plane, use (185) of the determined knee movement pattern (5) for the selection of a shoe category from a plurality of predefined shoe categories, wherein the knee movement pattern (K) is directly or indirectly used as a parameter in a predefined formula for determining the shoe category.

2. Method according to claim 1, characterized in that the beginning (41) and the end (43) of the stance phase (4) are determined by evaluating the movement of at least one key point (37) on the head and / or on the ankle joint, wherein the vertical movement of the key point (37) is evaluated.

3. Method according to claim 1 or 2, characterized in that the knee movement pattern (K) is evaluated by comparing skeleton points (31-37) at the time of initial contact (41) and at the time of maximum foot load (44) during the stance phase (4).

4. Method according to one of the preceding claims, characterized in that the knee movement pattern (K) is determined via an evaluation of the dynamic leg axis (61, 62, o), wherein the dynamic leg axis comprises the imaginary line (62) between the ankle joint and the knee, the imaginary line (61) between the knee and the hip joint, and the dynamic angle (o) between these two lines.

5. Method according to claim 4, characterized in that a change in the angle (o) is evaluated and this is converted into the knee movement pattern (K).

6. Method according to one of the preceding claims, characterized in that the knee movement pattern (K) is determined by evaluating the medio-lateral movement of the knee key point (33, 34) during the stance phase (4).

7. Method according to one of the preceding claims, characterized in that the following are evaluated as key points: the position of the nose as the key point (37) assigned to the head; the center of the femoral head as the key point assigned to the femoral head (31, 32); the center of the knee or the center of the kneecap as the key point assigned to the knee (33, 34); and the center of the ankle joint (35, 36) as the key point assigned to the ankle joint.

8. Method according to one of the preceding claims, characterized in that the key points (31-37) are determined by means of a first trained neural network.

9. Method according to claim 8, characterized in that the determination of the stance phase (4) and / or the knee movement pattern (K) is carried out by means of the first trained neural network, a second trained neural network or an algorithm.

10. Method according to one of the preceding claims, characterized in that the method comprises the further steps: Also receiving a photo (P) of a foot (15) of the user (1), the photo (1) being received via the communication link, the photo (P) representing an internal side view of one of the user's feet, Determining a foot type (FT1-FT4) of the user by evaluating the photo (P), wherein the specific foot type (FT1-FT4) belongs to a number of predefined foot types that differ in the shape of the inner longitudinal arch of the foot (150), and Use of the specific foot type (FT1-FT4) to determine the shoe category, whereby the foot type is directly or indirectly included as an additional parameter in the predefined formula for determining the shoe category.

11. Method according to claim 10, characterized in that the determination of a foot type (FT1-FT4) of the user is carried out by evaluating the photo (P) by means of a further trained neural network or an algorithm.

12. Method according to one of the preceding claims, characterized in that the provided video (V) is returned to the user in edited form, wherein the video (V) additionally displays the determined key points (31-37).

13. Method according to one of the preceding claims, as far as dependent on claim 10, characterized in that the predefined formula for determining the shoe category has as parameters: the knee movement pattern (K) for one knee or the mean value of the knee movement patterns (K) of both knees, and an insole profile (SP), wherein the insole profile (SP) is determined by the combination of the specific foot type (FT1-FT4) with a static leg axis type (SB1-SB3) of the user.

14. Method according to claim 13, characterized in that the static leg axis type (SB1-SB3) is provided as information by the user or determined by evaluation of the video (V) and the Static leg axis types (SB1-SB3) include the types “straight leg axis”, “knock-kneed leg axis” and “bow-kneed leg axis”.

15. Method according to claim 13 or 14, characterized in that the insole profile (SB1-SB3) determined by the combination of the foot type (FT1-FT4) with the static leg axis type (SB1-SB3) of the user is communicated separately to the user.

16. Method according to one of claims 13 to 15, characterized in that the predefined formula for determining the shoe category assigns points to the parameter of the knee movement pattern (K) and to the parameter of the insole profile (SB), which are added together, the shoe category being selected from the plurality of predefined shoe categories depending on the total number of points.

17. Method according to one of claims 13 to 16, characterized in that the predefined formula for determining the shoe category has as further parameters: information on the running profile of the user, which is provided by the user, wherein the information comprises at least the indication of whether the user is a beginner, an advanced runner or an experienced runner, information on the gender of the user, which is provided by the user.

18. Method according to claim 17, as far as it relates back to claim 16, characterized in that the predefined formula for determining the shoe category further assigns points to the parameter of the running profile and the parameter of the gender, which correspond to the points for the parameter of the Knee movement pattern (K) and the insole profile parameter (SB) are added, whereby depending on the total number of points the shoe category is selected from the majority of predefined shoe categories.

19. Method according to one of the preceding claims, characterized in that a total of at least three shoe categories are predefined, from which one shoe category is selected, wherein the shoe categories correspond to different stability classes of shoes.

20. Method according to one of the preceding claims, characterized in that the method comprises the following further steps: Sending the user information about at least one shoe that corresponds to the specified shoe category.

21. Computer program with program code for carrying out the method steps according to one of claims 1 to 20, when the computer program is executed in a computer.

22. A data processing system for selecting a shoe category for a user, comprising: at least one processor; a memory operatively coupled to the at least one processor and storing program instructions, the at least one processor being configured to execute the program instructions, and the program instructions comprising: Receiving (181) a video (V) of the user (1), wherein the video (V) is received via a communication connection, and wherein the user (1) in the video (V) walks in several steps towards the front of the recording camera or walks away from it from behind, Determining (182) key points (31-37) of the skeleton of the user (1) and their movement by evaluating the video (V), wherein the key points comprise key points (31-37) on the head, the femoral head, the knee and the ankle joint, - Determining (183) a stance phase (4) of at least one foot from at least some of the key points (31-37), wherein the stance phase (4) is defined as the period (40) between an initial contact (41) of the foot with the ground and the lifting (43) of the foot from the ground, - Determining (184) a knee movement pattern (K) during the Stance phase (4) from at least some of the key points (31-37), which indicates the medio-lateral position change of a knee key point (33, 34) in the frontal plane, use (185) of the determined knee movement pattern (5) for the selection of a shoe category from a plurality of predefined Shoe categories, whereby the knee movement pattern (K) is used directly or indirectly as a parameter in a predefined formula for determining the shoe category.

Images