Interactive training system and method based on eye-foot coordination

Abstract

The present invention proposes an interactive training method and system based on eye-foot coordination, including: capturing user body joint information and judging the user's posture based on the user's joint information; wherein different postures represent different interactive operations; during the training phase, the user to be trained is trained by interacting with moving target elements in the interactive interface until the training is completed; wherein, during training, according to the different foot lengths of the users to be trained and different training records, the user is adaptively provided with moving targets of different sizes and speeds.

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual reality, and in particular relates to an interactive training system and method based on eye-foot coordination. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of virtual reality (VR) technology, it has been widely used in education, healthcare, culture and entertainment. The novel interactive methods and novel experience of VR technology are often very attractive to children. Designing a training system based on VR technology has the potential to address problems such as adolescent addiction to video games, lack of physical exercise, and poor physical fitness.

[0004] Common virtual reality implementations include head-mounted displays (HMDs) and CAVE immersive display systems. However, HMD-based implementations are often associated with symptoms such as motion sickness, and manufacturers do not recommend their use by children under 13. CAVE-based implementations often require complex setup environments and are not suitable for home use.

[0005] However, with the increasing use of projectors in homes, projection-based display and interaction have become a viable approach. Projection-based floor interaction systems eliminate the need for complex interactive environments, transforming the floor into a real-time, responsive environment. Furthermore, they improve the user's eye-foot coordination during movement and interaction with virtual objects.

[0006] The mobile target selection task consists of two phases: target tracking and target capture. During the target tracking phase, users need to track the mobile target and make appropriate plans to accurately time their selection. Compared to static targets, capturing a mobile target often takes longer and has a higher capture failure rate. Furthermore, factors such as latency in information transmission from interactive devices and the choice of interaction method can negatively impact target capture. In interactive design, solutions to inaccurate mobile target selection include adding a tail to the target, pausing target movement during selection, and expanding the range of target options. These solutions often visually alter interface elements and even the original selection logic. Several target prediction models have been proposed for mobile target selection, but these models are primarily based on desktop or mobile devices, and research on mobile target selection based on foot interaction is limited. Summary of the Invention

[0007] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides an interactive training system based on eye-foot coordination, which performs interactive training by capturing moving elements in a ground interface.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] In a first aspect, an interactive training method based on eye-foot coordination is disclosed, comprising:

[0010] Capture the user's body joint information and determine the user's posture based on the user's joint information; different postures represent different interactive operations;

[0011] In the training phase, the user to be trained interacts with the target elements moving in the interactive interface until the training is completed;

[0012] During training, the system adaptively provides users with moving targets of different sizes and speeds based on the foot lengths and training records of different users to be trained.

[0013] As a further technical solution, before training, the user to be trained first captures the user's color image information and identifies the user's registration information;

[0014] If the user information to be trained is the information captured and recognized for the first time, the user's registration information is received and the user image is stored in the user database; otherwise, the user registration information is directly retrieved from the database;

[0015] After successful registration or login, the user registration information will be displayed in the interactive interface.

[0016] As a further technical solution, the specific method of capturing the user's color image information is:

[0017] Obtaining user color image and user depth image information;

[0018] Estimate the 3D coordinate information of the user's nose and neck joints through the captured images;

[0019] Convert the obtained 3D coordinates into 2D coordinates based on the color image;

[0020] The user's face rectangular image is cropped using the obtained 2D coordinates.

[0021] As a further technical solution, the method for generating the moving target element in the interactive interface is as follows:

[0022] The diameter of the moving target in the interactive interface is related to the user's foot length;

[0023] estimating the user's height and the user's foot length based on the captured user's color image information;

[0024] Estimate the pixel length of the foot in the interaction area based on the estimated length of the user's foot;

[0025] The moving target elements in the interactive interface are all regular graphic targets. In this invention, a circular target is used as an example for explanation.

[0026] The diameter range of the moving target is determined based on the estimated length of the foot in the interaction area.

[0027] As a further technical solution, the movement speed of the target in the interactive interface increases with the training time.

[0028] As a further technical solution, the specific steps for the user to interact with the moving target element in the interactive interface are:

[0029] Generate interactive interface;

[0030] Get the 3D position coordinates in the user's physical space;

[0031] Convert 3D position coordinates to 2D plane coordinates based on the ground;

[0032] Convert the position coordinates based on the ground coordinate system into the position coordinates based on the velocity coordinate system;

[0033] If the target is in single moving target mode, determine whether the location of the target selected using foot interaction (referred to as the landing point) is within the 2σ confidence interval of the three-Gaussian model corresponding to the moving target's landing point distribution. If the landing point is within the confidence interval, it means that the user has successfully captured the moving target;

[0034] If it is a multi-moving target capture mode, traverse all the moving targets in the interaction area and determine whether the landing point is within the 2σ confidence interval of the three-Gaussian model corresponding to the moving target landing point distribution. If so, find the target that best meets the user's selection intention among all the moving targets that meet the conditions.

[0035] As a further technical solution, when finding the target that best meets the user's selection intention among all the moving targets that meet the conditions, specifically:

[0036] For all moving targets in the interactive area, the user's landing point coordinates are input into the Bayesian decision formula of all targets that meet the conditions (the landing point coordinates are within the 2σ confidence interval of their landing point distribution model), and the probability of the landing point being in each target is obtained;

[0037] The target with the highest calculated probability is selected as the final target.

[0038] As a further technical solution, the step of determining whether the training is completed includes:

[0039] Determine whether the user has made an end sign gesture, or whether the training time has been reached. If one of the above conditions is met, the training is exited.

[0040] In a second aspect, an interactive training system based on eye-foot coordination is disclosed, comprising:

[0041] The user posture judgment module is configured to: capture the user's body joint information and judge the user's posture based on the user's joint information; different postures represent different interactive operations;

[0042] The training module is configured as follows: in the training phase, the user to be trained is trained by interacting with the target element moving in the interactive interface until the training is completed;

[0043] During training, the system adaptively provides users with moving targets of different sizes and speeds based on the foot lengths and training records of different users to be trained.

[0044] In a third aspect, an interactive training device based on eye-foot coordination is disclosed, comprising:

[0045] Projection equipment, host computer and location tracking equipment;

[0046] The projection device is used to project the content of the interactive training and interactive feedback information run by the host onto the ground area;

[0047] The position tracking device captures the user's body joint information and transmits the captured data to the host, which is configured to execute the above method, specifically:

[0048] Determine the user's posture based on the user's joint information; different postures represent different interactive operations;

[0049] In the training phase, the user to be trained interacts with the target elements moving in the interactive interface until the training is completed;

[0050] During training, the system adaptively provides users with moving targets of different sizes and speeds based on the foot lengths and training records of different users to be trained.

[0051] One or more of the above technical solutions have the following beneficial effects:

[0052] According to different users' foot lengths and different training records, the technical solution of the present invention adaptively provides users with moving targets of different sizes and speeds, thereby providing users with a comfortable training experience and effective training results.

[0053] To address the difficulty in capturing moving target elements in foot-based motion interactions, the technical solution of the present invention uses the landing point distribution of moving targets captured under different sizes and speeds to fit a two-dimensional three-Gaussian model. In practical applications, this model is applied to the prediction of target landing point distribution, thereby improving the accuracy of user selection and enhancing system usability.

[0054] This technical solution avoids the discomfort of VR motion sickness often associated with HMD devices and solves the tedious and complex issue of CAVE environment construction. It offers the advantages of convenience and ease of operation, requiring no additional user learning, reducing the learning burden. Users can interact simply by moving around, creating a novel training experience that allows for effortless training.

[0055] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0057] Figure 1 This is a specific flow chart for implementing interactive training in the first embodiment of the present disclosure;

[0058] Figure 2 This is a flow chart of mobile target selection according to the first embodiment of the present disclosure;

[0059] Figure 3 This is a schematic diagram of the overall architecture of the second embodiment of the present disclosure;

[0060] Figure 4 This is a specific flow chart of a reaction training according to the second embodiment of the present disclosure;

[0061] Figure 5 This is a schematic diagram of the reaction test module interface of the second embodiment of the present disclosure;

[0062] Figure 6 This is a diagram showing the operating effect of the training system supporting eye-foot coordination according to the second embodiment of the present disclosure;

[0063] Figure 7 This is a specific flow chart of interactive learning in Example 3 of the present disclosure;

[0064] Figure 8 This is a schematic diagram of the two-player function selection interface of the third embodiment of the present disclosure;

[0065] Figure 9 This is a diagram showing the operating effect of the learning system supporting eye-foot coordination according to the third embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0067] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0068] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0069] Example 1

[0070] This embodiment discloses an interactive training method based on eye-foot coordination, such as Figure 1 As shown, this embodiment includes the following steps:

[0071] Step 1: Use a Kinect device to capture a color image of the user and use a facial recognition algorithm to identify the user's registered information. If the user is new, their information (including user ID, gender, height, etc.) is registered and the image is stored in the user database. If the user is an existing user, the user information is directly retrieved from the database. After successful registration or login, the user information is displayed on the interactive interface.

[0072] By using the Kinect device to track the user's location while capturing color images of the user for face recognition registration and login, this method can meet the need of assigning different training to different users and can provide the most effective training personalized for each user.

[0073] The specific method for performing face recognition using user color image data is as follows:

[0074] Step 101: Obtain user color image and user depth image information through a Kinect device;

[0075] Step 102: Estimate the 3D coordinate information of the user's nose and neck joints using the image captured in step 101;

[0076] Step 103: converting the two 3D coordinates obtained in step 102 into 2D coordinates based on the color image;

[0077] Step 104: A rectangular image of the user's face is cropped using the 2D coordinates obtained in step 103.

[0078] Step 105: Compare the user's facial image obtained in step 104 with the data in the face database. If the user information exists in the face database, log in directly; otherwise, the user registers and stores the facial image in the face database.

[0079] Step 2: Provide different training for different users. Based on the user information identified in step 1, different training is assigned to users: default training content is provided for new users; for old users, training content is provided based on their historical progress.

[0080] Step 3: In the training phase, the user is trained by interacting with target elements (including single target elements or multiple target elements) moving in the interactive interface.

[0081] Among them, the target elements are adaptively generated based on the user's foot length information and historical training information, providing users with effective training while giving them the best visual experience. The generation method of moving target elements in the interactive interface is as follows:

[0082] (1) The moving target elements in the interactive interface are all regular graphics. The present invention takes a circular target as an example for explanation.

[0083] (2) The diameter of the moving object in the interactive interface is related to the user's foot length. From experience, we know that the length of a person's foot is approximately: foot length = height / 7.

[0084] (3) Estimate the user's height using the Kinect device: The estimation formula is:

[0085] uHeight=(uHead.Y-uFootL.Y+a) / 10(cm) or

[0086] uHeight=(uHead.Y-uFootR.Y+a) / 10(cm)

[0087] Where a = 150, which is an empirical value; uHeight represents the user's height; uHead is a three-dimensional variable representing the three-dimensional coordinates of the user's head joints; uFootL and uFootR represent the three-dimensional coordinates of the user's foot joints, respectively. The coordinates of the human body joints captured by the Kinect device are in millimeters (mm), so unit conversion is required here.

[0088] The estimated length of the user's foot is:

[0089] uFootLen=uHeight / 7(cm)

[0090] Convert the position of the user's foot in the 3D coordinate system to the 2D plane coordinate based on the ground. The length of the foot in the 2D plane coordinate based on the ground is:

[0091] uFootLenS=uFootLen*a(px)

[0092] Where a = 100 / 22 is an empirical value. uFootLenS represents the length of the foot in the 2D plane coordinates based on the ground after conversion.

[0093] In the present invention, the diameter of the circular moving target ranges from {uFootLenS, uFootLenS+10, uFootLenS+20, uFootLenS+30} (px). Circular targets of different sizes appear randomly.

[0094] (4) The speed of the moving object in the interactive interface increases with the training time, and its range is {150, 170, 190, 210, 230, 250, 270, 290} (px / s). For new users, the initial speed is 150px / s. For old users, the initial speed is determined by their historical records. In a single training session, the speed changes four times. For example, for a new user, the initial speed is 150px / s, and the maximum speed during this training session is 210px / s.

[0095] The user controls the selection of the target by moving and standing still. The user's body movement indicates tracking of the target, and when the user's motion state changes from moving to standing still, it indicates that the user has made a selection. This interaction method requires the user to follow the target with their eyes and select the target with their feet, which not only exercises the user's eye-foot coordination ability, but also mobilizes the user's enthusiasm for physical activities. Among them, the specific steps for the user to interact with the moving target element in the interactive interface are as follows: Figure 2 As shown:

[0096] Step 301: Initialization, generating an interactive interface.

[0097] The resolution of the interactive area is pWidth*pHeight, the width of the interactive area is sWidth, the height of the interactive area is sHeight, the coordinate of the upper boundary of the interactive area in the Z direction in the Kinect coordinate system is y0, and the coordinate of the left side of the interactive area in the X direction in the Kinect coordinate system is x0.

[0098] Step 302: Obtain the user's location coordinates through the Kinect device.

[0099] Use the Kinect device to obtain the actual position (x_real, y_real, z_real) of the user's pelvic joint point (SPINE_NAVAL) in three-dimensional space.

[0100] The specific capture method is as follows:

[0101] (1) The Kinect device can capture 30 frames of images per second;

[0102] (2) Setting a queue of length N to record the user's SPINE_NAVAL joint point positions in the latest N frames in real time, where N=10 in the present invention;

[0103] (3) Determine whether the jitter of all SPINE_NAVAL joint positions in the queue in the x-direction and the z-direction is less than a pre-designed threshold delt. In the present invention, the threshold delt=20.

[0104] (4) If the conditions in (3) are met, it means that the user has made an action of selecting a target;

[0105] (5) Record the user’s current SPINE_NAVAL joint point information (x_real, y_real, z_real).

[0106] Step 303: Convert the 3D coordinates based on the Kinect device into 2D plane coordinates based on the ground.

[0107] Convert the 3D coordinates (x_real, y_real, z_real) based on the Kinect device to 2D coordinates (x_screen, y_screen) based on the ground. The specific conversion method is as follows:

[0108] x_screen=(x0-x_real) / sWidth*pWidth

[0109] y_screen=(z_real-y0) / sHeight*pHeight

[0110] Step 304: Convert the position in the ground-based 2D coordinate system into a position coordinate based on the velocity coordinate system.

[0111] Converting the position coordinates (x_screen, y_screen) based on the ground coordinate system to the coordinates (x_velocity, v_velocity) based on the target motion velocity coordinate system can avoid the difficulty in processing the velocity components in the x and y directions caused by the different target movement directions and reduce the complexity of the problem. The specific method is: set the coordinates of the center of the circular target to (x_c, y_c), the angle between the target movement direction and the positive x direction of the interaction area to β, and at the same time, stipulate that the target movement direction is the positive x-axis direction of the velocity coordinate system, and the velocity normal direction is the positive y-axis direction. The specific conversion method is as follows:

[0112] x_velocity=(x_screen-x_c)*cos(β)+(y_screen-y_c)*sin(β)

[0113] y_velocity=-(y_screen-y_c)*cos(β)+(x_screen-x_c)*sin(β)

[0114] Step 305: Determine whether it is single target capture.

[0115] Determine whether the training mode is a single target capture mode or a multi-target capture mode. If it is a single target mode, jump to step 306; otherwise, jump to step 307.

[0116] Step 306: After the user makes a selection action, determine whether the coordinates of the user's landing point are within the 2σ confidence interval of the three-Gaussian model corresponding to the target landing point distribution.

[0117] The three-Gaussian model used in the present invention is expressed as:

[0118] X~(μ,∑)

[0119]

[0120]

[0121] For capturing moving targets, the three-Gaussian model assumes that the landing point distribution follows a Gaussian distribution consisting of three Gaussian components, which are generated by the uncertainty of the input device, target size, and target speed. W and V represent the size and speed of the circular target, respectively. Ax, bx, cx, dx, dy, ex, ey, fx, fy, and gx are empirically determined constants. In this embodiment, the values ​​of the above parameters are:

[0122]

[0123] Determine whether the landing point coordinates are within the 2σ confidence interval of the three-Gaussian model corresponding to the target landing point distribution in the velocity coordinate system. If the landing point is within this range, it means that the user has captured the moving target and the capture is successful. Otherwise, jump to step 302.

[0124] Step 307: Find a target whose landing point is within its 2σ confidence interval.

[0125] The definition of the three-Gaussian model parameters is the same as that described in step 306 and will not be repeated here.

[0126] Traverse all the moving targets in the interaction area and determine whether the landing point is within the 2σ confidence interval of the three-Gaussian model corresponding to the moving target landing point distribution.

[0127] Step 308: Determine whether the target number is greater than 0.

[0128] Determine whether there is a target that meets the conditions of step 307. If such a target exists, it means that the user has made a choice. Then, it is necessary to find the target that best meets the user's selection intention among all targets that meet the conditions and jump to step 309. Otherwise, it means that the user has not made a choice. The reason for this phenomenon may be the misrecognition of the Kinect device. At this time, it is necessary to jump to step 302.

[0129] Step 309: Input the landing point coordinates into the Bayesian decision formula.

[0130] The three-Gaussian landing point distribution model is embedded in the Bayesian decision formula in the form of a likelihood function, and the landing point coordinates are input into the Bayesian decision formula of all targets that meet the conditions in step 307 to obtain the probability that the landing point is located in each target.

[0131] Assume there are n moving targets {t1, t2, ... tn} in the scene, s represents the landing point generated by the user's selection, the conditional probability that target t is the intended target is P(t|s), P(t) represents the prior probability of selecting target t, P(s|t) is the likelihood function composed of the probability density function of the landing point distribution, and P(s) is the normalization constant. The above Bayesian decision formula is:

[0132] P(t|s)=P(s|t)P(t) / P(s)

[0133] Step 310: Select the target with the highest probability calculated by Bayes' rule.

[0134] The target with the highest probability calculated in step 309 is selected as the final selected target.

[0135] Step 4: Determine whether the training is completed.

[0136] There are two ways to determine whether the user has made the end sign gesture, or whether the training time has been reached. If either of the above conditions is met, the user exits the training. After the user exits the training, the user's training information will be fed back to the interactive interface, including the score, history record, training time, and exercise amount.

[0137] The user's sign of finishing the training is that the user raises both hands horizontally and faces the Kinect device.

[0138] Specifically, the posture is defined as follows:

[0139] The Kinect device is used to obtain the corresponding 3D coordinate values ​​of the user's left hand, right hand, left shoulder, and right shoulder joints: hand_left, hand_right, shoulder__left, shoulder_right. When both hands are raised horizontally, the absolute value of the x-value difference between the hand and the shoulder on the same side is approximately equal to the length of the user's arm.

[0140] The user's arm length can be estimated by calculating the Euclidean distance between the hand and the shoulder on the same side, and the user's arm length is set as length;

[0141] The method for judging whether both hands are raised horizontally is as follows:

[0142] |hand_left.X-shoulder_left.X|>length-a and |hand_right.X-shoulder_right.X|>length-a

[0143] Here, a is a value determined by experience, and is 10 in this embodiment.

[0144] In particular, due to factors such as the error in the Kinect device's estimation of human joint points and the standardization of the user's posture, the judgment condition only needs to meet the threshold range, that is, the X-direction distance between the hands and the shoulders is greater than the user's arm length minus a.

[0145] Among them, you can also quit training if you reach the training time requirement.

[0146] To ensure the training effect of the user, in this embodiment, the duration of a single training session is set to 8 minutes, or 480 seconds. After the user starts training, the countdown begins. When the countdown reaches 0, the user will automatically exit the training.

[0147] The disclosed technical solution features simple implementation, novel content, and easy operation. It supports single or dual users interacting in an interactive area using body movements and real-world walking. It utilizes position tracking devices to identify user postures and obtain real-time location information. It utilizes projection equipment to transform the floor into an interactive environment, creating a training environment that combines movement and fun, and promotes the development of cognitive, social, and motor skills. The method for capturing moving targets is foot interaction, rather than flexible fingers, making the task more challenging and improving the user's eye-foot coordination.

[0148] To achieve the goal of motion, this disclosure requires users to capture moving elements in the interactive interface. To improve the accuracy of target selection and enhance user stickiness, this disclosure introduces a three-Gaussian moving target selection model to enhance single-target capture and predict multi-target selection.

[0149] Example 2

[0150] This embodiment discloses an interactive training system based on eye-foot coordination, such as Figure 3 Shown, including:

[0151] Projection equipment, host computer, location tracking equipment and interactive area;

[0152] Projection equipment: used to present the content and interactive feedback of the interactive training system on the ground;

[0153] Host: used to run the interactive training system program;

[0154] Location tracking device: used to capture the user's real-time location and enable interaction between the user and the interactive area.

[0155] Interactive area: used to display the system interface and is the main training implementation area for users;

[0156] The projection device and position tracking device are connected to the host computer. The position tracking device captures the user's body joint information and transmits the captured data to the host computer. The host computer uses the user's joint information to determine the user's posture. Different postures represent different interactive operations, such as raising a hand to indicate user login. The training system invokes different functions based on different user postures, using the user's position information to interact with the training system's moving targets.

[0157] As one or more embodiments, the method of using the user's location information to interact with the moving target of the training system includes the following specific steps:

[0158] The position tracking device captures the user's SPINE_NAVAL joint information and transmits the captured data to the host;

[0159] The host converts the user's SPINE_NAVAL joint point information into two-dimensional coordinates of the interactive interface and uses the coordinates to interact with the interactive interface elements. The position tracking device, such as an RGB-D camera, is an Azure Kinect DK device (hereinafter referred to as Kinect) used in this disclosure.

[0160] Specifically, the host is configured to include:

[0161] Face recognition module, feature selection module, training module, reaction test module and ranking module.

[0162] It should be noted that, in this embodiment, in order to achieve the reaction training effect, the user needs to move quickly to the moving target, so the number of moving targets is 1.

[0163] like Figure 4 As shown, the flow chart of this embodiment:

[0164] S401: Face recognition module: This module is the same as the face recognition method in the first embodiment, and will not be described in detail.

[0165] S402: Function selection module. User-selectable functions include starting training, reaction test, and viewing leaderboards.

[0166] The user selects a function by moving their body to the corresponding function button. The specific judgment method is as follows:

[0167] (1) The 2D coordinates (user_x, user_y) of the user's pelvis (SPINE_NAVAL) joint in the interactive interface are estimated using the Kinect device. The coordinate conversion method is the same as that described in Example 1.

[0168] (2) Set the upper left corner of the button to the (left, top) position in the interactive interface, the button width to btn_width, and the button height to btn_height.

[0169] (3) The conditions for selecting a button are:

[0170] left <user_x<left+btn_width&&top<user_y<top+btn_height

[0171] S403: Determine whether the user chooses to start training. If the user chooses to start training, jump to S404; otherwise, jump to S405.

[0172] S404: Training module: The training method is the same as the single moving target training mode in the first embodiment, and will not be described in detail.

[0173] S405: Determine whether the user selects the reaction test function. If the user selects the reaction test function, jump to S406; otherwise, jump to S408.

[0174] S406: Reaction test module, used to test the user's simple reaction time.

[0175] Among them, the reaction test module mainly tests the user's simple reaction time.

[0176] like Figure 5 As shown in the figure, the interactive interface has three rectangular blocks arranged one above the other. The lower block is the preparation block, the middle block is the test block, and the upper block is the time block. During the test, the user is asked to stand on the preparation block. Once the user stands on the preparation block, at a random time in the next 10 seconds, the test block will turn from red to green. After observing the color change, the user must quickly move to the test block and record the time required for this process, which is reflected in the time block. A reaction test requires the user to complete the above steps five times, and the average test time is taken as the test result.

[0177] S407: Determine whether the user chooses to start training.

[0178] In the reaction test interface, there is a start training button embedded, and the user can start training by moving the body to the button. If the user chooses to start training, the process jumps to S404, otherwise, the process jumps to S408.

[0179] S408: Ranking module.

[0180] This module shows the highest single training records of all users.

[0181] S409: Determine whether the user chooses to start training.

[0182] In the ranking interface, there is a start training button embedded, and the user can start training by moving the body to the button. If the user chooses to start training, the process jumps to S404, otherwise, the process jumps to S411.

[0183] S410: Determine whether to end the training.

[0184] The end mode is the same as the end mode of training in the embodiment, and will not be described in detail. If the training is finished, jump to S411, otherwise jump to S404.

[0185] S411: End of training.

[0186] Training effect see attached Figure 6 shown.

[0187] Example 3

[0188] This embodiment provides an interactive learning system that supports eye-foot coordination, including:

[0189] Face recognition module, single-player learning module, two-player competition module, and two-player cooperation module.

[0190] It should be noted that, in this embodiment, in order to meet learning requirements, the questions are set as multiple-choice questions, so the number of moving targets is 4.

[0191] like Figure 7 As shown, the flow chart of this embodiment:

[0192] S701: Face recognition module.

[0193] This module uses the same face recognition method as in Example 1, so I won't go into detail here. It should be noted that this example allows up to two users to log in. The specific recognition method is:

[0194] (1) User logs in;

[0195] (2) The system enters a 10-second countdown phase and waits for user 2 to log in;

[0196] (3) The countdown ends and the learning module begins.

[0197] S702: Determine the number of logged-in users.

[0198] According to the face recognition result, the number of users is determined. If the number of users is 1, jump to S703; otherwise, jump to S704.

[0199] S703: Single-study module.

[0200] The title of this module is a multiple-choice question, and the interactive interface includes an answer moving area, a question display box and an information prompt bar. Four moving targets of random size and random speed (the random variation range is defined in Example 1) are generated at random positions in the answer moving area. When the target moves to the boundary of the interactive interface, it will collide and rebound, and its rebound method is the same as the principle of light refraction. The way the user interacts with the moving target is the same as the interaction method in Example 1, so I will not go into details here. When the user chooses the correct answer, the user's points will be increased by 1, otherwise, the user's points will be reduced by 1, and the change in the score will be displayed in real time in the information prompt bar.

[0201] S704: Double function selection module. Figure 8 As shown in the figure, in this module, four buttons are designed, and users control the buttons on both sides respectively. If and only if two users select the button with the same function, will the next module be jumped to. For example, only if two users select the competition button will the next module be jumped to.

[0202] S705: Determine the choices of the two users.

[0203] The user's selection intention is judged based on the positions of the two users. When the user selects the competition mode, the process jumps to S706. When the user selects the cooperation mode, the process jumps to S707. Otherwise, the process waits for the user to make a correct selection.

[0204] S706: Two-player competition module.

[0205] The interactive interface of this module is essentially the same as that in S703, with the difference being that the score information bar now displays information about two users. It's important to note that in this module, two users share a single answering opportunity. The answer is determined by the user who first captures the moving target. If the user's answer is correct, their score increases by 1; otherwise, their score decreases by 1. If the user fails to get the answer, their score remains unchanged. The scores of both users are displayed in real time in the information bar.

[0206] S707: Two-player cooperation module.

[0207] The interface of this module is essentially the same as that in S706, except that it features multiple-choice questions. Only when both users choose the correct answer will their scores increase. If either user chooses the wrong answer, both users' scores decrease by 1. It's important to note that each user only has one chance to choose, requiring collaboration between the two users to complete the question. The scores of both users are displayed in real time in the notification bar.

[0208] S708: Determine whether the process is finished.

[0209] In the single-player mode, the termination determination method is the same as that in the first embodiment, and will not be described in detail here. In the two-player mode, unlike the single-player mode, active exit requires both users to make an exit gesture at the same time, and passive exit is the same as that described in the first embodiment.

[0210] S709: Exit learning.

[0211] Learning effect see attached Figure 9 shown.

[0212] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0213] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. The interactive training method based on eye-foot coordination is characterized by: The user can control the system function through body posture and interact with the training system through body movement status, including: The user's body joint information is captured by a position tracking device, and the user's posture is determined based on the user's joint information; different postures represent different interactive operations; In the training phase, the user to be trained interacts with the target elements moving in the interactive interface until the training is completed; During training, the system provides users with moving targets of different sizes and speeds based on their foot lengths and training records. The method for generating the mobile target element in the interactive interface is as follows: estimating the user's height and the user's foot length based on the captured user's color image information; Estimate the pixel length of the user's foot in the interactive interface based on the estimated length of the user's foot; The moving target elements in the interactive interface are all circular targets; The diameter range of the circular moving target is determined according to the length of the middle foot in the interactive interface.

2. The interactive training method based on eye-foot coordination as claimed in claim 1, wherein: Before training, the user to be trained first captures the user's color image information and identifies the user's registration information; If the user information to be trained is the information captured and recognized for the first time, the user's registration information is received and the user image is stored in the user database; otherwise, the user registration information is directly retrieved from the database; After successful registration or login, the user registration information will be displayed in the interactive interface.

3. The interactive training method based on eye-foot coordination as claimed in claim 1, wherein: The specific method of capturing the user's color image information is as follows: Obtaining user color image and user depth image information; Estimate the 3D coordinate information of the user's nose and neck joints through the captured images; Convert the obtained 3D coordinates into 2D coordinates based on the color image; The user's face rectangular image is cropped using the obtained 2D coordinates.

4. The interactive training method based on eye-foot coordination as claimed in claim 1, wherein: The movement speed of the moving target in the interactive interface increases with the training time.

5. The interactive training method based on eye-foot coordination as claimed in claim 1, characterized in that: The specific steps for users to interact with the mobile target element in the interactive interface are: Generate interactive interface; Get the user's 3D position coordinates; Convert 3D coordinates to 2D coordinates based on a plane; Convert the position coordinates based on the plane coordinate system into the position coordinates based on the velocity coordinate system; If it is a single target mode, determine whether the landing point is within the 2σ confidence interval of the three-Gaussian model of the target landing point distribution. If the landing point is within this range, it means that the user has captured the moving target and the capture is successful; If it is a multi-target capture mode, traverse all the moving targets in the interaction area and determine whether the landing point is within the 2σ confidence interval of the three-Gaussian model of the moving target landing point distribution. If so, find the target that best meets the user's selection intention among all the moving targets that meet the conditions.

6. The interactive training method based on eye-foot coordination as claimed in claim 5, characterized in that: Among all eligible mobile targets, find the one that best meets the user's preferences, specifically: For all moving targets in the interaction area, the landing point coordinates are input into the Bayesian decision formula for all targets that meet the conditions, and the probability of the landing point being located in each target is obtained; all conditions that meet the conditions are that the landing point coordinates are within the 2σ confidence interval of the landing point distribution model; The target with the highest calculated probability is selected as the final target.

7. The interactive training method based on eye-foot coordination as claimed in claim 1, characterized in that: Users can choose to actively exit training and passively exit training, including: Determine whether the user has made an end sign gesture, or whether the training time has been reached. If one of the above conditions is met, the training is exited.

8. An interactive training system based on eye-foot coordination, characterized by: include: The user posture judgment module is configured to: capture the user's body joint information and judge the user's posture based on the user's joint information; Different postures represent different interactive operations; The training module is configured as follows: in the training phase, the user to be trained is trained by interacting with the target element moving in the interactive interface until the training is completed; During training, the system provides users with moving targets of different sizes and speeds based on their foot lengths and training records. The method for generating the mobile target element in the interactive interface is as follows: estimating the user's height and the user's foot length based on the captured user's color image information; Estimate the pixel length of the user's foot in the interactive interface based on the estimated length of the user's foot; The moving target elements in the interactive interface are all circular targets; The diameter range of the circular moving target is determined according to the length of the middle foot in the interactive interface.

9. An interactive training device based on eye-foot coordination, characterized in that: include: Projection equipment, host computer and location tracking equipment; The projection device is used to project the content of the interactive training and interactive feedback information run by the host onto the ground area; The position tracking device captures the user's body joint information and transmits the captured data to a host, and the host is configured to execute the method according to any one of claims 1 to 7.

Images