Finger multi-perception feedback training method and system

Abstract

The invention discloses a finger multi-perception feedback training method and system, and the method comprises the steps: S1, responding to an input instruction, calling scene data, and transmitting the scene data to a display module; s2, when the called scene data is static target data, obtaining finger contact data of the trainee, and comparing the finger contact data of the trainee with the static target data to obtain a comparison result; s3, when the called scene data is activity target data, obtaining activity data of an activity target, and comparing the activity data of the activity target with the training rule data to obtain a comparison result; and S5, sending different audio information and prompt information according to different comparison results. According to the finger multi-perception feedback training system, the man-machine interaction performance, the training experience feeling and the training effect of an existing finger multi-perception feedback training system can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human hand function training, and in particular to a finger multi-sensory feedback training method and system. Background Art

[0002] The application of human-computer interaction technology in the field of rehabilitation medicine is of great significance. By creating an interactive rehabilitation training system, combined with sensors, virtual reality and augmented reality technologies, a personalized and immersive training experience can be provided for trainees. Designing an intuitive user interface and timely feedback mechanisms such as music can improve trainees' participation and training effects, while motivating trainees to actively participate through gamification elements. The development of interactive software should support multimodal interaction and adaptive learning algorithms to dynamically adjust training content to meet the needs of different trainees. In addition, data analysis functions can generate detailed training reports to help doctors and trainees understand rehabilitation progress and optimize treatment plans. In summary, the introduction of human-computer interaction technology into rehabilitation medicine can not only improve the effect of rehabilitation training, but also promote the physical and mental health of trainees, improve their psychological state, and enhance social interaction, and has broad application prospects.

[0003] However, the interactive rehabilitation training systems in the prior art, especially the finger multi-sensory feedback training systems, often only require trainees to train according to the movements on the screen, and the trainees' specific finger function training movements cannot be fed back to the screen or wearable display devices. The human-computer interaction is limited, the trainees' experience is poor, or only the trainees' favorite music is played in reality, resulting in poor training effects. Summary of the invention

[0004] The present invention provides a finger multi-sensory feedback training method and system that effectively improves human-computer interactivity, finger training experience and training effect, which can solve at least one of the above-mentioned technical problems.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A finger multi-sensory feedback training method comprises the following steps:

[0007] S1. Respond to input instructions, call scene data and send it to the display module, the scene data includes static target data or active target data;

[0008] S2. When the called scene data is static target data, the trainee's finger contact information is obtained, and the trainee's finger contact data is obtained by analysis and processing, and the trainee's finger contact data is compared with the static target data to obtain a comparison result;

[0009] S3, when the called scene data is activity target data, the scene data also includes training rule data of the activity target, a control instruction for the activity target is generated according to the scene data and the training rule data, the control instruction is transmitted to the display module, the activity of the activity target is controlled by the control instruction, and the finger contact information of the trainee is obtained, and the activity data of the activity target is obtained by analysis and processing, and the activity data of the activity target is compared with the training rule data to obtain a comparison result;

[0010] S5. Send different audio information to the audio module and / or send different prompt information to the display module according to different comparison results.

[0011] Furthermore, in S3, the training data stored in the storage module is called in sequence according to the pre-set training rule data, and sent to the audio module.

[0012] Furthermore, in S3, the step of generating a control instruction for the active target according to the scene data and the training rule data further includes:

[0013] S3.1. Satisfy the following formula (1) to generate the first instruction:

[0014] F bf >K1, F if +F mf +F rf +F lf +F xp +F yp <E1 (1);

[0015] S3.2, satisfying the following formula (2), generating the second instruction:

[0016] F if >K2, F bf +F mf +F rf +F lf +F xp +F yp <E2 (2);

[0017] S3.3. Satisfy the following formula (3) to generate the third instruction:

[0018] F mf >K3, F bf +F if +F rf +F lf +F xp +F yp <E3 (3);

[0019] S3.4. Satisfy the following formula (4) to generate the fourth instruction:

[0020] F rf >K4, F bf +F mf +F if +F lf +F xp +F yp <E4 (4);

[0021] S3.5. Satisfy the following formula (5) to generate the fifth instruction:

[0022] F lf >K5, F bf +F mf +F rf +F if +F xp +F yp <E5 (5);

[0023] S3.6. Satisfy the following formula (6) to generate the sixth instruction:

[0024] F xp >K6, F bf +F mf +F rf +F lf +F if +F yp <E6 (6);

[0025] S3.7. Satisfy the following formula (7) to generate the seventh instruction:

[0026] F yp >K7, F bf +F mf +F rf +F lf +F if +F xp <E7 (7);

[0027] S3.8, when the activity target stop generation condition is met, stop the current instruction;

[0028] in:

[0029] K i ≥m,E i ≤n,K i 、E i is the threshold of touch force, i is any number from 1 to 7, and m and n are thresholds set by the trainees;

[0030] F bf 、F if 、F mf 、F rf 、F lfare the current values ​​of the touch force of the thumb area, index finger area, middle finger area, ring finger area, and little finger area on either side of the hand, respectively. xp is the current value of the touch force in the outer palm area or the inner palm area, F yp The current value of the touch force of the inner palm area or the outer palm area.

[0031] Furthermore, in S3, the step of generating a control instruction for the active target according to the scene data and the training rule data further includes:

[0032] S3.9, satisfying the following formula (8) to generate the first instruction:

[0033] F bf >K1, F if +F mf +F rf +F lf <E1 (8);

[0034] S3.10, satisfying the following formula (9), generating the second instruction:

[0035] F if >K2, F bf +F mf +F rf +F lf <E2 (9);

[0036] S3.11. Satisfy the following formula (10) to generate the third instruction:

[0037] F mf >K3, F bf +F if +F rf +F lf <E3 (10);

[0038] S3.12, satisfying the following formula (11), generating the fourth instruction:

[0039] F rf >K4, F bf +F mf +F if +F lf <E4 (11);

[0040] S3.13, satisfying the following formula (12), generating the fifth instruction:

[0041] F lf >K5, F bf +F mf +F rf +F if <E5 (12);

[0042] S3.14, satisfying the following formula (13), generating the sixth instruction:

[0043] F bf +F pf >K6, F mf +F rf +F lf <E6 (13);

[0044] S3.15. Satisfy the following formula (14) to generate the seventh instruction:

[0045] F bf +F qf >K7, F if +F mf +F lf <E7 (14);

[0046] S3.16, when the activity target stop generation condition is met, stop the current instruction;

[0047] in:

[0048] K i ≥x1,K j ≥x2,E i ≤y1,E j ≤y2, i is any number from 1 to 5, j is any number from 6 to 7, x1, x2, y1, y2 are thresholds set by the trainee;

[0049] F bf 、F if 、F mf 、F rf 、F lf are the current values ​​of the touch force of the thumb area, index finger area, middle finger area, ring finger area and little finger area on either side of the hand, respectively. pf To divide F bf 、F qf The current value of the touch force of any area outside qf To divide F bf 、F pf The current value of the touch force in any area outside the touch zone.

[0050] Furthermore, in S3, the step of generating a control instruction for the active target according to the scene data and the training rule data further includes:

[0051] S3.17, satisfying the following formula (15), generating the first instruction:

[0052] F bf >Q1,F if +F mf +F rf +Flf +F xp +F yp <G1 (15);

[0053] S3.18, satisfying the following formula (16), generating the second instruction:

[0054] F x >Q2, F bf +F if +F mf +F rf +F xp +F yp <G2 (16);

[0055] S3.19, when the activity target stop generation condition is met, stop the current instruction;

[0056] in:

[0057] Q i ≥j,G i ≤k,Q i , G i is the threshold of touch force, i is any number from 1 to 2, j and k are thresholds set by the trainees;

[0058] F bf and F x They are respectively the current value of the touch force of the thumb area on the other side of the hand and the current value of the touch force of any one of the index finger area, middle finger area, ring finger area, and little finger area.

[0059] Furthermore, the relaxation training data in the storage module is called and sent to the audio module.

[0060] Furthermore, the storage module stores data of a training mode, and the training mode has multiple training dimensions, and the training dimensions include at least one or more of a training scenario, a difficulty gradient, a training action combination, and a duration.

[0061] A finger multi-sensory feedback training system, applicable to the finger multi-sensory feedback training method, comprising:

[0062] A finger touch sensing module, used for detecting finger contact information of the trainee;

[0063] A storage module storing scene data and game data;

[0064] A display module, connected to the storage module, for displaying a scene image;

[0065] An audio module, connected to the storage module, for dubbing the scene or adding background music;

[0066] A data processing module is connected to the finger touch sensing module and the storage module, and is used to receive and analyze finger contact information in real time to obtain the trainee's training data, and store the training data in the storage module. The data processing module is also connected to the display module, and is used to call the training data and / or scene data and / or game data in the storage module, and generate control instructions to send to the display module.

[0067] Furthermore, the finger touch sensing module includes a thumb area sensor, an index finger area sensor, a middle finger area sensor, a ring finger area sensor and a little finger area sensor arranged on the fingers, or includes an outer palm area sensor and an inner palm area sensor arranged on the palm.

[0068] Furthermore, it also includes an input module, which is connected to the data processing module and is used for the trainee's input instructions, and the input instructions at least include one or more of the selection of training mode, simple music score and background music.

[0069] The beneficial effects of the present invention are embodied in:

[0070] 1. Divide finger touch perception into five areas, namely thumb area, index finger area, middle finger area, ring finger area and little finger area, to clearly obtain the force information of different areas of each finger. The same force information can be better compared. Through the connection between force information of different fingers, more human body movements can be identified, and better and more specific training strategies can be formulated to enable trainees to perform better finger training.

[0071] 2. The finger contact information of the trainees is detected through the finger touch perception module, so that each movement of the trainees can be acquired by the finger multi-sensory feedback training system. After being processed by the data processing module, it can be displayed in real time on the display module. Combined with sensors, virtual reality and augmented reality technology, it can provide trainees with a more personalized and immersive training experience.

[0072] 3. Designing an intuitive user interface and timely feedback mechanism can improve trainees' participation and training effects. At the same time, gamification elements can motivate trainees to actively participate, reduce trainees' resistance to rehabilitation training, and enhance recovery effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0074] Figure 1 It is a schematic diagram of the structure of a finger multi-sensory feedback training system according to an embodiment of the present invention.

[0075] Figure 2 It is a flowchart of one of the methods of finger multi-sensory feedback training in an embodiment of the present invention.

[0076] Figure 3 It is another flowchart of a method for finger multi-sensory feedback training according to an embodiment of the present invention.

[0077] Figure 4 It is a schematic diagram of the operation flow of the finger multi-sensory feedback training system according to an embodiment of the present invention.

[0078] Figure 5 It is a structural block diagram of human-computer interaction of a finger multi-sensory feedback training system according to an embodiment of the present invention.

[0079] Figure 6 Schematic diagram of finger touch area distribution according to an embodiment of the present invention.

[0080] Figure 7 It is a structural block diagram of a computer device according to an embodiment of the present invention.

[0081] The components in the accompanying drawings are marked as follows: 1. finger touch sensing module; 2. display module; 3. audio module; 4. storage module; 5. data processing module; 6. input module. DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. In the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0083] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or schemes that A and B meet at the same time. In addition, "multiple" refers to more than two. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0084] See also Figure 1 The embodiment of the present invention provides a finger multi-sensory feedback training system, including:

[0085] The finger touch sensing module 1 is used to detect the finger contact information of the trainee;

[0086] Display module 2, connected to storage module 4, for displaying scene images;

[0087] The audio module 3 is connected to the storage module 4 and is used to dub the scene or add background music;

[0088] Storage module 4, storing scene data and game data;

[0089] A data processing module 5, connected to the finger touch sensing module 1 and the storage module 4, for receiving and analyzing finger contact information in real time to obtain the trainee's training data, and storing the training data in the storage module 4. The data processing module 5 is also connected to the display module 2, for calling the training data and / or scene data and / or game data in the storage module 4, and generating a control instruction to send to the display module 2;

[0090] An input module 6, which is connected to the data processing module 5 and is used for inputting instructions from the trainee, wherein the input instructions at least include selecting one or more of a training mode, a simple musical score and background music.

[0091] This finger multi-sensory feedback training system combines the feedback of multiple senses such as vision, touch, and hearing to perform movement imagination in consciousness, thereby improving the effect of training and rehabilitation. Among them, the combination of music and exercise therapy can significantly improve the rehabilitation of the nerves and motor functions of stroke patients and reduce the negative emotions of patients during training. At the same time, the training scenes, games, music and human hand rehabilitation training are combined to achieve interesting rehabilitation training. On the one hand, it can improve the interests and hobbies of the trainees. On the other hand, since in the game, the consciousness and body of the person can reach a special state of separation of body and mind, the combination of game elements can relax the body and mind of the trainees, which is beneficial to the rehabilitation training of the trainees, thereby improving the rehabilitation effect of the trainees' hand functions.

[0092] In this embodiment, the finger touch sensing module 1 includes a thumb area sensor, an index finger area sensor, a middle finger area sensor, a ring finger area sensor and a little finger area sensor arranged on the fingers, or includes an outer palm area sensor and an inner palm area sensor arranged on the palm.

[0093] In the present application, the finger touch sensing module 1, the display module 2, the storage module 4 and the data processing module 5 are all hardware devices. Among them, the finger touch sensing module 1 is selected as a finger touch measurement board or a finger touch measurement glove. The finger touch measurement board can be a wearable structure or a non-wearable structure. When the finger touch measurement board is a wearable structure, the trainee needs to fix his wrist on the device; the finger touch measurement board or the finger touch measurement glove is provided with a thumb area sensor, an index finger area sensor, a middle finger area sensor, a ring finger area sensor and a little finger area sensor located at the finger part. The storage module 4 is selected as a memory, and the scene data and training data in the memory are all in the form of interactive software. The input module 6 is selected as a keyboard, a mouse or a control button or a switch or a touch screen input screen. The data processing module 5 is selected as a host computer or a processor, such as a computer. The display module 2 is selected as a VR smart eye mask or a display.

[0094] In this embodiment, the scene data stored in the storage module 4 includes background music data, training guidance data, error correction prompt sound data, game scene special effect audio data, etc. that are adapted to the corresponding scene.

[0095] In this embodiment, the training data processed by the data processing module 5 includes finger contact data and score data, the finger contact data includes the thumb contact state, index finger contact state, middle finger contact state, ring finger contact state, little finger contact state, outer palm contact state and inner palm contact state; the score data includes the game score state.

[0096] See also Figure 4This finger multi-sensory feedback training system specifically designs training scenes and interactive games based on finger contact information, combines training scenes and games with human hand function rehabilitation training theories (such as motor imagery therapy theory and music therapy theory), and selects training modes based on trainees' input instructions. The training modes include normal training mode and competitive training mode. Both normal training mode and competitive training mode are carried out in a gamified way, such as fingertip piano game and fingertip encouragement game.

[0097] Take the fingertip piano game as an example: when the normal training mode is selected, the corresponding sensor area is tapped in sequence according to the music score, and the background music makes the trainees more involved in the training; when the speed training mode is selected, the corresponding piano blocks are generated in sequence according to the selected music, and the trainees tap the corresponding finger sensor area in sequence to make the piano blocks disappear, and the background music makes the game more rhythmic.

[0098] During the game, the data processing module 5 generates various interactive control instructions through finger contact data information. In the interactive scenes and games, the general training mode and the racing training mode are integrated with the theory of motor imagery therapy and music therapy. The data obtained by each sensor set in the finger touch perception module 1 and the score are visualized to achieve the output of the interactive results.

[0099] At the same time, the storage module 4 stores the program or data of the guiding unit, which is used to guide the trainee to perform actions according to the audio or text guidance displayed on the display module 2, such as guiding the trainee to perform corresponding hand function training through the music score and background music in the ordinary training mode. For example, if the trainee selects the racing training mode, the display module 2 will display racing-related patterns or audio. When the trainee performs corresponding actions according to the prompt information, the finger touch module 1 detects the trainee's finger contact information, and after being processed by the data processing module 5, a control instruction is issued to the display module 2 to control the activity of the activity target.

[0100] See also Figure 2 The embodiment of the present invention further provides a finger multi-sensory feedback training method, which is implemented by using the above-mentioned finger multi-sensory feedback training system, and includes the following steps:

[0101] S1, the input module 6 responds to the input instruction and sends the input instruction to the data processing module 5, the data processing module 5 calls the scene data in the storage module 4 and sends it to the display module 2, the scene data includes static target data, the static target data includes piano timbre, and the static target is a piano key;

[0102] S2, acquiring the trainee's finger contact information through the finger touch sensing module 1, and analyzing and processing it through the data processing module 5 to obtain the trainee's finger contact data, wherein the finger contact data is common training data;

[0103] S3, the data processing module 5 compares the trainee's finger contact data with the static target data to obtain a comparison result. If the trainee's touch sensor area is inconsistent with the syllables on the musical notation, it is a partial action error or a partial action is not standard;

[0104] S4, according to different comparison results, different audio information is sent to the audio module 3, and the audio information is reminded by playing the wrong timbre. If the hand training movement made by the trainee is consistent with the corresponding movement of the training score, the data processing module 5 sends the piano key audio of the corresponding timbre to the audio module 3;

[0105] S5. According to different comparison results, different prompt information is sent to the display module 2. The prompt information is reminded in the form of markings on the music score. If the hand training movements made by the trainee are consistent with the corresponding movements in the training score, the data processing module 5 sends the corresponding animation to the display module 2.

[0106] See also Figure 3 The embodiment of the present invention also provides another finger multi-sensory feedback training method, which is implemented by using the above-mentioned finger multi-sensory feedback training system, and includes the following steps:

[0107] S10, the input module 6 responds to the input instruction and sends the input instruction to the data processing module 5, the data processing module 5 calls the scene data and sends it to the display module 2, the scene data includes the activity target data and the training rule data of the activity target, the activity target data includes the piano block timbre, the activity target is the falling piano block or other blocks, the training rule data of the activity target is the score obtained when the target cannot touch the set virtual line when it is active and the activity target disappears;

[0108] S20, the data processing module 5 generates a control instruction for the activity target according to the scene data and the training data, and transmits the control instruction to the display module 2, and controls the activity of the activity target through the control instruction;

[0109] The changes of the trainee's hand gestures can be detected by the finger touch sensing module 1 and transmitted to the data processing module 5 to generate control commands to control the activity of the activity target;

[0110] S30, acquiring the trainee's finger contact information through the finger touch sensing module 1, and obtaining the activity data of the activity target through analysis and processing by the data processing module 5, wherein the activity data of the activity target includes data such as the falling of the activity target, the contact of the activity target with or away from the set virtual line, or other series of action data;

[0111] S40, the data processing module 5 compares the activity data of the activity target with the training rule data to obtain a comparison result;

[0112] S50, the data processing module 5 sends corresponding audio information to the audio module 3 according to different comparison results;

[0113] S60: The data processing module 5 sends corresponding prompt information to the display module 2 according to different comparison results.

[0114] In the above method, the prompt information is used as a reminder in the form of text or background music. For example, if the activity target is a piano block, the trainee performs different finger contact actions to make the piano block disappear, and according to the rule data, such as the piano block cannot touch the set virtual line. When the piano block touches the set virtual line, the activity data of the activity target is inconsistent with the training rule data, and the data processing module 5 sends a corresponding prompt information to the display module 2 to prompt the end of this training; when the piano block does not touch the set virtual line and disappears smoothly, the activity data of the activity target is consistent with the training rule data, and the data processing module 5 sends a corresponding prompt information to the display module 2 to give the trainee a score reward.

[0115] See also Figure 4 In the above two methods, after the trainee selects the training mode and / or the racing mode, the data processing module 5 sends a prompt message to the display module 2 to remind the trainee to select a favorite background music to prepare for relaxation. The specific time for relaxation preparation can be selected according to the trainee's training performance;

[0116] The step of generating a control instruction for an activity target according to the scene data and the training rule data further includes:

[0117] S3.1. Satisfy the following formula (1) to generate the first instruction:

[0118] F bf >K1, F if +F mf +F rf +F lf +F xp +F yp <E1 (1);

[0119] or:

[0120] F bf >K1, F if +F mf +F rf +F lf <E1

[0121] For example, according to different training modes, the piano generates the timbre of "do" or the command of hitting the snare drum. For example, when the trainee selects the fingertip piano game in the game mode, the auxiliary side will change the piano timbre to a bass timbre when the formula is satisfied, and when the training side satisfies the formula, the data processing module 5 will generate the timbre of "do" emitted by the piano; if the trainee selects the fingertip drum game in the game mode, the data processing module 5 will generate the command of hitting the snare drum;

[0122] S3.2, satisfying the following formula (2), generating the second instruction:

[0123] F if >K2, F bf +F mf +F rf +F lf +F xp +F yp <E2 (2);

[0124] or:

[0125] F if >K2, F bf +F mf +F rf +F lf <E2

[0126] For example, according to different training modes, the timbre of the piano making r or the instruction of hitting the middle drum can be generated. For example, when the trainee selects the fingertip piano game in the game mode, the training side will generate the timbre of the piano making r when the formula is satisfied; if the trainee selects the fingertip drum game in the game mode, the data processing module 5 will generate the instruction of hitting the middle drum;

[0127] S3.3. Satisfy the following formula (3) to generate the third instruction:

[0128] F mf >K3, F bf +F if +F rf +F lf +F xp +F yp <E3 (3);

[0129] or:

[0130] F mf >K3, Fbf +F if +F rf +F lf <E3

[0131] For example, according to different training modes, the piano can generate the timbre of mi or the instruction of hitting the snare drum. For example, when the trainee selects the fingertip piano game in the game mode, the training side will generate the timbre of mi produced by the piano when the formula is satisfied; if the trainee selects the fingertip drum game in the game mode, the data processing module 5 will generate the instruction of hitting the snare drum;

[0132] S3.4. Satisfy the following formula (4) to generate the fourth instruction:

[0133] F rf >K4, F bf +F mf +F if +F lf +F xp +F yp <E4 (4);

[0134] or:

[0135] F rf >K4, F bf +F mf +F if +F lf <E4

[0136] For example, according to different training modes, the piano can generate the timbre of "fa" or the instruction of hitting the floor drum. For example, when the trainee selects the fingertip piano game in the game mode, the data processing module 5 will generate the timbre of "fa" produced by the piano when the formula is satisfied; if the trainee selects the fingertip drum game in the game mode, the data processing module 5 will generate the instruction of hitting the floor drum;

[0137] S3.5. Satisfy the following formula (5) to generate the fifth instruction:

[0138] F lf >K5, F bf +F mf +F rf +F if +F xp +F yp <E5 (5);

[0139] or:

[0140] F lf >K5, F bf +F mf +F rf +F if <E5

[0141] For example, according to different training modes, the piano can generate the timbre of sol or the striking and rubbing instructions. For example, when the trainee selects the fingertip piano game in the game mode, the auxiliary side will change the piano timbre to a high-pitched timbre when the formula is satisfied, and when the training side satisfies the formula, the data processing module 5 will generate the timbre of sol produced by the piano; if the trainee selects the fingertip agitation game in the game mode, the data processing module 5 will generate the striking and rubbing instructions;

[0142] S3.6. Satisfy the following formula (6) to generate the sixth instruction:

[0143] F xp >K6, F bf +F mf +F rf +F lf +F if +F yp <E6 (6);

[0144] or:

[0145] F bf +F pf >K6, F mf +F rf +F lf <E6

[0146] For example, according to different training modes, the piano can generate a timbre of la or a strike and rub instruction. For example, when the trainee selects the fingertip piano game in the game mode, the training side will generate the timbre of la from the piano when the formula is met; if the trainee selects the fingertip agitation game in the game mode, the data processing module 5 will generate a strike and rub instruction;

[0147] S3.7. Satisfy the following formula (7) to generate the seventh instruction:

[0148] F yp >K7, F bf +F mf +F rf +F lf +F xp +F if <E7 (7);

[0149] or:

[0150] F bf +F qf >K7, F if +F mf +F lf <E7

[0151] For example, when the fingertip piano game in the game mode is selected, when the training side satisfies the formula, the data processing module 5 will generate the timbre of the piano si;

[0152] S3.8, when the activity target stop generation condition is met, stop the current instruction;

[0153] in:

[0154] K i ≥m,E i ≤n,K i 、E i is the threshold of touch force, i is any number from 1 to 7, and m and n are thresholds that can be set by the trainee;

[0155] F bf 、F if 、F mf 、F rf 、F lf are respectively the current values ​​of the touch forces detected by the sensors of the thumb area, index finger area, middle finger area, ring finger area, and little finger area on either side of the hand;

[0156] F xp is the current value of the touch force detected by the outer palm area or inner palm area sensor, F yp is the current value of the touch force detected by the inner palm area or outer palm area sensor, F pf To divide F bf 、F qf The current value of the touch force of any area outside qf To divide F bf 、F pf The current value of the touch force in any area outside the touch zone.

[0157] See also Figure 6 By dividing the human hand into thumb area, index finger area, middle finger area, ring finger area, little finger area, outer palm area and inner palm area, a more appropriate specific training strategy can be formulated, and trainees can perform better hand function training.

[0158] In the above two method embodiments, the storage module 4 stores data of the training mode, and the training mode has multiple training dimensions, and the training dimensions at least include one or more of training scenes, difficulty gradients, training action combinations, and durations. For example, by changing different training scenes, adding different difficulty gradients, and achieving corresponding training objectives through different training action components, the entire training content is enriched.

[0159] As one embodiment of this method, a user data recording unit and a training feedback unit are further stored in the storage module 4. Both the training feedback unit and the user data recording unit are connected to the finger touch sensing module 1. The training feedback unit is used to obtain a training result based on the human hand contact data and provide feedback. The training result provided by the training feedback unit includes the completion status of various training actions, a training score, and historical performance.

[0160] See Figure 5 , hereinafter, this application will be analyzed and explained in combination with specific case situations:

[0161] First, take the fingertip piano game as an example:

[0162] The trainee clicks on "Active Training" to enter the game selection interface. After selection, the training mode is chosen and the background music is played. After starting the training, making corresponding contact gestures can complete the control of different keys. The actions, from simple to difficult, are: single-finger contact, two-finger contact, and five-finger contact actions;

[0163] In the fingertip piano game, the model during scene modeling includes at least piano white keys, piano black keys, etc.;

[0164] The fingertip piano game includes the finger touch sensing module 1 obtaining the contact information of the trainee, and the data processing module 5 controlling the pressing of the corresponding keys. It also includes functions such as a timing and scoring function, selection of a simple music score, a combat drill area, as well as functions such as selection of a background music file, play / pause, volume adjustment, visualization of sensor data, and exiting the training function;

[0165] The original voltage data collected by the sensors in the finger touch sensing module 1 is calculated by the data processing module 5 to obtain the estimated value of each sensor. The force magnitude of each partition is the estimated value of the sensor in that partition, for example, one corresponding sensor is set for each partition;

[0166] When the sensor value on the training side satisfies the following formula, F bf >K1, F if +F mf +F rf +F lf +F xp +F yp <E1, or: F bf >K1, F if +F mf +F rf +F lf <E1 sets the piano to emit the tone of do;

[0167] When the sensor value on the training side satisfies the following formula, F if>K2, F bf +F mf +F rf +F lf +F xp +F yp <E2, or: F if >K2, F bf +F mf +F rf +F lf <E2 sets the piano to emit the tone of r;

[0168] When the value of the training side sensor satisfies the following formula, F mf >K3, F bf +F if +F rf +F lf +F xp +F yp <E3, or: F mf >K3, F bf +F if +F rf +F lf <E3 sets the piano to emit the tone of mi;

[0169] When the value of the training side sensor satisfies the following formula, F rf >K4, F bf +F mf +F if +F lf +F xp +F yp <E4, or: F rf >K4, F bf +F mf +F if +F lf <E4 sets the piano to emit the tone of fa;

[0170] When the value of the training side sensor satisfies the following formula, F lf >K5, F bf +F mf +F rf +F if +F xp +F yp <E5, or: F lf >K5, F bf +F mf +F rf +F if <E5 sets the piano to emit the tone of sol;

[0171] When the value of the training side sensor satisfies the following formula, F xp >K6, F bf+F mf +F rf +F lf +F if +F yp <E6, or: F bf +F pf >K6, F mf +F rf +F lf <E6, set the piano to emit the timbre of la;

[0172] When the value of the training side sensor satisfies the following formula, F yp >K7, F bf +F mf +F rf +F lf +F xp +F if <E7, or: F bf +F qf >K7, F if +F mf +F lf <E7, set the piano to emit the timbre of si;

[0173] When the value of the auxiliary side sensor satisfies the following formula, F bf >Q1, F if +F mf +F rf +F lf +F xp +F yp <G1, set the piano tone to the bass timbre;

[0174] When the value of the auxiliary side sensor satisfies the following formula, F x >Q2, F bf +F if +F mf +F rf +F xp +F yp <G2, set the piano tone to the treble timbre;

[0175] Where:

[0176] Q i ≥j, G i ≤k, Q i 、G i are the thresholds of the touch force, i is any number from 1 to 2, and j and k are thresholds that can be set by the trainee;

[0177] F bf and F xThe current value of the touch force detected by the sensor in the thumb area on the other side of each hand, and the current value of the touch force detected by the sensor in any one of the index finger area / middle finger area / ring finger area / little finger area;

[0178] When the activity target stop generation condition is met, stop the current instruction.

[0179] In this fingertip piano game, the timing score rule is that piano blocks start to be generated when entering the scene training. After a piano block successfully disappears, the score increases by 1. The longer the game duration, the higher the score, indicating that the hand training is more sufficient.

[0180] Second, taking the drumming fingertip game as an example:

[0181] In the drumming fingertip game, the trainee clicks on the drumming fingertip game to enter the drumming fingertip game training interface. Similar to the above fingertip piano game, the sensors set in the finger touch perception 1 are used to collect finger information to identify control signals, thereby striking different musical instruments.

[0182] In the drumming fingertip game, the model during scene modeling includes at least a snare drum, a tom-tom drum, a bass drum, a floor tom, an open hi-hat, a pedal hi-hat, corresponding sensor data reading, and task scoring functions. At the same time, it also has functions such as background music file selection, play / pause, volume adjustment, sensor data visualization, and exit training.

[0183] The rules of the drumming fingertip game are similar to the control method in the above fingertip piano game:

[0184] When the sensor value satisfies the following formula, F bf >K1, F if +F mf +F rf +F lf +F xp +F yp <E1, or: F bf >K1, F if +F mf +F rf +F lf <E1 is set to strike the snare drum;

[0185] When the sensor value satisfies the following formula, F if >K2, F bf +F mf +F rf +F lf +F xp +F yp <E2, or: F if >K2, F bf +F mf +F rf +Flf <E2 set to strike the tom-tom;

[0186] When the sensor value satisfies the following formula, F mf > K3, F bf + F if + F rf + F lf + F xp + F yp <E3, or: F mf > K3, F bf + F if + F rf + F lf <E3 set to strike the snare drum;

[0187] When the sensor value satisfies the following formula, F rf > K4, F bf + F mf + F if + F lf + F xp + F yp <E4, or: F rf > K4, F bf + F mf + F if + F lf <E4 set to strike the bass drum;

[0188] When the sensor value satisfies the following formula, F lf > K5, F bf + F mf + F rf + F if + F xp + F yp <E5, or: F lf > K5, F bf + F mf + F rf + F if <E5 set to strike the hi-hat;

[0189] When the sensor value satisfies the following formula, F xp > K6, F bf + F mf + F rf + F lf + F if + F yp <E6, or: F bf + F pf > K6, F mf + F rf + F lf <E6, set to strike the open hi-hat;

[0190] Among them, K i ≥m,E i ≤n,K i 、E i is the threshold of touch force, i is any number from 1 to 7, and m and n are thresholds that can be set by the trainee.

[0191] During training, the fingertip drum game provides drum sets with different timbres; specifically, different types of drum sets are composed of instruments of different materials, and corresponding timbres are emitted at the same time; the main scoring rule of the game is timed scoring, and the score increases by 1 every 30 seconds of training. The longer the game lasts, the higher the final score, and the more complete the hand training is considered to be.

[0192] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned finger multi-sensory feedback training method.

[0193] See also Figure 7 An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned finger multi-sensory feedback training method.

[0194] The embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the above-mentioned finger multi-sensory feedback training method.

[0195] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above-mentioned finger multi-sensory feedback training method.

[0196] It should be noted that those skilled in the art can understand that all or part of the steps implemented in the embodiments of the present invention can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using hardware, it can be implemented in whole or in part in the form of purchasing standard parts or modified parts. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0197] In summary, the present invention provides a finger multi-sensory feedback training system and method, which divides finger touch perception into five areas, namely the thumb area, the index finger area, the middle finger area, the ring finger area and the little finger area, so as to obtain force information of different areas of each finger, and the same force information can be better compared. Through the connection of force information between different fingers, more human body movements can be identified, and better and more specific training strategies can be formulated, so that trainees can perform better finger training;

[0198] At the same time, the finger contact information of the trainees is detected by the finger touch perception module, so that each movement of the trainees can be acquired by the finger multi-sensory feedback training system. After being processed by the data processing module, it can be displayed in real time on the display module. Combined with sensors, virtual reality and augmented reality technology, it can provide trainees with personalized and immersive training experience.

[0199] It should be understood that the examples and implementation modes described herein are for illustrative purposes only and are not intended to limit the present invention. Those skilled in the art may make various modifications or changes based on the examples and implementation modes. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A finger multi-sensory feedback training method, characterized in that: The following steps are involved: S1. Respond to input instructions, call scene data and send it to the display module, the scene data includes static target data or active target data; S2. When the called scene data is static target data, the trainee's finger contact information is obtained, and the trainee's finger contact data is obtained by analysis and processing, and the trainee's finger contact data is compared with the static target data to obtain a comparison result; S3, when the called scene data is activity target data, the scene data also includes training rule data of the activity target, a control instruction for the activity target is generated according to the scene data and the training rule data, the control instruction is transmitted to the display module, the activity of the activity target is controlled by the control instruction, and the finger contact information of the trainee is obtained, and the activity data of the activity target is obtained by analysis and processing, and the activity data of the activity target is compared with the training rule data to obtain a comparison result; S5. Send different audio information to the audio module and / or send different prompt information to the display module according to different comparison results.

2. The finger multi-sensory feedback training method according to claim 1, characterized in that: In S3, the training data stored in the storage module are called in sequence according to the pre-set training rule data, and sent to the audio module.

3. The finger multi-sensory feedback training method according to claim 1, characterized in that: In S3, the step of generating a control instruction for the active target according to the scene data and the training rule data further includes: S3.

1. Satisfy the following formula (1) to generate the first instruction: F bf >K1,F if +F mf +F rf +F lf +F xp +F yp <E1 (1); S3.2, satisfying the following formula (2), generating the second instruction: F if >K2,F bf +F mf +F rf +F lf +F xp +F yp <E2 (2); S3.

3. Satisfy the following formula (3) to generate the third instruction: F mf >K3,F bf +F if +F rf +F lf +F xp +F yp <E3 (3); S3.

4. Satisfy the following formula (4) to generate the fourth instruction: F rf >K4,F bf +F mf +F if +F lf +F xp +F yp <E4 (4); S3.

5. Satisfy the following formula (5) to generate the fifth instruction: F lf >K5,F bf +F mf +F rf +F if +F xp +F yp <E5 (5); S3.

6. Satisfy the following formula (6) to generate the sixth instruction: F xp >K6,F bf +F mf +F rf +F lf +F if +F yp <E6 (6); S3.

7. Satisfy the following formula (7) to generate the seventh instruction: F yp >K7,F bf +F mf +F rf +F lf +F if +F xp <E7 (7); S3.8, when the activity target stop generation condition is met, stop the current instruction; in: K i ≥m,E i ≤n,K i 、E i is the threshold of touch force, i is any number from 1 to 7, and m and n are thresholds set by the trainees; F bf 、F if 、F mf 、F rf 、F lf are the current values ​​of the touch force of the thumb area, index finger area, middle finger area, ring finger area, and little finger area on either side of the hand, respectively. xp is the current value of the touch force in the outer palm area or the inner palm area, F yp The current value of the touch force of the inner palm area or the outer palm area.

4. The finger multi-sensory feedback training method according to claim 1, characterized in that: In S3, the step of generating a control instruction for the active target according to the scene data and the training rule data further includes: S3.9, satisfying the following formula (8) to generate the first instruction: F bf >K1,F if +F mf +F rf +F lf <E1 (8); S3.10, satisfying the following formula (9), generating the second instruction: F if >K2,F bf +F mf +F rf +F lf <E2 (9); S3.

11. Satisfy the following formula (10) to generate the third instruction: F mf >K3,F bf +F if +F rf +F lf <E3 (10); S3.12, satisfying the following formula (11), generating the fourth instruction: F rf >K4,F bf +F mf +F if +F lf <E4 (11); S3.13, satisfying the following formula (12), generating the fifth instruction: F lf >K5,F bf +F mf +F rf +F if <E5 (12); S3.14, satisfying the following formula (13), generating the sixth instruction: F bf +F pf >K6,F mf +F rf +F lf <E6 (13); S3.

15. Satisfy the following formula (14) to generate the seventh instruction: F bf +F qf >K7,F if +F mf +F lf <E7 (14); S3.16, when the activity target stop generation condition is met, stop the current instruction; in: K i ≥x1,K j ≥x2,E i ≤y1,E j ≤y2, i is any number from 1 to 5, j is any number from 6 to 7, x1, x2, y1, y2 are thresholds set by the trainee; F bf 、F if 、F mf 、F rf 、F lf are the current values ​​of the touch force of the thumb area, index finger area, middle finger area, ring finger area and little finger area on either side of the hand, respectively. pf To divide F bf 、F qf The current value of the touch force of any area outside qf To divide F bf 、F pf The current value of the touch force in any area outside the touch zone.

5. The finger multi-sensory feedback training method according to claim 1, characterized in that: In S3, the step of generating a control instruction for the active target according to the scene data and the training rule data further includes: S3.17, satisfying the following formula (15), generating the first instruction: F bf >Q1,F if +F mf +F rf +F lf +F xp +F yp <G1 (15); S3.18, satisfying the following formula (16), generating the second instruction: F x >Q2, F bf +F if +F mf +F rf +F xp +F yp <G2 (16); S3.19, when the activity target stop generation condition is met, stop the current instruction; in: Q i ≥j,G i ≤k,Q i , G i is the threshold of touch force, i is any number from 1 to 2, j and k are thresholds set by the trainees; F bf and F x They are respectively the current value of the touch force of the thumb area on the other side of the hand and the current value of the touch force of any one of the index finger area, middle finger area, ring finger area, and little finger area.

6. The finger multi-sensory feedback training method according to claim 2, characterized in that: Recall the relaxation training data in the storage module and send it to the audio module.

7. The finger multi-sensory feedback training method according to claim 6, characterized in that: The storage module stores data of a training mode, where the training mode has multiple training dimensions, and the training dimensions include at least one or more of a training scenario, a difficulty gradient, a training action combination, and a duration.

8. A finger multi-sensory feedback training system, applicable to the finger multi-sensory feedback training method according to any one of claims 1 to 7, characterized in that: include: A finger touch sensing module, used for detecting finger contact information of the trainee; A storage module storing scene data and game data; A display module, connected to the storage module, for displaying a scene image; An audio module, connected to the storage module, for dubbing the scene images and / or adding background music; A data processing module is connected to the finger touch sensing module and the storage module, and is used to receive and analyze finger contact information in real time to obtain the trainee's training data, and store the training data in the storage module. The data processing module is also connected to the display module, and is used to call the training data and / or scene data and / or game data in the storage module, and generate control instructions to send to the display module.

9. The finger multi-sensory feedback training system according to claim 8, characterized in that: The finger touch sensing module includes a thumb area sensor, an index finger area sensor, a middle finger area sensor, a ring finger area sensor and a little finger area sensor arranged on the fingers, or also includes an outer palm area sensor and an inner palm area sensor arranged on the palm.

10. The finger multi-sensory feedback training system according to claim 8, characterized in that: It also includes an input module, which is connected to the data processing module and is used for the trainee's input instructions, and the input instructions at least include one or more of the selection of training mode, simple music score and background music.

Images