Motion training guidance method and device based on MR mixed reality

Abstract

The invention provides an action training guidance method and device based on MR mixed reality, and the method comprises the steps: projecting training elements of a simulation training scene to a real-time environment of a real region, and generating a mixed reality scene; at least according to the real-time movement track and posture change of the trainee, driving the virtual human substitute to execute real-time interaction of a digital virtual training tool and training elements based on a simulation training scene; generating an AI coach in the mixed reality scene; action demonstration and training guidance are performed based on real-time interaction of an AI coach to a virtual human. According to the method, a precondition is provided for real-time interaction of the AI coach on the digital virtual training tool and the training elements, meanwhile, the intelligent level of action training is improved, the guiding effect is guaranteed, and the use experience of the user is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of MR devices, and particularly to a method and device for action training guidance based on MR mixed reality. Background Art

[0002] Currently, the application of the existing MR mixed reality technology in action guidance usually projects pre-stored holographic animations or videos into the mixed reality scene. Users can observe the contour images of the holographic animations or videos in the mixed reality scene to judge the standard degree of their own actions.

[0003] However, when the holographic animation or video is played, it cannot interact in real time with the digital virtual training tools (such as rackets, baseball bats or poles, etc.) and training elements (such as runways, tennis nets, billiard tables or high jump poles, etc.) of the currently generated simulation training scene. The degree of intelligence is low, and there is also a lack of personalized action demonstrations and training guidance for the trainees. This results in poor guidance effects and also affects the user experience. Summary of the Invention

[0004] In view of the deficiencies of the prior art, the present invention proposes a method for action training guidance based on MR mixed reality, including:

[0005] Projecting the training elements of the simulation training scene into the real-time environment of the real area to generate a mixed reality scene;

[0006] At least according to the real-time movement trajectory and posture changes of the trainee, driving a virtual human substitute to perform real-time interaction based on the digital virtual training tools and training elements of the simulation training scene;

[0007] Generating an AI coach in the mixed reality scene;

[0008] Performing action demonstration and training guidance on the real-time interaction of the virtual human based on the AI coach.

[0009] In one embodiment, the step of generating an AI coach in the mixed reality scene described above includes:

[0010] Obtaining the origin coordinates of the trainee at the start of training;

[0011] Based on the origin coordinates of the trainee and the training content, determining the initial position of the AI coach, and the initial position is located in the real area;

[0012] Projecting the digital AI coach at the initial position through the display module of the MR device.

[0013] In one embodiment, the step of performing action demonstration and training guidance on the real-time interaction of the virtual human based on the AI coach described above includes:

[0014] Analyze the real-time interaction actions of the virtual human based on the training scenario model to determine whether the real-time interaction actions are standard;

[0015] If the real-time interaction actions are not standard, obtain at least one NG key frame of the virtual human;

[0016] Project the NG key frames of the virtual human onto the NG occurrence positions in the mixed reality scenario;

[0017] Based on the NG key frames at the NG occurrence positions, perform a first prominent marking on the NG force application points and NG human body contours of the virtual human;

[0018] Drive the AI coach to perform action demonstrations and training guidance.

[0019] In one embodiment, the above-mentioned driving the AI coach to perform action demonstrations includes:

[0020] Obtain the training actions of the current training content based on the training scenario model;

[0021] Obtain the frame-by-frame contour model of the AI coach;

[0022] Based on the origin coordinates of the trainee, the digital virtual training tool, and the training elements, determine the demonstration starting position, demonstration movement trajectory, and demonstration movement speed of the AI coach;

[0023] Based on the demonstration starting position, demonstration movement trajectory, demonstration movement speed, and frame-by-frame contour model, drive the AI coach to perform action demonstrations.

[0024] In one embodiment, the above-mentioned driving the AI coach to conduct training guidance includes:

[0025] Based on the current staying position of the trainee, determine the target position of the AI coach;

[0026] Control the AI coach to move from the current position to the target position;

[0027] Control the AI coach to conduct training guidance based on the NG force application points and NG human body contours, and the training guidance includes voice information and text information.

[0028] In one embodiment, after the above-mentioned step of controlling the AI coach to conduct training guidance based on the NG force application points and NG human body contours, and the training guidance includes voice information and text information, it further includes:

[0029] When the real-time position of the AI coach exceeds the real area, perform a second prominent marking on the exceeded contour part.

[0030] The present invention also provides an action training guidance device based on MR mixed reality, including:

[0031] A first generation module is used to project the training elements of the simulation training scene into the real-time environment of the real area to generate a mixed reality scene;

[0032] A driving module, for driving the virtual human avatar to perform real-time interaction with digital virtual training tools and training elements based on a simulation training scenario at least according to the real-time motion trajectory and posture changes of the trainee;

[0033] The second generation module is used to generate an AI coach in a mixed reality scenario;

[0034] The training module is used to demonstrate actions and provide training guidance based on the real-time interaction of the AI ​​coach with the virtual human.

[0035] The present invention also provides a motion training system based on MR mixed reality, and the motion training system based on MR mixed reality includes: one or more motion training guidance devices based on MR mixed reality;

[0036] The MR mixed reality-based motion training guidance device is controlled by any one of the MR mixed reality-based motion training guidance methods described above.

[0037] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned MR mixed reality-based action training guidance method when executing the computer program.

[0038] The present invention also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned MR mixed reality-based action training guidance method is implemented.

[0039] In an embodiment of the present invention, by projecting the training elements of the simulation training scenario onto the real-time environment of the real area, a mixed reality scenario is generated; at least according to the real-time movement trajectory and posture change of the trainee, the virtual human avatar is driven to execute real-time interaction of the digital virtual training tools and training elements based on the simulation training scenario; an AI coach is generated in the mixed reality scenario; based on the AI coach, action demonstration and training guidance are provided for the real-time interaction of the virtual human. The real-time movement trajectory and posture change of the trainee can be embodied through the virtual human avatar in the mixed reality scenario, and the actions of the virtual human are based on the actions of the trainee, which is convenient for the trainee to review and view later. At the same time, by generating an AI coach in the mixed reality scenario, compared with the existing method of playing holographic animations or films, the AI coach can provide action demonstration and training guidance for the trainee according to the real-time interaction actions of the virtual human, referring to the real-time movement trajectory and posture change of the trainee. Since the training elements and training tools are projected into the real-time environment, the AI coach also provides a prerequisite for the real-time interaction of the digital virtual training tools and training elements, improving the intelligent level of action training, ensuring the guidance effect, and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 It is a flowchart of a method for guiding action training based on MR mixed reality according to an embodiment of the present invention;

[0042] Figure 2 It is a flowchart of a method for guiding action training based on MR mixed reality according to another embodiment of the present invention;

[0043] Figure 3 It is a detailed flowchart of S21 according to another embodiment of the present invention;

[0044] Figure 4 It is a detailed flowchart of S22 according to another embodiment of the present invention;

[0045] Figure 5 It is a detailed flowchart of S222 according to another embodiment of the present invention;

[0046] Figure 6 It is a detailed flowchart of S24 according to another embodiment of the present invention;

[0047] Figure 7Detailed flowchart of S25 for another embodiment of the present invention;

[0048] Figure 8 Detailed flowchart of S26 for another embodiment of the present invention;

[0049] Figure 9 Detailed flowchart of driving an AI coach for action demonstration in an embodiment of the present invention;

[0050] Figure 10 Detailed flowchart of driving an AI coach for training guidance in an embodiment of the present invention;

[0051] Figure 11 Block diagram of an action training guidance device based on MR mixed reality in an embodiment of the present invention;

[0052] Figure 12 Schematic diagram of the internal structure of a computer for another embodiment of the present invention. Specific embodiments

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Well-known modules, units, and their connections, links, communications, or operations therebetween are not shown or not described in detail. And the described features, architectures, or functions can be combined in any way in one or more embodiments. Those skilled in the art should understand that the following various embodiments are only for illustration, rather than for limiting the protection scope of the present invention. It can also be easily understood that the modules, units, or processing methods in the embodiments described herein and shown in the drawings can be combined and designed in various different configurations. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0054] Please refer to Figure 1 As shown, an embodiment of the present invention discloses a method for action training guidance based on MR mixed reality, including:

[0055] S11, project the training elements of the simulation training scenario onto the real-time environment of the real area to generate a mixed reality scenario.

[0056] The simulation training scenario in this embodiment can be the target training scenario of the training item selected by the trainee. Exemplarily, for tennis, the simulation training scenario may include a tennis court and a fence. For volleyball, the simulation training scenario may include a volleyball court and a spectator stand, etc. The training elements in this step include but are not limited to training venues, markings, training simulation environments, and training equipment, etc.

[0057] S12. At least according to the real-time movement trajectory and posture change of the trainee, drive the virtual human substitute to perform real-time interaction with the digital virtual training tools and training elements based on the simulation training scenario.

[0058] The above digital virtual training tools and training elements can be automatically generated according to the current training project of the user. As a specific solution, the digital virtual training tools in this step can be generated based on the tools carried by the trainee himself, or selected by the trainee from the material library. Similarly, the forms of expression of the training elements can also be selected correspondingly according to the needs of the user.

[0059] S13. Generate an AI coach in the mixed reality scenario.

[0060] The AI coach in this embodiment can be a digitally constructed virtual human, or can be based on real sports stars or athletes. It should be noted that based on the requirements of subsequent action demonstrations, the image of the AI coach is usually close to the physique of real athletes to provide more accurate guidance for the trainee. As a preferred solution, the trainee can select his own idol star or athlete according to his own preference to improve the training enthusiasm.

[0061] S14. Based on the AI coach, conduct action demonstrations and training guidance for the real-time interaction of the virtual human.

[0062] In this embodiment, the AI coach can provide more accurate and reliable training guidance for the real-time interaction of the virtual human, combined with the real-time movement trajectory and posture change of the trainee. As an extension rather than a limitation, the AI coach can also, after determining the training content of the training project, before the trainee's training, automatically conduct the training process and action demonstrations according to the subject content. This embodiment is not limited.

[0063] In the embodiments of the present invention, the training elements of the simulation training scenario are projected onto the real-time environment of the real area to generate a mixed reality scenario; at least according to the real-time movement trajectory and posture change of the trainee, the virtual human substitute is driven to perform real-time interaction with the digital virtual training tools and training elements based on the simulation training scenario; an AI coach is generated in the mixed reality scenario; based on the AI coach, action demonstration and training guidance are provided for the real-time interaction of the virtual human. The real-time movement trajectory and posture change of the trainee can be embodied through the virtual human substitute in the mixed reality scenario, and the actions of the virtual human are based on the actions of the trainee, which is convenient for the trainee to review and view later. At the same time, by generating an AI coach in the mixed reality scenario, compared with the existing method of playing holographic animations or films, the AI coach can provide action demonstration and training guidance for the trainee according to the real-time interaction actions of the virtual human, referring to the real-time movement trajectory and posture change of the trainee. Since the training elements and training tools are projected into the real-time environment, the AI coach also provides a prerequisite for the real-time interaction of the digital virtual training tools and training elements, improving the intelligent level of action training, ensuring the guidance effect, and enhancing the user experience.

[0064] Current MR mixed reality technology is usually applied in the entertainment field. In game scenarios, the demand for the venue is relatively low, and users can usually experience MR mixed reality in a small space.

[0065] With the progress of technology, the demand for MR mixed reality in training scenarios is also gradually increasing. Sports such as tennis, volleyball, and high jump are restricted by the training venue and the range of user activities, and have a higher demand for the venue, making the traditional MR mixed reality technology unable to be applied to most training scenarios, restricting the application of MR mixed reality scenarios in corresponding training projects.

[0066] At least based on the above-mentioned defects of the existing technology, please refer to Figures 2 to 10 As shown, the embodiments of the present invention disclose another action training guidance method based on MR mixed reality, including S21 - S27, where:

[0067] S21, generating a digital three-dimensional map based on the on-site environment, where the digital three-dimensional map at least includes the first height information, the first position information, and the first shape information of the environmental object.

[0068] In this embodiment, the digital three-dimensional map is used to objectively reflect the on-site environment and the surrounding environmental objects. The environmental objects can be fixed objects or non-fixed objects in the on-site environment, including but not limited to pillars, ceilings, steps, slopes, furniture, sundries, trees, shrubs, grasslands, flower beds, or stones, etc. Through at least the first height information, the first position information, and the first shape information of the environmental object, the digital three-dimensional map can accurately reflect the on-site environment.

[0069] As a preferred solution rather than a limitation, please refer to Figure 3 As shown, this step S21 specifically includes S211 - S213, where:

[0070] S211, determine the current position information of the trainee as the origin coordinates;

[0071] S212, obtain the on-site environment by at least real-time scanning and shooting based on the cameras and depth cameras on the MR device;

[0072] S213, based on the origin coordinates, generate a digital three-dimensional map including the ground, the first height information, the first position information, and the first shape information of environmental objects.

[0073] The above steps S211 - S213 use the current position information of the trainee as the origin coordinates, and at the same time, obtain the on-site environment by real-time scanning and shooting based on the cameras and depth cameras on the MR device, so that the positions, shapes, and size information of the ground and environmental objects can be determined more accurately and reliably, improving the data richness and reliability of the digital three-dimensional map.

[0074] S22, based on the benchmark parameters of the simulation training scenario, determine the real area corresponding to the simulation training scenario according to the digital three-dimensional map.

[0075] The simulation training scenario can be the target training scenario of the training project selected by the trainee. Exemplarily, for tennis, the simulation training scenario can include a tennis court and a fence. For volleyball, the simulation training scenario can include a volleyball court and a spectator stand, etc. Usually, the benchmark parameters of the simulation training scenario are the size data of the training venue in the target training scenario. As a preferred solution rather than a limitation, please refer to Figure 4 As shown, this step S22 specifically includes S221 - S222, where:

[0076] S221, obtain the benchmark parameters of the simulation training scenario, and at least determine the actual shape and size data of the training scenario.

[0077] In this step, the actual shape and size data of the training scenario can be the actual shape and size data of the above-mentioned training venue. Or, as a further improvement, the actual shape and size data of the training scenario can be the union of half of the training venue area (such as half of a tennis court or a volleyball court) and the activity area of the trainee. To minimize the site occupancy of the real area corresponding to the training scenario.

[0078] S222, at least based on the actual shape and size data of the training scenario, determine the real area corresponding to the simulation training scenario in the three-dimensional digital map.

[0079] Limited by the technical conditions of the cameras and depth cameras on existing MR devices, in some embodiments, the action training guidance method for MR mixed reality may still lack accuracy in reflecting the on-site environment in complex scenarios, such as when there are many environmental objects, the front environmental objects block the rear environmental objects, the shape features of the environmental objects are relatively complex, or the size is small, which may lead to the possibility of trainees getting injured during the movement. Based on this, the embodiments of the present invention also provide an implementation scheme for accurately determining the on-site environment. As a preferred scheme rather than a limitation, please refer to Figure 5 As shown, this step S222 specifically includes S22201 - S22210, where:

[0080] S22201, determine at least one exploration route in the three-dimensional digital map based at least on the actual shape and size data of the training scene, and the exploration route includes at least a starting point, feature points, random points, and an end point.

[0081] In this step, by determining one or more exploration routes in the three-dimensional digital map based on the actual shape and size data of the training scene, where the selection of the exploration route can be along the edge of the actual shape of the training scene, or through relatively many environmental objects. Among different exploration routes, the MR device will pass through the same environmental object from different angles, so that the height, position, and shape of the environmental object are more rich and accurate.

[0082] As a specific implementation scheme rather than a limitation, guiding markings and guiding points can be generated on the MR device to guide the trainee through the exploration route and perform operations of rotating the MR device to obtain images / information at the corresponding guiding points. The MR device includes but is not limited to an MR helmet and a wearable / handheld MR camera. The guiding points include a starting point, feature points, random points, and an end point. Among them, the feature points correspond to the best exploration positions of different environmental objects at different angles to avoid inaccurate restoration of the subsequent on-site environment caused by occlusion or poor distance. There can be multiple feature points for the same environmental object, and these multiple feature points can be located in different exploration routes or in the same exploration route.

[0083] S22202, based on the starting point coordinates, obtain the second height information, second position information, and second shape information of the environmental object;

[0084] S22203, based on the random point coordinates, obtain the third height information, third position information, and third shape information of the environmental object;

[0085] S22204, based on the end point coordinates, obtain the fourth height information, fourth position information, and fourth shape information of the environmental object.

[0086] The above steps S22202 - S22204 are used to obtain the height information, position information, and shape information of the environmental object based on the starting point coordinates, random point coordinates, and ending point coordinates respectively, so as to make the height information, position information, and shape information of the environmental object more abundant.

[0087] The above random points can be evenly distributed on the survey line or randomly generated according to the three - dimensional digital map. Exemplarily, the distribution interval of the random points can be 1 meter per point.

[0088] S22205, based on the first height information, second height information, third height information, and fourth height information of the same environmental object, determine the standard height information of the corresponding environmental object;

[0089] S22206, based on the first position information, second position information, third position information, and fourth position information of the same environmental object, determine the standard position information of the corresponding environmental object;

[0090] S22207, based on the first shape information, second shape information, third shape information, and fourth shape information of the same environmental object, determine the standard shape information of the corresponding environmental object.

[0091] The standard height information, standard position information, and standard shape information can be generated according to a preset algorithm or model, and are used to represent the median values of the height, position, and shape of the corresponding environmental object obtained based on the starting point, random point, and ending point.

[0092] S22208, based on the feature point coordinates, obtain the verification height information, verification position information, and verification shape information of the environmental object.

[0093] The verification height information, verification position information, and verification shape information can be generated according to a preset algorithm or model, and are used to represent the median values of the height, position, and shape of the corresponding environmental object obtained based on multiple feature points.

[0094] S22209, based on the verification height information, verification position information, and verification shape information of the environmental object, verify the standard height information, standard position information, and standard shape information, and determine the final height information, final position information, and final shape information of the environmental object.

[0095] As a specific example rather than a limitation, when the difference between the verification information and the standard information meets the preset threshold, it is determined that the verification passes; when the difference between the verification information and the standard information exceeds the preset threshold, it is determined that the verification fails. For the height / position / shape that fails, it is determined based on the larger value of the verification information and the standard information / the maximum shape contour.

[0096] S22210. Determine the real - world area corresponding to the simulation training scenario in the 3D digital map based on the final height information, final position information, and final shape information of all environmental objects in the 3D digital map, the actual shape and size data of the training scenario, and the activity range of the trainer.

[0097] In this step, according to the final height information, final position information, and final shape information of the environmental objects, update the height information, position information, and shape information of the environmental objects in the 3D digital map, and determine the coordinates of the 3D digital map where potential safety hazards may exist. Then, according to the actual shape, size data of the training scenario, and the activity range of the trainer, select the optimal safe area in the 3D map as the real - world area of the simulation training scenario. When the real - world area is re - determined, the origin coordinates will be regenerated and updated.

[0098] S23. Project the training elements of the simulation training scenario onto the real - time environment of the real - world area to generate a mixed - reality scene.

[0099] The training elements in this step include but are not limited to training venues, markings, training simulation environments, training equipment, etc.

[0100] S24. At least according to the real - time movement trajectory and posture changes of the trainee, drive the virtual human avatar to perform real - time interactions based on the digital virtual training tools and training elements of the simulation training scenario.

[0101] As a specific implementation manner rather than a limitation of this embodiment, please refer to Figure 6 As shown, this step S24 specifically includes S241 - S244, where:

[0102] S241. Based on the training content, establish a digital human execution proportional joint model and a training scenario model, and determine digital virtual training tools and training elements.

[0103] S242. At least based on the digital 3D map of the ground, the first height information, first position information, and first shape information of the environmental objects, determine the body size information of the trainee according to the digital human execution proportional joint model, and generate a virtual human corresponding to the body size information of the trainee.

[0104] S243. Based on the MR device and motion perception device of the trainee, determine the real - time movement trajectory and posture changes of the trainee according to the training scenario model and body size information.

[0105] S244. Based on the real - time movement trajectory and posture changes, simulate and drive the virtual human to perform real - time interactions with the digital virtual training tools and training elements in the simulation mixed - reality scene.

[0106] After the above steps S241 - S244 establish the digital human execution proportional joint model, the purpose of generating an equi - proportional virtual human can be achieved. In some cases, the weight and body fat percentage information of the trainee can also be collected to make the virtual human more conform to the trainee's figure. The motion perception device in this embodiment includes, but is not limited to, a gesture capture function module, a micro - wearable gyroscope / acceleration sensor, etc. After the above steps S241 - S244 establish the training scene model, by combining the training scene corresponding to the training project, as well as the trainee's MR device and motion perception device, the real - time motion trajectory and posture changes of the trainee are determined according to the body information to quickly and accurately determine the trainee's current action, and the interaction process and result are presented to the trainee through the display module of the MR device, completing a process of "the trainee makes an action and obtains feedback from the training system".

[0107] In the embodiment of the present invention, a digital three - dimensional map based on the on - site environment is generated. The digital three - dimensional map at least includes the first height information, the first position information, and the first shape information of the environmental object; based on the benchmark parameters of the simulation training scene, the real - world area corresponding to the simulation training scene is determined according to the digital three - dimensional map; the training elements of the simulation training scene are projected onto the real - time environment of the real - world area to generate a mixed reality scene; at least according to the real - time motion trajectory and posture changes of the trainee, the virtual human double is driven to execute real - time interaction of the digital virtual training tools and training elements based on the simulation training scene. The first height information, the first position information, and the first shape information of each environmental object in the real environment can be used to judge the generation conditions of the mixed reality scene, and the benchmark parameters of the simulation training scene also ensure that the training venue can be projected onto the real - time environment of the real - world area in an equi - proportional or smaller occupied space, making the generation of the mixed reality scene safer and more efficient, and effectively ensuring the application of the MR mixed reality technology in the training scene.

[0108] S25. Generate an AI coach in the mixed reality scene.

[0109] Different from the above - mentioned method embodiment, please refer to Figure 7 As shown, this step S25 further includes S251 - S253:

[0110] S251. Obtain the origin coordinates of the trainee at the start of training.

[0111] Generally, the origin coordinates are the updated origin coordinates of S22210. Determining the origin coordinates is beneficial for generating the AI coach at the initial position in the subsequent process, making the AI coach in a position closer to the center of the real - world area for the interaction between the AI coach and the trainee.

[0112] S252. Based on the origin coordinates of the trainee and the training content, determine the initial position of the AI coach, and the initial position is located in the real - world area;

[0113] In this step, the initial position of the AI coach is determined based on the origin coordinates of the trainee and the training content. Specifically, by obtaining the training content, the training route of the trainee and the movement trajectory of the training tool are determined, so that the initial position of the AI coach will not affect the normal training process of the trainee, and the situation where the AI coach interferes with the training route and the movement trajectory of the training tool and affects the trainee's training concentration is avoided.

[0114] S253, project the digital AI coach at the initial position through the display module of the MR device.

[0115] S26, conduct action demonstrations and training guidance based on the real-time interaction between the AI coach and the virtual human.

[0116] Different from the above method embodiments, please refer to Figure 8 As shown, this step S26 further includes:

[0117] S261, analyze the real-time interaction actions of the virtual human based on the training scenario model, and judge whether the real-time interaction actions are standard.

[0118] S262, if the real-time interaction actions are not standard, obtain at least one NG key frame of the virtual human;

[0119] S263, project the NG key frame of the virtual human at the NG occurrence position in the mixed reality scene;

[0120] S264, based on the NG key frame at the NG occurrence position, perform a first prominent marking on the NG force application point and the NG human body contour of the virtual human;

[0121] S265, drive the AI coach to conduct action demonstrations and training guidance.

[0122] The NG key frames in the above steps S261-S265 may include a first key frame and a second key frame. Among them, the NG key frame may be the first key frame of the real-time interaction of the virtual human when the NG action appears. In the first key frame, the real-time interaction actions of the virtual human have been deformed or deviated; the NG key frame may also be the second key frame of the real-time interaction with insufficient force application derived by the training scenario model based on the first key frame when the real-time interaction actions of the virtual human have been deformed or deviated. In the second key frame, although the real-time interaction actions of the virtual human have not been deformed or deviated, due to the insufficient force application in the second key frame, the real-time interaction actions of the virtual human in the subsequent first key frame will be deformed or deviated.

[0123] By projecting the NG key frames at the NG occurrence positions in the mixed reality scene to restore the incorrect actions of the virtual human when completing real-time interaction actions, the trainee can approach the virtual human image of the NG key frame, observe from multiple perspectives, and combine the first prominent identifier to find their own action defects, so as to pay attention to and improve them subsequently. The first prominent identifier may include the first color based on the first shape contour. Among them, the first shape contour may be the contour of the force-generating muscle group, or the contour of a partial limb of the virtual human whose action is deformed or deviated.

[0124] As a preferred solution rather than a limitation, please refer to Figure 9 As shown, driving the AI coach to perform action demonstrations includes:

[0125] S2651, obtaining the training actions of the current training content based on the training scene model.

[0126] The determination of the specific training actions in this embodiment can be input by the trainee or obtained by analyzing the real-time interaction actions of the virtual human.

[0127] S2652, obtaining the frame-by-frame contour model of the AI coach.

[0128] In this step, according to the training actions, the corresponding frame-by-frame contour model of the AI coach is retrieved in the training scene model, where the frame-by-frame contour model includes data such as the posture, height, and angle of the AI coach model.

[0129] S2653, based on the origin coordinates of the trainee, the digital virtual training tool, and the training elements, determining the demonstration start position, demonstration movement trajectory, and demonstration movement speed of the AI coach.

[0130] In this embodiment, the demonstration start position of the AI coach can be at the origin coordinates or adjusted according to the origin coordinates. Exemplarily, when the distance from the origin coordinates to the training element (such as the high jump bar) is relatively close, resulting in insufficient running-up distance, the demonstration start position of the AI coach can be adjusted according to the actual distance required for the running-up. Similarly, the demonstration movement trajectory of the AI coach can be determined based on the demonstration start position, the digital virtual training tool, and the training elements, and then the training scene model generates the demonstration movement speed personalized according to the demonstration start position, the demonstration movement trajectory, and the training actions. The demonstration movement speed can be different at different travel positions of the demonstration movement trajectory.

[0131] S2654, driving the AI coach to perform action demonstrations based on the demonstration start position, the demonstration movement trajectory, the demonstration movement speed, and the frame-by-frame contour model.

[0132] The above steps provide a specific solution for an AI coach to perform action demonstrations, obtain training actions and frame-by-frame contour models of the AI coach according to the training scenario model, and then determine the demonstration starting position, demonstration movement trajectory, and demonstration movement speed of the AI coach. This enables the demonstration actions of the AI coach to closely interact with the digital virtual training tools and training elements in the current mixed reality scenario. At the same time, the demonstration starting position of the AI coach can be the same as or close to the position of the trainee, so as to further fit the real-time actions of the trainee, which is conducive to disassembling training actions and improving training effects.

[0133] As a preferred solution rather than a limitation, please refer to Figure 10 As shown, driving the AI coach to conduct training guidance includes:

[0134] S2655, based on the current staying position of the trainee, determine the target position of the AI coach;

[0135] S2656, control the AI coach to move from the current position to the target position;

[0136] S2657, control the AI coach to conduct training guidance based on the NG force application points and NG human body contours, and the training guidance includes voice information and text information.

[0137] The above steps S2655 - S2657 can determine the target position of the AI coach according to the real-time staying position of the trainee, control the movement of the AI coach, which is more in line with the actual scenario and conducive to ensuring the efficient interaction between the trainee and the AI coach. At the same time, through voice information and text information, training guidance is provided for the NG force application points and NG human body contours of the trainee, further ensuring the efficiency of communication.

[0138] S27, when the real-time position of the AI coach exceeds the real area, perform a second prominent marking on the exceeded contour part.

[0139] The second prominent marking may include a second color based on the second shape contour. The second shape contour may be the partial limb contour of the AI coach's body that exceeds the real area. Through the second prominent marking, a clear sense of the boundary of the real area can be provided to the trainee, which is conducive to providing safety guidance to the trainee when the trainee is immersed in training.

[0140] It should be noted that the embodiments of the present invention can generate multiple personal profiles according to user accounts. Each trainee will accumulate training data during each training. The training data is compared with the training objectives. Each training will generate corresponding training content based on the latest training effect to assist the trainee in achieving the training objectives in the next training, etc.

[0141] Please refer to Figure 11As shown in the figure, the present invention also provides an action training guidance device 100 based on MR mixed reality, including:

[0142] A first generation module 110, configured to project training elements of a simulation training scenario onto the real-time environment of the real area to generate a mixed reality scenario;

[0143] A driving module 120, configured to drive a virtual human substitute to perform real-time interaction of digital virtual training tools and training elements based on the simulation training scenario at least according to the real-time movement trajectory and posture change of the trainee;

[0144] A second generation module 130, configured to generate an AI coach in the mixed reality scenario;

[0145] A training module 140, configured to perform action demonstration and training guidance on the real-time interaction of the virtual human based on the AI coach.

[0146] The modules in this embodiment are the same as the corresponding steps in the above one or more method embodiments, and will not be elaborated here.

[0147] All the modules / units in this embodiment are the same as the corresponding steps in the above one or more method embodiments, and their logical relationships and working principles are also the same, and will not be elaborated here. Those skilled in the art can learn the corresponding virtual modules or units from the above method embodiments to make them correspond to the steps of the above method embodiments. The virtual modules / units not disclosed in this embodiment should also be regarded as part of the content disclosed in the present invention.

[0148] In the embodiment of the present invention, by projecting the training elements of the simulation training scenario onto the real-time environment of the real area, a mixed reality scenario is generated; at least according to the real-time movement trajectory and posture change of the trainee, a virtual human substitute is driven to perform real-time interaction of digital virtual training tools and training elements based on the simulation training scenario; an AI coach is generated in the mixed reality scenario; and action demonstration and training guidance are performed on the real-time interaction of the virtual human based on the AI coach. This enables the real-time movement trajectory and posture change of the trainee to be embodied through the virtual human substitute in the mixed reality scenario, and the actions of the virtual human are based on the actions of the trainee, facilitating the later review and viewing by the trainee. At the same time, by generating an AI coach in the mixed reality scenario, compared with the existing method of playing holographic animations or films, the AI coach can provide action demonstration and training guidance for the trainee according to the real-time interaction actions of the virtual human and with reference to the real-time movement trajectory and posture change of the trainee. Since the training elements and training tools are projected into the real-time environment, it provides a prerequisite for the AI coach to perform real-time interaction with the digital virtual training tools and training elements, improving the intelligent level of action training, ensuring the guidance effect, and enhancing the user experience.

[0149] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working processes of the systems, devices, and units described above, reference can be made to the corresponding processes in the foregoing method embodiments, which will not be elaborated here.

[0150] Another aspect of the embodiments of the present invention provides an action training system based on MR mixed reality. The action training system based on MR mixed reality includes: an action training guidance device based on MR mixed reality;

[0151] The action training guidance device based on MR mixed reality is controlled by the action training guidance method based on MR mixed reality in one or more embodiments.

[0152] The training system of this embodiment can perform data communication with multiple MR devices simultaneously through a communication port. By binding the unique identity identification (SN, MAC) of the MR device + the trainee account, the training processes of multiple trainees can be precisely managed simultaneously.

[0153] The embodiments of the present invention also provide a computer storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the action training guidance method based on MR mixed reality in the above embodiments.

[0154] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above various MR-based mixed reality action training guidance methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0155] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the related technology can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the methods of the various embodiments of the present invention. And the aforementioned storage medium includes: various media such as removable storage devices, RAM, ROM, magnetic disks, or optical discs that can store program codes.

[0156] Corresponding to the above computer storage medium, in one embodiment, a computer device is further provided. The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. Wherein, when the processor executes the program, it implements the MR-based mixed reality action training guidance methods in the above various embodiments.

[0157] The computer device can be a terminal, and its internal structural diagram can be as Figure 12As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes a motion training guidance method based on MR mixed reality. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad set on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0158] In the embodiment of the present invention, by projecting the training elements of the simulation training scenario onto the real-time environment of the real area, a mixed reality scenario is generated; at least according to the real-time motion trajectory and posture change of the trainee, driving the virtual human substitute to perform real-time interaction of digital virtual training tools and training elements based on the simulation training scenario; generating an AI coach in the mixed reality scenario; and performing action demonstration and training guidance on the real-time interaction of the virtual human based on the AI coach. The real-time motion trajectory and posture change of the trainee can be embodied through the virtual human substitute in the mixed reality scenario, and the actions of the virtual human are based on the actions of the trainee, which is convenient for the trainee to review and view later. At the same time, by generating an AI coach in the mixed reality scenario, compared with the existing method of playing holographic animations or films, the AI coach can provide action demonstration and training guidance for the trainee according to the real-time interaction actions of the virtual human, referring to the real-time motion trajectory and posture change of the trainee. Since the training elements and training tools are projected into the real-time environment, the AI coach also provides a prerequisite for the real-time interaction of digital virtual training tools and training elements, improving the intelligent level of action training, ensuring the guidance effect, and enhancing the user experience.

[0159] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0160] The above embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. A motion training guidance method based on MR mixed reality, characterized in that: include: Project the training elements of the simulation training scenario into the real-time environment of the real area to generate a mixed reality scenario; At least according to the real-time motion trajectory and posture changes of the trainee, drive the virtual human avatar to perform real-time interaction with the digital virtual training tools and training elements based on the simulation training scene; Generate an AI coach in a mixed reality scenario; Based on the real-time interaction of AI coaches with virtual humans, action demonstrations and training guidance are provided.

2. The method according to claim 1, characterized in that The step of generating an AI coach in a mixed reality scenario includes: Get the trainee's origin coordinates at the beginning of training; Determine an initial position of the AI ​​coach based on the trainee's origin coordinates and training content, where the initial position is located in the real area; The digital AI coach is projected onto the initial position through the display module of the MR device.

3. The method according to claim 2, characterized in that The steps of performing action demonstration and training guidance based on real-time interaction of the virtual person by the AI ​​coach include: Analyze the real-time interactive actions of the virtual human based on the training scenario model to determine whether the real-time interactive actions are standard; If the real-time interactive action is not standard, obtain at least one NG key frame of the virtual person; Projecting the NG key frame of the virtual person onto the NG occurrence position of the mixed reality scene; According to the NG key frame of the NG occurrence position, the NG force point and the NG human body contour of the virtual person are first prominently identified; Drive the AI ​​coach to perform action demonstrations and training guidance.

4. The method according to claim 3, characterized in that The driving AI coach performs action demonstrations, including: Obtaining training actions of current training content based on the training scenario model; Get the frame-by-frame silhouette model of the AI ​​coach; Determine the demonstration starting position, demonstration movement trajectory and demonstration movement rate of the AI ​​coach based on the trainee's origin coordinates, digital virtual training tools and training elements; Based on the demonstration starting position, demonstration motion trajectory, demonstration motion rate and frame-by-frame contour model, the AI ​​coach is driven to perform action demonstration.

5. The method according to claim 4, characterized in that The driving AI coach to provide training guidance includes: Determine the target location of the AI ​​coach based on the trainee’s current location; Control the AI ​​coach to move from the current position to the target position; The AI ​​coach is controlled to provide training guidance based on the NG force points and the NG human body contour, wherein the training guidance includes voice information and text information.

6. The method according to claim 5, characterized in that After the step of controlling the AI ​​coach to provide training guidance based on the NG force point and the NG human body contour, wherein the training guidance includes voice information and text information, the step further includes: When the real-time position of the AI ​​coach exceeds the actual area, a second eye-catching mark is performed on the exceeded contour portion.

7. An action training guidance device based on MR mixed reality, characterized in that: include: A first generation module is used to project the training elements of the simulation training scene into the real-time environment of the real area to generate a mixed reality scene; A driving module, for driving the virtual human avatar to perform real-time interaction with digital virtual training tools and training elements based on a simulation training scenario at least according to the real-time motion trajectory and posture changes of the trainee; The second generation module is used to generate an AI coach in a mixed reality scenario; The training module is used to demonstrate actions and provide training guidance based on the real-time interaction of the AI ​​coach with the virtual human.

8. A motion training system based on MR mixed reality, characterized in that: The MR mixed reality-based motion training system comprises: one or more MR mixed reality-based motion training guidance devices; The MR mixed reality-based motion training guidance device is controlled by the MR mixed reality-based motion training guidance method described in any one of claims 1 to 7.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the action training guidance method based on MR mixed reality is implemented as described in any one of claims 1 to 6.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the MR mixed reality-based action training guidance method as described in any one of claims 1 to 6 is implemented.

Images