Motion training method and device based on MR mixed reality

Abstract

The invention provides an action training method and device based on MR mixed reality, and the method comprises the steps: generating a digital three-dimensional map based on an on-site environment, and the digital three-dimensional map at least comprises the first height information, the first position information and the first shape information of an environment object; based on reference parameters of a simulation training scene, determining a real area corresponding to the simulation training scene according to the digital three-dimensional map; projecting training elements of the simulation training scene to a real-time environment of the reality area to generate a mixed reality scene; and at least according to the real-time motion trail and the posture change of the trainee, driving the virtual human substitute to execute real-time interaction of the digital virtual training tool and the training elements based on the simulation training scene. According to the invention, the generation of the mixed reality scene is safer and more efficient, and the MR mixed reality technology can be effectively ensured to be applied to the training scene.

Description

Technical Field

[0001] The present invention relates to the technical field of MR equipment, and in particular to a motion training method and device based on MR mixed reality. Background Art

[0002] Current MR mixed reality technology is usually used in the entertainment field. In gaming scenarios, the demand for venues is relatively low, and users can usually experience MR mixed reality in a small space.

[0003] However, with the advancement of technology, the demand for MR mixed reality in training scenarios is gradually increasing. Sports such as tennis, volleyball, and high jump are limited by training venues and user activity ranges, and have higher requirements for venues, making traditional MR mixed reality technology unable to be applied to most training scenarios, limiting the application of MR mixed reality scenes in corresponding training projects. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention proposes a motion training method based on MR mixed reality, comprising:

[0005] Generate a digital three-dimensional map based on the on-site environment, the digital three-dimensional map including at least first height information, first position information and first shape information of the environmental object;

[0006] Based on the reference parameters of the simulation training scene, determining a real area corresponding to the simulation training scene according to the digital three-dimensional map;

[0007] Projecting the training elements of the simulation training scene into the real-time environment of the real area to generate a mixed reality scene;

[0008] At least according to the real-time motion trajectory and posture changes of the trainee, the virtual human avatar is driven to perform real-time interaction with digital virtual training tools and training elements based on the simulation training scene.

[0009] In one embodiment, the step of generating a digital three-dimensional map based on a site environment, wherein the digital three-dimensional map includes at least first height information, first position information, and first shape information of an environmental object, comprises:

[0010] Determine the trainee's current position information as the origin coordinates;

[0011] At least based on the camera and depth camera on the MR device to scan and shoot in real time to obtain the scene environment;

[0012] Based on the origin coordinates, a digital three-dimensional map including the ground, first height information, first position information and first shape information of environmental objects is generated.

[0013] In one embodiment, the step of determining, according to the digital three-dimensional map, the real area corresponding to the simulation training scenario for the above-mentioned benchmark parameters based on the simulation training scenario includes:

[0014] Obtain the benchmark parameters of the simulation training scenario and at least determine the actual shape and dimension data of the training scenario;

[0015] Based on at least the actual shape and dimension data of the training scenario, determine the real area corresponding to the simulation training scenario in the three-dimensional digital map.

[0016] In one embodiment, the step of determining, based on at least the actual shape and dimension data of the training scenario, the real area corresponding to the simulation training scenario in the three-dimensional digital map includes:

[0017] Based on at least the actual shape and dimension data of the training scenario, determine at least one survey line in the three-dimensional digital map, where the survey line includes at least a starting point, feature points, random points, and an end point;

[0018] Based on the starting point coordinates, obtain the second height information, second position information, and second shape information of the environmental object;

[0019] Based on the random point coordinates, obtain the third height information, third position information, and third shape information of the environmental object;

[0020] Based on the end point coordinates, obtain the fourth height information, fourth position information, and fourth shape information of the environmental object;

[0021] Based on the first height information, second height information, third height information, and fourth height information of the same environmental object, determine the standard height information of the corresponding environmental object;

[0022] Based on the first position information, second position information, third position information, and fourth position information of the same environmental object, determine the standard position information of the corresponding environmental object;

[0023] Based on the first shape information, second shape information, third shape information, and fourth shape information of the same environmental object, determine the standard shape information of the corresponding environmental object;

[0024] Based on the feature point coordinates, obtain the verification height information, verification position information, and verification shape information of the environmental object;

[0025] Based on the verification height information, verification position information, and verification shape information of the environmental object, verify the standard height information, standard position information, and standard shape information, and determine the final height information, final position information, and final shape information of the environmental object;

[0026] Based on the final height information, final position information, and final shape information of all environmental objects in the three-dimensional digital map, the actual shape, size data of the training scenario, and the activity range of the trainer, determine the real area corresponding to the simulation training scenario in the three-dimensional digital map.

[0027] In one embodiment, the step of driving the virtual human double to perform real-time interaction with the digital virtual training tools and training elements based on the simulation training scenario at least according to the real-time movement trajectory and posture change of the trainee includes:

[0028] Based on the training content, establish a digital human execution proportional joint model and a training scenario model, and determine the digital virtual training tools and training elements;

[0029] Based on at least the digital three-dimensional map of the ground, the first height information, first position information, and first shape information of the environmental objects, determine the body information of the trainee according to the digital human execution proportional joint model, and generate a virtual human corresponding to the body information of the trainee;

[0030] Based on the MR device and motion perception device of the trainee, determine the real-time movement trajectory and posture change of the trainee according to the training scenario model and body information;

[0031] Based on the real-time movement trajectory and posture change, simulate driving the virtual human to perform real-time interaction with the digital virtual training tools and training elements in the simulation mixed reality scenario.

[0032] In one embodiment, after the step of driving the virtual human double to perform real-time interaction with the digital virtual training tools and training elements based on the simulation training scenario at least according to the real-time movement trajectory and posture change of the trainee, it further includes:

[0033] Generate an AI coach in the mixed reality scenario;

[0034] Based on the AI coach, conduct action demonstrations and training guidance for the real-time interaction of the virtual human.

[0035] The present invention also provides an action training device based on MR mixed reality, including:

[0036] A first generation module for generating a digital three-dimensional map based on the on-site environment, the digital three-dimensional map at least including the first height information, first position information, and first shape information of environmental objects;

[0037] A determination module for determining the real area corresponding to the simulation training scenario according to the digital three-dimensional map based on the benchmark parameters of the simulation training scenario;

[0038] A second generating module is used to project the training elements of the simulation training scene into the real-time environment of the real area to generate a mixed reality scene;

[0039] The processing module is used to drive the virtual human avatar to perform real-time interaction with digital virtual training tools and training elements based on the simulation training scene at least according to the real-time movement trajectory and posture changes of the trainee.

[0040] The present invention also provides a motion training system based on MR mixed reality, and the motion training system based on MR mixed reality includes: a motion training device based on MR mixed reality;

[0041] The MR mixed reality-based motion training device is controlled by any one of the MR mixed reality-based motion training methods described above.

[0042] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned MR mixed reality-based motion training method when executing the computer program.

[0043] The present invention also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned MR mixed reality-based action training method is implemented.

[0044] The embodiment of the present invention generates a digital three-dimensional map based on the on-site environment, the digital three-dimensional map includes at least the first height information, the first position information and the first shape information of the environmental object; based on the benchmark parameters of the simulation training scene, the real area corresponding to the simulation training scene is determined according to the digital three-dimensional map; the training elements of the simulation training scene are projected to the real-time environment of the real area to generate a mixed reality scene; at least according to the real-time motion trajectory and posture changes of the trainee, the virtual human avatar is driven to perform real-time interaction of the digital virtual training tools and training elements based on the simulation training scene. The first height information, the first position information and the first shape information of each environmental object in the real environment can be used to determine the generation conditions of the mixed reality scene, and the benchmark parameters of the simulation training scene also ensure that the training venue can be projected in the real-time environment of the real area in equal proportion or with a smaller footprint, making the generation of the mixed reality scene safer and more efficient, and effectively ensuring the application of MR mixed reality technology in training scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0046] Figure 1 It is a flowchart of the action training method based on MR mixed reality in an embodiment of the present invention;

[0047] Figure 2 It is a detailed flowchart of S11 in another embodiment of the present invention;

[0048] Figure 3 It is a detailed flowchart of S12 in another embodiment of the present invention;

[0049] Figure 4 It is a detailed flowchart of S122 in another embodiment of the present invention;

[0050] Figure 5 It is a detailed flowchart of S14 in another embodiment of the present invention;

[0051] Figure 6 For Figure 1 supplementary flowchart;

[0052] Figure 7 It is a detailed flowchart of S15 in another embodiment of the present invention;

[0053] Figure 8 It is a detailed flowchart of S16 in another embodiment of the present invention;

[0054] Figure 9 It is a detailed flowchart of driving an AI coach to perform action demonstrations in an embodiment of the present invention;

[0055] Figure 10 It is a detailed flowchart of driving an AI coach to provide training guidance in an embodiment of the present invention;

[0056] Figure 11 It is a structural block diagram of the action training device based on MR mixed reality in an embodiment of the present invention;

[0057] Figure 12 It is a schematic diagram of the internal structure of a computer in another embodiment of the present invention. Detailed implementation manner

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Well-known modules, units, and their connections, links, communications, or operations are not shown or not described in detail. And the described features, architectures, or functions can be combined in any way in one or more embodiments. Those skilled in the art should understand that the following various embodiments are only for illustration, rather than for limiting the protection scope of the present invention. It can also be easily understood that the modules, units, or processing methods in the various embodiments described herein and shown in the drawings can be combined and designed in various different configurations. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.

[0059] Please refer to Figures 1 to 10 As shown, the embodiments of the present invention disclose an action training method based on MR mixed reality, including:

[0060] S11, generating a digital three-dimensional map based on the on-site environment, where the digital three-dimensional map at least includes the first height information, the first position information, and the first shape information of the environmental objects.

[0061] In this embodiment, the digital three-dimensional map is used to objectively reflect the on-site environment and the surrounding environmental objects. The environmental objects can be fixed objects or non-fixed objects in the on-site environment, including but not limited to pillars, ceilings, steps, slopes, furniture, sundries, trees, shrubs, grasslands, flower beds, or stones, etc. Through at least the first height information, the first position information, and the first shape information of the environmental objects, the digital three-dimensional map can accurately reflect the on-site environment.

[0062] As a preferred solution rather than a limitation, please refer to Figure 2 As shown, this step S11 specifically includes S111 - S113, where:

[0063] S111, determining the current position information of the trainee as the origin coordinates;

[0064] S112, obtaining the on-site environment by real-time scanning and shooting at least based on the cameras and depth cameras on the MR device;

[0065] S113, generating a digital three-dimensional map including the ground, the first height information, the first position information, and the first shape information of the environmental objects based on the origin coordinates.

[0066] The above steps S111 - S113 use the current position information of the trainee as the origin coordinates. At the same time, the on-site environment is obtained by real-time scanning and shooting with the camera and depth camera on the MR device, so that the positions, shapes and size information of the ground and environmental objects can be determined more accurately and reliably, improving the data richness and reliability of the digital three-dimensional map.

[0067] S12. Based on the reference parameters of the simulation training scenario, determine the real area corresponding to the simulation training scenario according to the digital three-dimensional map.

[0068] The simulation training scenario can be the target training scenario of the training project selected by the trainee. Exemplarily, for tennis, the simulation training scenario can include a tennis court and a fence. For volleyball, the simulation training scenario can include a volleyball court and a spectator stand, etc. Usually, the reference parameters of the simulation training scenario are the size data of the training venue in the target training scenario. As a preferred solution rather than a limitation, please refer to Figure 3 As shown, this step S12 specifically includes S121 - S122, where:

[0069] S121. Obtain the reference parameters of the simulation training scenario and at least determine the actual shape and size data of the training scenario.

[0070] In this step, the actual shape and size data of the training scenario can be the actual shape and size data of the above-mentioned training venue. Or, as a further improvement, the actual shape and size data of the training scenario can be the union of half of the training venue (such as half of a tennis court or a volleyball court) and the activity area of the trainee. To minimize the site occupation of the real area corresponding to the training scenario.

[0071] S122. Based on at least the actual shape and size data of the training scenario, determine the real area corresponding to the simulation training scenario in the three-dimensional digital map.

[0072] Limited by the technical conditions of the existing cameras and depth cameras on the MR device, in some embodiments, the action training method of MR mixed reality may still lack accuracy in reflecting the on-site environment in complex scenarios, such as when there are many environmental objects, the front environmental objects block the rear environmental objects, the shape features of the environmental objects are relatively complex or the size is small, resulting in the possibility of trainees getting injured during the movement. Based on this, the embodiments of the present invention also provide an implementation scheme for accurately determining the on-site environment. As a preferred solution rather than a limitation, please refer to Figure 4 As shown, this step S122 specifically includes S12201 - S12210, where:

[0073] S12201. Determine at least one survey line in the 3D digital map based at least on the actual shape and size data of the training scenario, where the survey line includes at least a starting point, feature points, random points, and an ending point.

[0074] In this step, determine one or more survey lines in the 3D digital map based on the actual shape and size data of the training scenario. Among them, the selection of the survey line can be based on the edge of the actual shape of the training scenario or through a relatively large number of environmental objects. Among different survey routes, the MR device will pass through the same environmental object from different angles, so that the height, position, and shape of the environmental object are more rich and accurate.

[0075] As a specific implementation solution rather than a limitation, guiding marking lines and guiding points can be generated on the MR device to guide the trainee through the survey line, and perform operations of rotating the MR device to obtain images / information at the corresponding guiding points. The MR device includes but is not limited to an MR helmet and a wearable / handheld MR camera. The guiding points include a starting point, feature points, random points, and an ending point. Among them, the feature points correspond to the best survey positions of different environmental objects at different angles to avoid inaccurate subsequent restoration of the on-site environment caused by occlusion or poor distance. There can be multiple feature points for the same environmental object, and these multiple feature points can be located in different survey lines or in the same survey line.

[0076] S12202. Based on the starting point coordinates, obtain the second height information, second position information, and second shape information of the environmental object;

[0077] S12203. Based on the random point coordinates, obtain the third height information, third position information, and third shape information of the environmental object;

[0078] S12204. Based on the ending point coordinates, obtain the fourth height information, fourth position information, and fourth shape information of the environmental object.

[0079] The above steps S12202 - S12204 are used to obtain the height information, position information, and shape information of the environmental object according to the starting point coordinates, random point coordinates, and ending point coordinates respectively, so that the height information, position information, and shape information of the environmental object are more rich.

[0080] The above random points can be evenly distributed on the survey line or randomly generated according to the 3D digital map. Exemplarily, the distribution interval of the random points can be 1 meter per point.

[0081] S12205. Based on the first height information, second height information, third height information, and fourth height information of the same environmental object, determine the standard height information of the corresponding environmental object;

[0082] S12206, determine the standard position information of the corresponding environmental object based on the first position information, second position information, third position information, and fourth position information of the same environmental object;

[0083] S12207, determine the standard shape information of the corresponding environmental object based on the first shape information, second shape information, third shape information, and fourth shape information of the same environmental object.

[0084] The standard height information, standard position information, and standard shape information can be generated according to a preset algorithm or model, and are used to represent the median values of the height, position, and shape of the corresponding environmental object obtained based on the starting point, random point, and ending point.

[0085] S12208, obtain the verification height information, verification position information, and verification shape information of the environmental object based on the feature point coordinates.

[0086] The verification height information, verification position information, and verification shape information can be generated according to a preset algorithm or model, and are used to represent the median values of the height, position, and shape of the corresponding environmental object obtained based on multiple feature points.

[0087] S12209, verify the standard height information, standard position information, and standard shape information based on the verification height information, verification position information, and verification shape information of the environmental object, and determine the final height information, final position information, and final shape information of the environmental object.

[0088] As a specific example rather than a limitation, when the difference between the verification information and the standard information meets the preset threshold, it is determined that the verification passes; when the difference between the verification information and the standard information exceeds the preset threshold, it is determined that the verification fails. For the height / position / shape that fails, the larger value / maximum shape contour of the verification information and the standard information is used for determination.

[0089] S12210, determine the real area corresponding to the simulation training scenario in the three-dimensional digital map based on the final height information, final position information, and final shape information of all environmental objects in the three-dimensional digital map, the actual shape and size data of the training scenario, and the activity range of the trainer.

[0090] In this step, according to the final height information, final position information, and final shape information of the environmental object, update the height information, position information, and shape information of the environmental object in the three-dimensional digital map, and determine the coordinates of the three-dimensional digital map where potential safety hazards may exist. Then, according to the actual shape and size data of the training scenario and the activity range of the trainer, select the optimal safe area in the three-dimensional map as the real area of the simulation training scenario. When the real area is re-determined, the origin coordinates will be re-generated and updated.

[0091] S13, projecting the training elements of the simulation training scene into the real-time environment of the real area to generate a mixed reality scene.

[0092] The training elements of this step include but are not limited to training venues, markings, training simulation environments and training equipment.

[0093] S14, at least according to the real-time motion trajectory and posture changes of the trainee, driving the virtual human avatar to perform real-time interaction with the digital virtual training tools and training elements based on the simulation training scene.

[0094] The above-mentioned digital virtual training tools and training elements can be automatically generated according to the user's current training project. As a specific solution, the digital virtual training tools in this step can be generated based on the trainees' own tools, or they can be selected by the trainees from the material library. Similarly, the training element presentation form can also be selected according to the user's needs.

[0095] In this embodiment, the real-time interactive actions of the virtual human avatar can be displayed in the first person or in the third person. The user can set it as needed, and this embodiment does not limit it.

[0096] As a specific implementation of this embodiment but not a limitation, please refer to Figure 5 As shown, this step S14 specifically includes S141-S144, wherein:

[0097] S141, based on the training content, establish a digital human body execution proportion joint model and a training scene model, and determine the digital virtual training tools and training elements.

[0098] S142, based on at least the digital three-dimensional map of the first height information, the first position information and the first shape information of the ground and the environmental objects, and executing the proportional joint model of the digital human, determine the body information of the trainee, and generate a virtual human corresponding to the body information of the trainee.

[0099] S143, based on the trainee's MR device and motion sensing device, determine the trainee's real-time motion trajectory and posture changes according to the training scene model and body shape information.

[0100] S144, based on real-time motion trajectory and posture changes, simulates and drives the real-time interaction of virtual humans with digital virtual training tools and training elements in a simulated mixed reality scene.

[0101] After the above steps S141 - S144 to establish a digital human execution ratio joint model, the purpose of generating an equi - proportion virtual human can be achieved. In some cases, the weight and body fat percentage information of the trainee can also be collected to make the virtual human more conform to the trainee's figure. The motion perception device of this embodiment includes, but is not limited to, a gesture capture function module, a micro - wearable gyroscope / acceleration sensor, etc. After the above steps S141 - S144 to establish a training scenario model, by combining the training scenario corresponding to the training item, as well as the trainee's MR device and motion perception device, the real - time motion trajectory and posture change of the trainee are determined according to the figure information to quickly and accurately determine the current action of the trainee. The interaction process and result are presented to the trainee through the display module of the MR device, completing a process of "the trainee makes an action and obtains feedback from the training system".

[0102] Currently, the application of the existing MR mixed - reality technology in action guidance usually projects pre - stored holographic animations or films into the mixed - reality scene. Users can observe the contour image of the holographic animation or film in the mixed - reality scene to judge the standard degree of their actions.

[0103] However, when the holographic animation or film is playing, it cannot interact in real - time with the digital virtual training tools (such as rackets, baseball bats or poles, etc.) and training elements (such as runways, tennis nets, billiard tables or high - jump poles, etc.) of the currently generated simulation training scene. The degree of intelligence is low, and there is also a lack of personalized action demonstrations and training guidance for the trainee. This makes the guidance effect poor and also affects the user experience.

[0104] At least based on the above - mentioned defects of the existing technology, please refer to Figure 6 As shown, after step S14, there are also steps S15 - S17, where:

[0105] S15, generate an AI coach in the mixed - reality scene.

[0106] Please refer to Figure 7 As shown, this step S15 also includes:

[0107] S151, obtain the origin coordinates of the trainee at the start of training.

[0108] Generally, the origin coordinates are the updated origin coordinates of S12210. The determination of the origin coordinates is beneficial to making the AI coach in a position closer to the center of the real area when generating the AI coach at the initial position in the subsequent process, so as to facilitate the interaction between the AI coach and the trainee.

[0109] S152, based on the origin coordinates of the trainee and the training content, determine the initial position of the AI coach, and the initial position is located in the real area;

[0110] In this step, the initial position of the AI coach is determined based on the origin coordinates of the trainee and the training content. Specifically, by obtaining the training content, the training route of the trainee and the movement trajectory of the training tool are determined, so that the initial position of the AI coach will not affect the normal training process of the trainee, and the situation where the AI coach interferes with the training route and the movement trajectory of the training tool and affects the trainee's training concentration is avoided.

[0111] S153, project the digital AI coach at the initial position through the display module of the MR device.

[0112] S16, conduct action demonstrations and training guidance based on the real-time interaction between the AI coach and the virtual human.

[0113] Different from the above method embodiment, please refer to Figure 8 As shown, this step S16 further includes:

[0114] S161, analyze the real-time interaction actions of the virtual human based on the training scenario model, and determine whether the real-time interaction actions are standard.

[0115] S162, if the real-time interaction actions are not standard, obtain at least one NG key frame of the virtual human;

[0116] S163, project the NG key frame of the virtual human at the NG occurrence position in the mixed reality scene;

[0117] S164, based on the NG key frame at the NG occurrence position, perform a first prominent marking on the NG force application point and the NG human body contour of the virtual human;

[0118] S165, drive the AI coach to conduct action demonstrations and training guidance.

[0119] The NG key frames in the above steps S161 - S165 may include a first key frame and a second key frame. Among them, the NG key frame may be the first key frame of the real-time interaction of the virtual human when the NG action appears. In the first key frame, the real-time interaction actions of the virtual human have already been deformed or deviated; the NG key frame may also be the second key frame of the real-time interaction with insufficient force application derived by the training scenario model based on the first key frame when the real-time interaction actions of the virtual human have already been deformed or deviated. In the second key frame, although the real-time interaction actions of the virtual human have not been deformed or deviated, due to the insufficient force application in the second key frame, the real-time interaction actions of the virtual human in the subsequent first key frame will be deformed or deviated.

[0120] By projecting the NG key frames at the NG occurrence positions in the mixed reality scene to restore the incorrect actions of the virtual human when completing real-time interaction actions, the trainee can approach the virtual human image of the NG key frame, observe from multiple perspectives, and combine the first prominent identifier to find their own action defects, so as to pay attention to and improve them subsequently. The first prominent identifier may include the first color based on the first shape contour. Among them, the first shape contour may be the contour of the muscle groups exerting force, or the contour of a part of the virtual human's limb where the action is deformed or deviated.

[0121] As a preferred solution rather than a limitation, please refer to Figure 9 As shown, the driving of the AI coach for action demonstration includes:

[0122] S1651, obtaining the training actions of the current training content based on the training scenario model.

[0123] The determination of the specific training actions in this embodiment can be input by the trainee or obtained by analyzing the real-time interaction actions of the virtual human.

[0124] S1652, obtaining the frame-by-frame contour model of the AI coach.

[0125] In this step, according to the training actions, the corresponding frame-by-frame contour model of the AI coach is retrieved in the training scenario model, where the frame-by-frame contour model includes data such as the posture, height, and angle of the AI coach model.

[0126] S1653, determining the demonstration start position, demonstration movement trajectory, and demonstration movement speed of the AI coach based on the origin coordinates of the trainee, the digital virtual training tool, and the training elements.

[0127] The demonstration start position of the AI coach in this embodiment can be at the origin coordinates or adjusted according to the origin coordinates. Exemplarily, when the distance from the origin coordinates to the training element (such as the high jump bar) is relatively close, resulting in insufficient running-up distance, the demonstration start position of the AI coach can be adjusted according to the actual distance required for the running-up. Similarly, the demonstration movement trajectory of the AI coach can be determined based on the demonstration start position, the digital virtual training tool, and the training elements, and then the training scenario model generates the demonstration movement speed personalized according to the demonstration start position, the demonstration movement trajectory, and the training actions. The demonstration movement speed can be different at different travel positions of the demonstration movement trajectory.

[0128] S1654, driving the AI coach to perform action demonstration based on the demonstration start position, the demonstration movement trajectory, the demonstration movement speed, and the frame-by-frame contour model.

[0129] The above steps provide a specific solution for the AI coach to perform action demonstrations, and obtain the training actions and the frame-by-frame contour model of the AI coach according to the training scenario model, so as to determine the demonstration starting position, demonstration movement trajectory, and demonstration movement speed of the AI coach. This enables the demonstration actions of the AI coach to closely interact with the digital virtual training tools and training elements in the current mixed reality scenario. At the same time, the demonstration starting position of the AI coach can be the same as or close to the position of the trainee, so as to further fit the real-time actions of the trainee, which is conducive to disassembling the training actions and improving the training effect.

[0130] As a preferred solution rather than a limitation, please refer to Figure 10 As shown, driving the AI coach to conduct training guidance includes:

[0131] S1655. Based on the current staying position of the trainee, determine the target position of the AI coach;

[0132] S1656. Control the AI coach to move from the current position to the target position;

[0133] S1657. Control the AI coach to conduct training guidance based on the NG force application points and the NG human body contour, and the training guidance includes voice information and text information.

[0134] The above steps S1655 - S1657 can determine the target position of the AI coach according to the real-time staying position of the trainee, control the movement of the AI coach, which is more in line with the actual scenario and is conducive to ensuring the efficient interaction between the trainee and the AI coach. At the same time, training guidance is provided for the trainee's NG force application points and NG human body contour through voice information and text information, further ensuring the efficiency of communication.

[0135] S17. When the real-time position of the AI coach exceeds the real area, perform a second prominent marking on the exceeded contour part.

[0136] The second prominent marking may include a second color based on the second shape contour. The second shape contour may be the partial limb contour of the AI coach's body that exceeds the real area. Through the second prominent marking, a clear sense of the boundary of the real area can be provided to the trainee, which is conducive to providing safety guidance to the trainee when the trainee is immersed in training.

[0137] In the embodiment of the present invention, the training elements of the simulation training scenario are projected onto the real-time environment of the real area to generate a mixed reality scenario; at least according to the real-time movement trajectory and posture change of the trainee, the virtual human substitute is driven to execute the real-time interaction of the digital virtual training tool and training elements based on the simulation training scenario; an AI coach is generated in the mixed reality scenario; based on the AI coach, action demonstration and training guidance are provided for the real-time interaction of the virtual human. The real-time movement trajectory and posture change of the trainee can be embodied through the virtual human substitute in the mixed reality scenario, and the actions of the virtual human are based on the actions of the trainee, which is convenient for the trainee to review and view later. At the same time, by generating an AI coach in the mixed reality scenario, compared with the existing method of playing holographic animations or films, the AI coach can provide action demonstration and training guidance for the trainee according to the real-time interaction actions of the virtual human, referring to the real-time movement trajectory and posture change of the trainee. Since the training elements and training tools are projected into the real-time environment, the AI coach also provides a prerequisite for the real-time interaction of the digital virtual training tool and training elements, improving the intelligent level of action training, ensuring the guidance effect, and enhancing the user experience.

[0138] It should be noted that in the embodiment of the present invention, multiple personal profiles can be generated according to the user account. Each trainee will accumulate training data during each training. The training data is compared with the training objective, and corresponding training content will be generated according to the latest training effect for each training, so as to assist the trainee to achieve the training objective in the next training, etc.

[0139] In the embodiment of the present invention, a digital three-dimensional map based on the on-site environment is generated. The digital three-dimensional map at least includes the first height information, the first position information, and the first shape information of the environmental object; based on the benchmark parameters of the simulation training scenario, the real area corresponding to the simulation training scenario is determined according to the digital three-dimensional map; the training elements of the simulation training scenario are projected onto the real-time environment of the real area to generate a mixed reality scenario; at least according to the real-time movement trajectory and posture change of the trainee, the virtual human substitute is driven to execute the real-time interaction of the digital virtual training tool and training elements based on the simulation training scenario. The first height information, the first position information, and the first shape information of each environmental object in the real environment can be used to judge the generation conditions of the mixed reality scenario, and the benchmark parameters of the simulation training scenario also ensure that the training venue can be projected in the real-time environment of the real area in equal proportion or with a smaller occupied space, making the generation of the mixed reality scenario safer and more efficient, and effectively ensuring the application of the MR mixed reality technology in the training scenario.

[0140] Please refer to Figure 10 As shown, the present invention also provides an action training device 100 based on MR mixed reality, including:

[0141] The first generation module 110 is configured to generate a digital three-dimensional map based on the on-site environment, where the digital three-dimensional map at least includes the first height information, the first position information, and the first shape information of the environmental object;

[0142] The determination module 120 is configured to determine the real area corresponding to the simulation training scenario according to the digital three-dimensional map based on the reference parameters of the simulation training scenario;

[0143] The second generation module 130 is configured to project the training elements of the simulation training scenario onto the real-time environment of the real area to generate a mixed reality scenario;

[0144] The processing module 140 is configured to drive the virtual human avatar to perform real-time interaction with the digital virtual training tools and training elements based on the simulation training scenario at least according to the real-time movement trajectory and posture change of the trainee.

[0145] The modules in this embodiment are the same as the corresponding steps in the above one or more method embodiments, and will not be elaborated here.

[0146] All the modules / units in this embodiment are the same as the corresponding steps in the above one or more method embodiments, and their logical relationships and working principles are also the same, and will not be elaborated here. Those skilled in the art can learn the corresponding virtual modules or units from the above method embodiments to make them correspond to the steps of the above method embodiments. The virtual modules / units not disclosed in this embodiment should also be regarded as the disclosed part of the present invention.

[0147] In the embodiment of the present invention, by generating a digital three-dimensional map based on the on-site environment, where the digital three-dimensional map at least includes the first height information, the first position information, and the first shape information of the environmental object; determining the real area corresponding to the simulation training scenario according to the digital three-dimensional map based on the reference parameters of the simulation training scenario; projecting the training elements of the simulation training scenario onto the real-time environment of the real area to generate a mixed reality scenario; driving the virtual human avatar to perform real-time interaction with the digital virtual training tools and training elements based on the simulation training scenario at least according to the real-time movement trajectory and posture change of the trainee. The first height information, the first position information, and the first shape information of each environmental object in the real environment can be used to judge the generation conditions of the mixed reality scenario, and the reference parameters of the simulation training scenario also ensure that the training venue can be projected in the real-time environment of the real area in equal proportion or with a smaller occupied space, making the generation of the mixed reality scenario safer and more efficient, and effectively ensuring the application of the MR mixed reality technology in the training scenario.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working processes of the systems, devices, and units described above, reference can be made to the corresponding processes in the foregoing method embodiments, which will not be elaborated here.

[0149] Another aspect of the embodiments of the present invention provides a motion training system based on MR mixed reality. The motion training system based on MR mixed reality includes: one or more motion training devices based on MR mixed reality;

[0150] The motion training device based on MR mixed reality is controlled by the motion training method based on MR mixed reality in one or more embodiments.

[0151] The training system of this embodiment can communicate with multiple MR devices simultaneously through a communication port, and through the unique identity identification (SN, MAC) of the MR device + the binding of the trainee account, it can accurately manage the training processes of multiple trainees simultaneously.

[0152] The embodiments of the present invention also provide a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the motion training method based on MR mixed reality in the above embodiments.

[0153] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above embodiments of the action training methods based on MR mixed reality. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0154] Alternatively, if the above-integrated unit is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the related technology can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the methods of the various embodiments of the present invention. And the aforementioned storage medium includes: various media such as a removable storage device, RAM, ROM, magnetic disk, or optical disc that can store program codes.

[0155] Corresponding to the above computer storage medium, in one embodiment, a computer device is also provided. The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. Wherein, when the processor executes the program, it implements the action training methods based on MR mixed reality in the above embodiments.

[0156] The computer device can be a terminal, and its internal structural diagram can be as Figure 11As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a motion training method based on MR mixed reality. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the outer shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0157] In the embodiment of the present invention, a digital three-dimensional map based on the on-site environment is generated. The digital three-dimensional map at least includes the first height information, the first position information, and the first shape information of the environmental object; based on the benchmark parameters of the simulation training scenario, the real area corresponding to the simulation training scenario is determined according to the digital three-dimensional map; the training elements of the simulation training scenario are projected onto the real-time environment of the real area to generate a mixed reality scenario; at least according to the real-time motion trajectory and posture change of the trainee, the virtual human double is driven to execute real-time interaction of the digital virtual training tools and training elements based on the simulation training scenario. The first height information, the first position information, and the first shape information of each environmental object in the real environment can be used to judge the generation conditions of the mixed reality scenario, and the benchmark parameters of the simulation training scenario also ensure that the training venue can be projected in the real-time environment of the real area in an equal proportion or with a smaller occupied space, making the generation of the mixed reality scenario safer and more efficient, and effectively ensuring the application of the MR mixed reality technology in the training scenario.

[0158] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0159] The above embodiments only represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the invention patent should be subject to the appended claims.

Claims

1. A motion training method based on MR mixed reality, characterized in that: include: Generate a digital three-dimensional map based on the on-site environment, the digital three-dimensional map including at least first height information, first position information and first shape information of the environmental object; Based on the reference parameters of the simulation training scene, determining a real area corresponding to the simulation training scene according to the digital three-dimensional map; Projecting the training elements of the simulation training scene into the real-time environment of the real area to generate a mixed reality scene; At least according to the real-time motion trajectory and posture changes of the trainee, the virtual human avatar is driven to perform real-time interaction with digital virtual training tools and training elements based on the simulation training scene.

2. The method according to claim 1, characterized in that The step of generating a digital three-dimensional map, generating a digital three-dimensional map based on a site environment, wherein the digital three-dimensional map includes at least first height information, first position information and first shape information of an environmental object, comprises: Determine the trainee's current position information as the origin coordinates; At least based on the camera and depth camera on the MR device to scan and shoot in real time to obtain the scene environment; Based on the origin coordinates, a digital three-dimensional map including the ground, first height information, first position information and first shape information of environmental objects is generated.

3. The method according to claim 2, characterized in that The step of determining the real area corresponding to the simulation training scene according to the digital three-dimensional map based on the benchmark parameters of the simulation training scene comprises: Obtaining the baseline parameters of the simulation training scene, at least determining the actual shape and size data of the training scene; Based at least on the actual shape and size data of the training scene, a realistic area corresponding to the simulated training scene is determined in the three-dimensional digital map.

4. The method according to claim 3, characterized in that The step of determining a real area corresponding to the simulated training scene in the three-dimensional digital map based at least on the actual shape and size data of the training scene comprises: Determining at least one survey route in the three-dimensional digital map based at least on the actual shape and size data of the training scene, the survey route comprising at least a starting point, a feature point, a random point, and an end point; Based on the starting point coordinates, obtaining second height information, second position information, and second shape information of the environmental object; Based on the random point coordinates, obtaining third height information, third position information, and third shape information of the environmental object; Based on the endpoint coordinates, acquiring fourth height information, fourth position information, and fourth shape information of the environmental object; Determine standard height information of the corresponding environmental object based on the first height information, the second height information, the third height information and the fourth height information of the same environmental object; Determine standard position information of the corresponding environmental object based on the first position information, the second position information, the third position information and the fourth position information of the same environmental object; Determine standard shape information of the corresponding environmental object based on the first shape information, the second shape information, the third shape information and the fourth shape information of the same environmental object; Based on the coordinates of the feature points, obtain the verified height information, verified position information and verified shape information of the environmental object; Based on the verified height information, verified position information and verified shape information of the environmental object, verifying the standard height information, standard position information and standard shape information to determine the final height information, final position information and final shape information of the environmental object; Based on the final height information, final position information and final shape information of all environmental objects in the three-dimensional digital map, the actual shape and size data of the training scene, and the activity range of the trainee, a real area corresponding to the simulation training scene is determined in the three-dimensional digital map.

5. The method according to claim 4, characterized in that The step of driving the virtual human avatar to perform real-time interaction with digital virtual training tools and training elements based on the simulation training scene at least according to the real-time motion trajectory and posture changes of the trainee includes: Based on the training content, establish a digital human body execution scale joint model and training scene model, and determine the digital virtual training tools and training elements; Based on at least the digital three-dimensional map of the first height information, the first position information and the first shape information of the ground and the environmental objects, the trainee's body information is determined according to the digital human execution scale joint model, and a virtual human corresponding to the trainee's body information is generated; Based on the trainee's MR device and motion sensing device, the trainee's real-time motion trajectory and posture changes are determined according to the training scene model and body information; Based on real-time motion trajectory and posture changes, the simulation drives the virtual human to interact with digital virtual training tools and training elements in a simulated mixed reality scene.

6. The method according to claim 5, characterized in that After the step of driving the virtual human avatar to perform real-time interaction with the digital virtual training tools and training elements based on the simulation training scene at least according to the real-time motion trajectory and posture changes of the trainee, the method further includes: Generate an AI coach in a mixed reality scenario; Based on the real-time interaction of AI coaches with virtual humans, action demonstrations and training guidance are provided.

7. A motion training device based on MR mixed reality, characterized in that: include: A first generating module, used to generate a digital three-dimensional map based on the on-site environment, wherein the digital three-dimensional map includes at least first height information, first position information and first shape information of the environmental object; A determination module, configured to determine a real area corresponding to the simulation training scene according to the digital three-dimensional map based on the reference parameters of the simulation training scene; A second generating module is used to project the training elements of the simulation training scene into the real-time environment of the real area to generate a mixed reality scene; The processing module is used to drive the virtual human avatar to perform real-time interaction with digital virtual training tools and training elements based on the simulation training scene at least according to the real-time movement trajectory and posture changes of the trainee.

8. A motion training system based on MR mixed reality, characterized in that: The MR mixed reality-based motion training system comprises: a MR mixed reality-based motion training device; The MR mixed reality-based motion training device is controlled by the MR mixed reality-based motion training method described in any one of claims 1 to 7.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the motion training method based on MR mixed reality is implemented as described in any one of claims 1 to 6.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the MR mixed reality-based motion training method as described in any one of claims 1 to 6 is implemented.

Images