Virtual and augmented reality personalized and customized fitness training activity or game, methods, devices, and systems

Abstract

To provide a method of supporting exercise and training in a virtual reality (VR) environment in relation to VR.SOLUTION: A method includes determining a customized action routine for a person to perform in a virtual world, wherein the person is wearing a virtual reality (VR) headset. An object associated with an event from the routine is presented to the person in the virtual world via the VR headset, and aspects of the person's virtual interaction with the object in the virtual world are rendered on the VR headset. The person's virtual interaction with the virtual object is analyzed and used to determine, in real-time, one or more next events for the routine.SELECTED DRAWING: Figure 6B

Description

Technical Field

[0001] [Copyright Notice] Part of the disclosure of this patent document contains subject matter that is subject to copyright protection. The copyright owner reserves all copyrights whatsoever in and to this patent document or this patent disclosure, except that no objection will be made to the reproduction of the same by anyone in the patent file or records of the United States Patent and Trademark Office. [Related Applications]

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 014,046, filed Apr. 22, 2020, the entire content of which is hereby incorporated by reference in its entirety for all purposes.

[0003] The present invention generally relates to virtual reality (VR), and more specifically, to methods, systems, and devices for supporting exercise and training in a VR environment.

Background Art

[0004] Virtual and augmented reality devices enable a user to view and interact with a virtual environment. The user can effectively immerse themselves in an unreal environment and interact with that environment. For example, a user can interact (e.g., play a game) in a virtual environment, where the user's real-world movements are translated into actions in the virtual world. Thus, for example, a user can simulate playing tennis or fencing, etc. in a virtual environment by their real-world movements.

[0005] A user can view their view of a virtual environment using a wearable VR / AR device such as a virtual reality (VR) headset or augmented reality (AR) glasses (collectively referred to as a head-mounted display (HMD)).

[0006] People should exercise regularly, and many people participate in group or individual exercise programs at gyms, schools, etc. In these programs, users typically receive instructions and guidance through an exercise routine by a person (e.g., a coach) who can monitor, instruct, and motivate the participants. A good coach or instructor may customize the routine for a user and may change the routine based on that user's performance. However, in some situations (e.g., during a pandemic quarantine period), a user may not be able to or may not want to participate in such programs.

[0007] It is desirable to provide a personalized exercise routine that receives instructions and guidance, which is an object of the present invention.

[0008] It is further desirable to monitor a user using such a routine and, if necessary, change the routine as needed, which is a further object of the present invention.

Summary of the Invention

[0009] The present invention is specifically defined in the claims and the following description. Preferred embodiments are specifically designated in the description of the dependent claims and various embodiments.

[0010] A system of one or more computers can be configured to perform certain operations or actions through software, firmware, hardware, or combinations thereof installed on the system that cause the actions to be performed on the system when it is operating. One or more computer programs can be configured to perform certain operations or actions by including instructions that cause the actions to be performed on the device when executed by a data processing device.

[0011] One general aspect includes a computer-implemented method comprising: (a) determining calibration information regarding a person wearing a user device in a real-world environment, wherein the user device may comprise a virtual reality (VR) headset worn by the person; (b) determining, based on the calibration information, a routine for the person, wherein the routine may include a sequence of events; (c) presenting, via the VR headset, an object associated with an event from the routine to the person in a virtual world; (d) rendering, in the virtual world on the VR headset, an aspect of a virtual interaction of the person with the object; (e) analyzing the virtual interaction with the object; and (f) determining, based on the analysis, one or more next events for the routine in real time.

[0012] Implementations and / or embodiments may include one or more of the following features, alone or in combination. · The method may comprise repeating operations (c), (d), (e), and (f) until the routine is performed or the person stops. · The method, wherein at least one VR hand-held controller is worn or held by the person. · The method, comprising directly tracking at least one of the person's hands. · The method, wherein the calibration information includes one or more of height, vertical reach, left range information, right range information, and squat information. · The method, wherein the routine may include events in a time series. · The method, wherein the user device may include two VR hand-held controllers worn or held by the person. · Analyzing the virtual interaction with the object may include recognizing and analyzing the movement of the person to determine movement data and mapping the movement data to the virtual world. · The method may include providing feedback to the person based on the analysis. · The method may include determining the performance level of the person. · The performance level is based on the person's virtual interaction with the object relative to the expected or desired interaction with the object. · The one or more next events for the routine are determined based on the performance level of the person. The performance level is also based on physiological data associated with the person. · An event has a corresponding object associated with the event. · The object may include one of a hit object, a squat object, or a lunge object. · The hit object may include a hit direction. · The hit object may include a tail. · When the object may include a hit object, analyzing the virtual interaction with the object may include determining whether the person virtually hits the object in the hit direction using a virtual baton controlled by the at least one VR handheld controller and whether the person follows through with the virtual hit. · When the object may include a hit object, analyzing the virtual interaction with the object may include determining whether the person virtually hits the object in the hit direction using a virtual baton controlled by the at least one VR handheld controller. · The squat object may include a symmetric triangle. · The lunge object may include an asymmetric triangle. · When the object may include a hit object, analyzing the virtual interaction with the object may include determining whether the person virtually hits the object using a virtual baton controlled by the at least one VR handheld controller. · When the object may include a squat object, analyzing the virtual interaction with the object may include determining the amount of squat by the person while the squat object passes through the person in the virtual world. · When the object may include a lunge object, analyzing the virtual interaction with the object may include determining the amount and direction of the lunge by the person while the lunge object passes through the person in the virtual world. · When the squat object or the lunge object may include a hold shape, analyzing the virtual interaction with the object may include determining the amount and duration of the squat or lunge by the person while the object passes through the person in the virtual world. · The object has a source location associated with the object, and the object appears at the source location in the virtual world and is presented on the VR headset so as to come from the source location towards the person. · The object has a speed associated with the object, and the object is presented on the VR headset such that it appears in the virtual world and arrives towards the person at the speed, the method. · The routine may include a fitness or exercise routine, the method.

[0013] Another general aspect includes a computer-implemented method comprising: (a) providing a user device in a real-world environment, the user device may comprise a virtual reality (VR) headset worn by a person, the VR headset being capable of providing an image of a scene and an object to the person through the VR headset to generate a visual representation of a virtual world; (b) determining a visual routine for the person to visually recognize in the virtual world, the visual routine may include a plurality of objects with which the person may interact using a specified action; (c) presenting a first object of the plurality of objects to the person in the virtual world; (d) enabling the person to virtually interact with the first object in the virtual world; (e) measuring a manner of the virtual interaction between the person and the presented first object; (f) analyzing the measured manner of the virtual interaction; and (g) presenting a second object of the plurality of objects to the person in the virtual world in real time based on the analysis.

[0014] Implementations and / or embodiments may include one or more of the following features, alone and / or in combination.

[0015] The computer-implemented method may comprise repeating operations (c) to (g) for a large number of objects among the plurality of objects of the visual routine. The plurality of objects may include a series of objects. The specified action by the user may include a swinging motion for hitting the first object, and the measured aspect may include the direction of the swinging motion with respect to the first object. The specified action by the user may include a swinging motion for hitting the first object, and the measured aspect may include the speed of the swinging motion with respect to the first object. The specified action by the user may include a swinging motion for hitting the first object, and the measured aspect may include the length of the swinging motion with respect to the first object. The first object may include a triangular shape, the specified action by the user may include a squatting motion, and the measured aspect may include the vertical displacement of the squatting motion with respect to the apex of the triangular shape. The first object may include a triangular shape, the specified action by the user may include a running motion, and the measured aspect may include the relative position of the user's head with respect to the location of the apex of the triangular shape. The first object includes a visual marker that indicates to the user the direction of the swinging action with respect to the first object. The first object is generated at a first portal within the virtual world, and the first object includes a visual marker that indicates to the user the direction of the swinging action with respect to the first object.

[0016] Another general aspect includes a computer implementation comprising: (a) providing a user device in a real-world environment, the user device may comprise a virtual reality (VR) headset worn by a person, the VR headset being capable of providing images of scenes and objects to the person through the VR headset to generate a visual representation of a virtual world. The method further comprises: (b) determining a visual routine for the person to view in the virtual world using the VR headset, the visual routine may include a plurality of triangles, each of the triangles having a defined shape and vertices. The method also comprises: (c) determining a position of the VR headset in the virtual world during the visual routine. The method further comprises: (d) presenting a first triangle of the plurality of triangles to the person in the virtual world based on the determination in (c). The method also comprises: (e) enabling the person to virtually interact with the first triangle presented in the virtual world. The method further comprises: (f) comparing the position of the VR headset with the locations of the vertices of the first triangle. The method also comprises: (g) recording the result of the comparison in (f).

[0017] The implementation and / or embodiment may include one or more of the following features, alone and / or in combination.

[0018] The enabling in (e) may have the method enabling the person to virtually interact with the first triangle presented in the virtual world by positioning their body in the real world until the VR headset is positioned below and adjacent to the location of the vertices of the first triangle presented.

[0019] Another general aspect includes a computer-implemented method comprising: (a) providing a user device in a real-world environment, the user device may comprise a virtual reality (VR) headset worn by a person, the VR headset being capable of providing images of scenes and objects to the person through the VR headset to generate a visual representation of a virtual world. The method further comprises: (b) determining a visual routine for the person to view in the virtual world using the VR headset, the visual routine may include a series of objects, the objects being generated from one of a first portal and a second portal, each of the objects advancing towards the person in the virtual world. The method also comprises: (c) presenting a first object from the first portal to the person in the virtual world. The method further comprises: (d) providing an indication of the location of the second portal with respect to the first object. The method also comprises: (e) presenting a second object from the second portal to the person in the virtual world.

[0020] Implementations and / or embodiments may include one or more of the following features, alone and / or in combination. The computer-implemented method may comprise repeating operations (c)-(e) for a number of objects of the visual routine. In (d), the indication with respect to the first object may include an arrow shape. In (d), the indication with respect to the first object may include a triangle shape.

[0021] Implementations of the described technology may include hardware, a method or process, or computer software on a computer-accessible medium.

[0022] Other embodiments of these aspects described above include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices each configured to perform the actions of the method.

[0023] One skilled in the art will appreciate that any method described above or below, and / or claimed and described as a sequence of acts or operations, is not limited in the sense of the order of acts or operations.

[0024] The following is a list of embodiments of a method or process. They are indicated using the letter "P". Whenever such an embodiment is referred to, this is always done by referring to the "P" embodiment. P1. (A) Determining a routine for a person wearing a user device in a real-world environment, the user device comprising a virtual reality (VR) headset worn by the person; (B) Presenting an object associated with an event from the routine to the person in a virtual world via the VR headset; (C) Rendering the manner of the person's virtual interaction with the object in the virtual world on the VR headset; (D) Analyzing the virtual interaction with the object; (E) Based on the analysis, determining one or more next events for the routine in real time; A computer-implemented method comprising: P2. The method according to any of the preceding embodiments, further comprising repeating operations (B), (C), (D), and (E) until the routine is performed or the person stops. P3. Further comprising determining calibration information regarding a person wearing the user device in the real-world environment, wherein the determining in (A) is based on the calibration information, the method according to any of the foregoing embodiments. P4. The method according to embodiment P3, wherein the calibration information includes one or more of height, vertical reach, left range information, right range information, and squat information. P5. The method according to any of the foregoing embodiments, wherein the routine includes time-series events. P6. The method according to any of the foregoing embodiments, wherein the user device includes at least one VR handheld controller worn or held by the person. P7. The method according to embodiment P6, wherein the at least one VR handheld controller is rendered in the virtual world based on the expected virtual interaction of the person with the object. P8. Analyzing the virtual interaction with the object includes recognizing and analyzing the movement of the person to determine movement data, and mapping the movement data to the virtual world, the method according to any of the foregoing embodiments. P9. The method according to any of the foregoing embodiments, further comprising providing feedback to the person based on the analysis. P10. The method according to any of the foregoing embodiments, further comprising determining a performance level of the person. P11. The method according to any of the foregoing embodiments, wherein the performance level is used to determine the one or more next events for the routine. P12. The method according to any of the foregoing embodiments, wherein the performance level is based on the virtual interaction of the person with the object relative to the expected or desired interaction with the object. P13. The one or more next events for the routine are determined based on the performance level of the person, according to the method described in any of the foregoing embodiments. P14. The event has a corresponding object associated with the event, according to the method described in any of the foregoing embodiments. P15. The object includes one of a hit object, a squat object, or a run object, according to the method described in any of the foregoing embodiments. P16. The hit object includes a hit direction, according to the method described in any of the foregoing embodiments. P17. The hit object includes a tail, according to the method described in any of the foregoing embodiments. P18. The squat object includes a symmetric triangle, according to the method described in any of the foregoing embodiments. P19. The run object includes an asymmetric triangle, according to the method described in any of the foregoing embodiments. P20. The object has a source location associated with the object, and the object appears at the source location in the virtual world and is presented on the VR headset so as to come towards the person from the source location, according to the method described in any of the foregoing embodiments. P21. The object has a speed associated with the object, and the object appears in the virtual world and is presented on the VR headset so as to come towards the person at the speed, according to the method described in any of the foregoing embodiments. P22. When the object includes a hit object, analyzing the virtual interaction with the object includes determining whether the object has been virtually hit by the person, according to the method described in any of the foregoing embodiments. P23. Analyzing the virtual interaction with the object includes determining whether the object has been virtually hit using a virtual baton controlled by at least one VR handheld controller worn or held by the person, according to any of the preceding embodiments. P24. When the object includes a hit object, analyzing the virtual interaction with the object includes determining whether the object has been virtually hit by the person in the hit direction, according to any of the preceding embodiments. P25. When the object includes a hit object, analyzing the virtual interaction with the object includes determining whether the object has been virtually hit in the hit direction using a virtual baton controlled by at least one VR handheld controller worn or held by the person, according to any of the preceding embodiments. P26. When the object includes a hit object, analyzing the virtual interaction with the object includes determining whether the object has been virtually hit in the hit direction using a virtual baton controlled by the at least one VR handheld controller, and whether the person has followed through with the virtual hit, according to any of the preceding embodiments. P27. The performance level is also based on physiological data associated with the person, according to any of the preceding embodiments. P28. When the object includes a squat object, analyzing the virtual interaction with the object includes determining the amount of squat by the person while the squat object passes through the person in the virtual world, according to any of the preceding embodiments. P29. When the object includes a running object, analyzing the virtual interaction with the object includes determining the amount and direction of the run by the person while the running object passes through the person in the virtual world, according to any of the foregoing embodiments of the method. P30. The running object includes a triangle, and determining the amount and / or direction of the run is based on the position of the person's head in the virtual world relative to the vertices of the triangle, according to any of the foregoing embodiments of the method. P31. The squat object or the running object includes a hold shape, and analyzing the virtual interaction with the object includes determining the amount and duration of the squat or run by the person while the object passes through the person in the virtual world, according to any of the foregoing embodiments of the method. P32. The object is presented in the virtual world as coming from a first location in the virtual world, and the object includes an indication of a second location in the virtual world for a subsequent object to be presented, according to any of the foregoing embodiments of the method. P33. The indication of the second location includes an arrow, according to the method described in embodiment P32. P34. The routine includes a fitness or exercise routine, according to any of the foregoing embodiments of the method. P35. (A) Providing a user device in a real-world environment, the user device comprising a virtual reality (VR) headset worn by a person, the VR headset being capable of providing images of a scene and objects to the person through the VR headset to generate a visual representation of a virtual world; (B) Determining a visual routine for the person to visually recognize in the virtual world using the VR headset, the visual routine including a plurality of objects with which the person can interact using the specified action; (C) Presenting a first object among the plurality of objects to the person in the virtual world; (D) Enabling the person to virtually interact with the first object in the virtual world; (E) Measuring the mode of virtual interaction between the person and the presented first object; (F) Analyzing the measured mode of virtual interaction; (G) Based on the analysis, presenting a second object among the plurality of objects to the person in the virtual world in real time A computer-implemented method comprising: P36. The computer-implemented method according to embodiment P35, further comprising repeating operations (C) to (G) for a number of objects among the plurality of objects of the visual routine. P37. The computer-implemented method according to any one of embodiments P35 to P36, wherein the plurality of objects includes a series of objects. P38. The computer-implemented method according to any one of embodiments P35 to P37, wherein the specified action by the user includes a swinging motion for hitting the first object, and the measured mode includes the direction of the swinging motion with respect to the first object. P39. The computer-implemented method according to any one of embodiments P35 to P38, wherein the specified action by the user includes a swinging motion for hitting the first object, and the measured mode includes the speed of the swinging motion with respect to the first object. P40. The specified action by the user includes a swinging motion for hitting the first object, and the measured aspect includes the length of the swinging motion with respect to the first object. The computer-implemented method according to any one of embodiments P35 to P40. P41. The first object includes a triangular shape, the specified action by the user includes a squatting motion, and the measured aspect includes the vertical displacement of the squatting motion with respect to the apex of the triangular shape. The computer-implemented method according to any one of embodiments P35 to P40. P42. The first object includes a triangular shape, the specified action by the user includes a running motion, and the measured aspect includes the relative position of the user's head with respect to the location of the apex of the triangular shape. The computer-implemented method according to any one of embodiments P35 to P41. P43. The first object includes a visual marker that indicates to the user the direction of the swinging action with respect to the first object. The computer-implemented method according to any one of embodiments P35 to P42. P44. The first object is generated at a first portal within the virtual world, and the first object includes a visual marker that indicates to the user the direction of the swinging action with respect to the first object. The computer-implemented method according to any one of embodiments P35 to P43. P45. (A) Providing a user device in a real-world environment, the user device comprising a virtual reality (VR) headset worn by a person, the VR headset being capable of providing images of scenes and objects to the person through the VR headset to generate a visual representation of a virtual world; (B) Determining a visual routine for the person to view in the virtual world using the VR headset, the visual routine including a plurality of triangles, each of the triangles having a defined shape and apexes; (C) During the visual routine, determining the position of the VR headset in the virtual world; (D) Presenting, in the virtual world, a first triangle among the plurality of triangles to the person based on the determination in (C); (E) Enabling the person to virtually interact with the first triangle presented in the virtual world; (F) Comparing the position of the VR headset with the locations of the vertices of the first triangle; (G) Recording the result of the comparison in (F); A computer-implemented method comprising the above. P46. The enabling in (E) has the person positioning their body in the real world until the VR headset is positioned below and adjacent to the location of the vertices of the first triangle presented, thereby enabling the person to virtually interact with the first triangle presented in the virtual world, according to the method described in Embodiment P45. P47. (A) Providing a user device in a real-world environment, the user device comprising a virtual reality (VR) headset worn by a person, the VR headset being capable of generating a visual representation of a virtual world by providing images of scenes and objects to the person through the VR headset; (B) Determining a visual routine for the person to view in the virtual world using the VR headset, the visual routine including a series of objects, the objects being generated from one of a first portal and a second portal, each of the objects advancing towards the person in the virtual world; (C) Presenting a first object from the first portal to the person in the virtual world; (D) providing an indication of the location of the second portal with respect to the first object; (E) presenting a second object from the second portal to the person in the virtual world; A computer-implemented method comprising: P48. The computer-implemented method according to embodiment P47, further comprising repeating operations (C), (D), and (E) for a number of objects of the visual routine. P49. The computer-implemented method according to any one of embodiments P47 to P48, wherein in (D), the indication with respect to the first object includes an arrow shape. P50. The computer-implemented method according to any one of embodiments P47 to P49, wherein in (D), the indication with respect to the first object includes a triangle shape.

[0025] The following is an embodiment of a device indicated by the letter "D". D51. (a) Hardware having a memory and at least one processor; (b) A service executed on the hardware, the service being configured to execute the method according to any one of embodiments P1 to P50 of the preceding method; A device comprising:

[0026] The following is an embodiment of a product indicated by the letter "M". M52. A product comprising a non-transitory computer-readable medium storing computer-readable instructions, the computer-readable instructions having instructions for implementing a computer-implemented method, the method being operable on a device comprising hardware having a memory and at least one processor, executing a service on the hardware, and the method including the method according to any one of embodiments P1 to P50 of the preceding method.

[0027] The following is an embodiment of a computer-readable recording medium indicated using the character "R". R53. A non-transitory computer-readable recording medium that stores one or more programs which, when executed, cause one or more processors to execute the method according to any one of the embodiments P1 to P50 of the preceding method.

[0028] The above features are further illustrated in the examples in this specification together with additional details of the present invention, which are intended to further illustrate the present invention but not to limit its scope in any way.

Brief Description of the Drawings

[0029] Considering the following description and the appended claims with reference to the accompanying drawings, the objects, features and characteristics of the present invention, as well as the methods of operation and functions of the related elements and the economy of the combination and manufacture of the parts, will become more apparent. All of these accompanying drawings form part of this specification.

[0030]

Figure 1

[0031]

Figure 2

[0032]

Figure 3

[0033]

Figure 4A

Figure 4B

[0034]

Figure 5a

Figure 5b

Figure 5c

Figure 5d

Figure 5e

Figure 5f

Figure 5g

[0035]

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 6E

Figure 6F

Figure 6G

Figure 6H

[0036]

Figure 7A

Figure 7B

Figure 7C

Figure 7D

Figure 7E

Figure 7F

[0037]

Figure 8A

Figure 8B

Figure 8C

Figure 8D

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 9E

Figure 9F

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 10F

Figure 10G

Figure 10H

Figure 11A

Figure 11B

Figure 11C

Figure 11D

Figure 12A

Figure 12B

Figure 12C

Figure 13A

Figure 13B

Figure 13C

Figure 14A

Figure 14B

Figure 14C

Figure 15A

Figure 15B

[0038]

Figure 16

Embodiments for Carrying Out the Invention

[0039] [Terms and Abbreviations] As used herein, unless otherwise used, the following terms or abbreviations have the following meanings.

[0040] "AR" means augmented reality.

[0041] "VR" means virtual reality.

[0042] "Mechanism" refers to any device, process, routine, service, or combination thereof. The mechanism may be implemented in hardware, software, firmware, using application-specific devices, or any combination thereof. The mechanism may be integrated into a single device or distributed across multiple devices. The various components of the mechanism may be located in the same place or distributed. The mechanism may be formed from other mechanisms. Generally, as used herein, the term "mechanism" may therefore be considered an abbreviated expression of the terms device and / or process and / or service. [Description]

[0043] In the following, exemplary embodiments of the present invention will be described with reference to the figures. These examples are provided to provide a further understanding of the present invention without limiting the scope of the present invention.

[0044] In the following description, a series of features and / or events will be described. Those skilled in the art will recognize that, unless required by the context, the order of features and events is not important for the resulting configuration and its effects. Furthermore, it will be apparent to those skilled in the art that, regardless of the order of features and events, the presence or absence of a time delay between events may exist between some or all of the events described.

[0045] It will be understood that modifications may be made to the foregoing embodiments of the present invention while still falling within the scope of the present invention. Alternative features that perform the same, equivalent, or similar function may replace the features disclosed herein unless otherwise stated. Therefore, unless otherwise stated, each feature disclosed represents one example of a general series of equivalent or similar features.

[0046] Here, a system 100 that supports a real-time virtual reality (VR) environment for a fitness training system personalized and customized by virtual and augmented reality is described with reference to FIG. 1. In FIG. 1, a person (VR user) 102 in a real-world environment or space 112 uses a VR device or headset 104 to view and interact with a virtual environment. The VR headset 104 may be connected (wired and / or wirelessly) to a training system 106 via, for example, an access point 108 (such as a Wi-Fi (registered trademark) access point, etc.). Since the user's activity may include many movements, the VR headset 104 is preferably wirelessly connected to the access point 108. In some cases, the VR headset 104 may be connected to the training system 106 via a user device or computer system (not shown). Although shown as a separate component, in some embodiments, the access point 108 may be incorporated into the VR headset 104.

[0047] Sensors (not shown in the figure) within the VR headset 104 and / or other sensors 110 in the user's environment may track the actual movements (such as head movements, etc.) of the VR user and other information. The VR headset 104 preferably provides user tracking without using external sensors. Here, in a preferred implementation, the VR headset 104 is an Oculus Quest headset made by Facebook Technologies, LLC.

[0048] Tracking or telemetry data from the VR headset 104 may be provided to the training system 106 in real time (as all or part of the data 118).

[0049] Similarly, data from the sensors 110 may also be provided to the training system 106 (for example, via the access point 108).

[0050] User 102 may also have one or two handheld devices 114-1, 114-2 (collectively, handheld devices and / or controllers 114) (e.g., Oculus Touch Controllers). Information about hand movements and / or control information from the handheld controllers 114 may be provided to the training system 106 along with the data 118 (e.g., via the access point 108).

[0051] In some embodiments, information about hand movements and / or control information from the handheld controllers 114 may be provided to the VR headset 104 or to another computing device, and the other computing device may then provide that information to the training system 106. In such a case, the handheld controllers 114 may communicate wirelessly with the VR headset 104.

[0052] In some embodiments, at least a portion of the information about the user's hand movements may be determined by tracking one or both of the user's hands (e.g., if the user does not have the handheld controllers 114 on one or both of their hands, the hand without the controller may be directly tracked, e.g., using 3D tracking).

[0053] Although one or two handheld controllers 114 are described herein as being used, one of ordinary skill in the art, upon reading this description, will understand that the user may not have a handheld controller, or may have only one. Further, even if the user has a handheld controller in their hand(s), the hand(s) may similarly (or alternatively) be directly tracked.

[0054] The VR headset 104 presents a view 124 to the VR user 102 that corresponds to the virtual or augmented environment of that VR user.

[0055] Preferably, the view 124 of the virtual environment of the VR user is shown as if viewed from the location, viewpoint, and orientation of the VR user 102. The view 124 of the VR user may be provided as a VR view or as an augmented view (e.g., an AR view).

[0056] In some embodiments, the user 102 may perform activities such as an exercise routine or a game in the virtual environment of the VR user. The training system 106 may provide exercise routine information to the VR headset 104. Here, in a preferred embodiment, the activity system 126 may provide a so-called beatmap and / or other information 128 to the headset (e.g., via the network 119 and the access point 108).

[0057] As the user progresses through an activity such as an exercise routine, the VR headset 104 may store information regarding the position and orientation of the VR headset 104 and the position and orientation of the controllers 114 for the user's left and right hands.

[0058] In this implementation, the user's activity (and beatmap) is divided into multiple sections (e.g., 20 - second sections), and information is collected and stored at a high frequency (e.g., 72 Hz) within the section. The VR headset 104 may also store information regarding the location of targets, portals, and all objects that are temporally variable, where they are in space, whether any have been hit, etc., at the same or a similar frequency. This collected information enables the fitness system to evaluate and / or recreate the scene at any point in time within the space of that section.

[0059] The information collected may then be transmitted to the training system 106, preferably in real time, as all or part of the data 118, as the user's activity / workout continues, and some of these sections may be transmitted to the training system 106 during the activity / workout. The data 118 provided to the training system 106 preferably includes beatmap information.

[0060] The training system 106 may be part of the backend / cloud framework 120. [Training System]

[0061] As will be described in more detail below, in some implementations / embodiments, the fitness training system provides the user with a VR training routine customized to the individual, tracks the user as the user executes the routine (in VR), changes the routine if necessary, and provides guidance to the user. The routine may include the user (virtually) interacting with various objects, and the system may monitor and evaluate the user's interactions and movements to determine possible changes to the routine. The system may use physiological data (e.g., heart rate data) to evaluate the user during the routine.

[0062] Referring to FIG. 2, the training system 106 is one or more servers having a computer system (discussed below), e.g., a processor 202, a memory 204, a communication mechanism 206, etc. One or more video creation programs 210 are executed on the training system 106. The training system 106 may store data in one or more data structures 224 in the memory 204 and / or one or more databases (not shown), and retrieve data therefrom. The database may include a user database that stores and maintains information about the users of the system.

[0063] Although only one user, user 102, is shown in FIG. 1, it should be recognized that the video training system 106 may interact with multiple users simultaneously. It should also be recognized that the following description of the operation of the training system 106 for one user extends to multiple users.

[0064] The training program 210 of the training system 106 may include a data collection mechanism 212, a motion / tracking mechanism 214, a mapping and conversion mechanism 216, a calibration mechanism 218, a routine generation mechanism 220, and a routine evaluation mechanism 222.

[0065] The data structure 224 may include a routine data structure 226 and a user data structure 228.

[0066] During operation, the data collection mechanism 212 obtains data 118 (FIG. 1) from the user (e.g., user 102 in FIG. 1). The data 118 may include at least a portion of information regarding user motion / telemetry data, targets, portals, and the locations of objects that are temporally variable, where they are in space, whether any were hit, where and how strongly they were hit, etc.

[0067] The motion / tracking mechanism 214 determines or approximates the actual motion of the user in the user's real-world space 112 from that data. The user's motion may be provided relative to the 3D coordinate system 116 of the user's real-world space 112. If the data 118 includes data from the user's handheld controller 114, the motion / tracking mechanism 214 may also determine the motion of one or both of the user's hands in the user's real-world space 112. In some cases, the user's headset 104 may provide the actual 3D coordinates of the user in the real-world space 112.

[0068] The motion / tracking mechanism 214 may determine or extrapolate aspects of the user's motion based on machine learning (ML) or other models of user motion. For example, the machine learning mechanism may be trained to recognize specific motions and / or types of motion, and may then be used to recognize those motions based on the data 118 provided by the user 102.

[0069] Referring to FIGS. 2 and 3, the mapping and transformation mechanism 216 (FIG. 2) may take in motion / tracking data (as determined by the motion / tracking mechanism 214) and transform that data from the real-world coordinate system 116 in the user's real-world space 112 to corresponding 3D coordinates in the virtual-world coordinate system 314 in the virtual world 312.

[0070] One of ordinary skill in the art, upon reading this description, will understand that the mapping and transformation mechanism 216 may operate prior to or in conjunction with the motion / tracking mechanism 214. As with all of the mechanisms described herein, logical boundaries are used to assist in the description and are not intended to limit the scope of the present invention.

[0071] For the purposes of this description, the user's motion data in the real-world space 112 is referred to as the user's real-world motion data, and the user's motion data in the virtual-world space 312 is referred to as the user's virtual motion data.

[0072] In some embodiments, the training system 106 may also receive or have other user data (e.g., physiological data, etc.), and may use a portion of the physiological data (e.g., heart rate, temperature, sweating level, respiratory rate, etc.) to determine or evaluate the user's motion and actions in the virtual space. Such physiological data may be worn by the user and / or acquired by one or more sensors 121 that monitor the user. The sensor 121 may be incorporated into another device such as a wristwatch worn by the user. For example, the sensor 121 may include a heart rate monitor included in an Apple Watch worn by the user.

[0073] The training system 106 may be placed in the same location as the user (e.g., in the same room), or may be placed entirely or partially in other locations. For example, the training system 106 may be located in a location different from the user, in which case the user's data 118 may be transmitted to the training system 106 via the network 119 (e.g., the Internet). In a preferred case, the user's data 118 is provided to the training system 106 when the data is generated (i.e., in real time), but in some cases, the user's data 118 may be collected and stored at the user's location and then transmitted to the training system 106. When placed away from the user and accessed via a network, the training system 106 may be regarded as a cloud-based system. [Routine]

[0074] As described above, the fitness training system may provide the user with a VR training routine customized according to the individual. The user's routine may be stored in the routine data structure 226 in the memory 204 of the training system 106.

[0075] Referring to FIG. 4A, the routine 400 may include time-series events 402. The event 402 may include a source location 404 and an object 406.

[0076] The object 406 may include a shape 408 and characteristics 410. Some characteristics may be shape-specific, as described below.

[0077] The shape 408 may be a hit shape 412 (e.g., a sphere or a circle, etc.), or a squat shape 414 (e.g., a symmetric triangle), or a lunge shape 416 (e.g., an oblique triangle or an asymmetric triangle).

[0078] The lunge shape 416 may have a lunge direction 418 (left or right) and, therefore, may be a left lunge shape or a right lunge shape.

[0079] The squat shape 414 or the lunge shape 416 may also include "hold" shapes 420, 422 that may include a hold duration (not shown).

[0080] The characteristics 410 of the shape may include its speed 411 (i.e., the speed at which the object or shape approaches the user in VR).

[0081] The hit shape 412 may include a direction indicator 424 indicating the direction in which the shape is to be hit. The hit shape 412 may include a color 426 or other indicator indicating which hand should be used to hit the shape.

[0082] It should be recalled that the user preferably has two controllers 114-1 and 114-2 (see FIG. 1). For example, as shown in FIG. 6A, in VR, the controllers are presented to the user (on their own display 124) as batons or sticks 614-1 and 614-2 of two colors (e.g., black and white). The user should attempt to hit the hit shape using the controller that matches the color of the shape. Therefore, for example, the user should attempt to hit a black hit shape object using their black controller and a white hit shape object using their white controller. The controllers may be presented to the user as batons or sticks, but those skilled in the art, upon reading this description, will understand that any real or virtual shape or object may be used to present the controller. Further, if the user has one controller or no controller, the system may directly track one or both of the user's hands (e.g., in 3D) and represent the user's hand as a hand in VR or as an object such as a stick, baton, etc.

[0083] The hit shape 412 may include an arc or tail 428 that indicates the type of hit (e.g., a flow or follow-through hit) to be used to hit the shape.

[0084] One of ordinary skill in the art, upon reading this description, will understand that different and / or other shapes and / or shape characteristics may be used.

[0085] Exemplary hit shapes 412A - 412H are shown in (A) - (H) of FIG. 5a, each showing a corresponding hit direction 424A - 424H. For example, the hit shape 412A may include a sphere having a triangular shape direction 424A indicating that the user should hit an object (sphere) in the direction of arrow A. Generally, the apex of the triangular direction indicator indicates the direction in which the object should be hit.

[0086] FIG. 5b (I) shows an example of a hit shape 412 - I having a corresponding direction indicator 424 - I and an arc 502. The arc 502 may extend from the object source 503. When the hit shape has an arc, the user should hit the object in the direction indicated by the object indicator 424 - I and should follow through the hit in a motion that draws the arc (as opposed to immediately stopping when the user contacts the object), as indicated by the arc 502. A similar situation is shown in the screenshot in FIG. 9B.

[0087] FIG. 5c (J) shows an exemplary squat shape 414 - J. As shown, the squat shape is preferably a symmetric triangle having an apex 415 - J. When the user is presented with the squat shape 414 - K (i.e., when the squat shape approaches the user in VR), the user should attempt to squat such that the user's head is below the apex 415 - J (e.g., inside the triangle) and passes through the inside of the squat shape.

[0088] (K) in FIG. 5d and (L) in FIG. 5e show a right ramp shape 414-K and a left ramp shape 414-L respectively, each having corresponding vertices 415-K and 415-L. As shown, the ramp shape is preferably an asymmetric triangle where the short side of the triangle indicates the desired ramp direction. When the ramp shape is presented to the user (i.e., when the ramp shape approaches the user in VR), the user should attempt to ramp in the direction indicated by the ramp shape such that the user's head is below (e.g., inside) the vertex of the triangle and passes inside the ramp shape. Thus, for example, in the case of the left ramp shape 414-L, the user should attempt to ramp to the left and ramp deep enough or low enough so as to pass the ramp shape below vertex 415-L to their own head. Exemplary squat and ramp shapes are shown in the screenshots in FIGS. 12A - 12C, FIGS. 13A - 13C, and FIGS. 14A - 14C.

[0089] (M) in FIG. 5f shows an exemplary squat and hold shape having a squat shape 422M and a hold portion 504. The squat portion 422M has the same role as the squat shape 414J described above. The hold portion 504 indicates that the user should attempt to hold the squat until the hold portion has passed (in VR). The ramp shape may also include a hold portion, in which case the user should attempt to hold their ramp until the ramp portion has passed.

[0090] The hold portion 504 may be represented by a series of repeating triangles, and each of the vertices of these triangles may be positioned at the same location as, or at a different location from, other adjacent triangles. In the latter case, the user will be instructed to squat and lunge in different directions based on the location of the vertices of each triangle as each triangle passes over the person. In each case, the person must position their head (VR headset) such that their head in the virtual world passes below and adjacent to each respective vertex of each passing triangle. Exemplary squat and hold shapes are shown in the screenshots in FIGS. 15A - 15B.

[0091] (N) of FIG. 5g is a diagram showing an exemplary portal transition according to an embodiment of the present invention. (N) of FIG. 5g shows a portal transition period during a workout, during which the first portal 750 may be deactivated (switched off, closed), while the adjacent second portal 752 may be activated (switched on, opened). The second portal 752 may be activated simultaneously with the closing of the first portal 750. The second portal 752 may open either to the left or right of the first portal 750 in the virtual world, and may be at the same height as the first portal 750 or higher or lower than it. During this transition, the baton 754 held by the player 756 is used to hit the final hit object 758 from the first portal 750. According to another aspect, the final hit object 758 from the first portal 750 that is closing may include a hit direction indicator 760 that is directed towards the adjacent second portal 752 that is opening. The direction indicated by the hit direction indicator 760 instructs or guides the player 756 to hit the hit object 758 in the direction of the arrow 762, and the arrow 762 points to the left in this example. In some implementations, the hit direction indicator 760 may point to the second portal 752 that is opening. Immediately adjacent to the hit object 758, an additional direction indicator 764 may be included between the hit object 758 and the newly opening second portal 752 and pointing in the direction of the newly opening second portal 752.

[0092] As should be appreciated, this feature enables the player 756 to be automatically oriented in front of an adjacent second portal 752 as soon as the second portal opens and potentially already sends its first second portal hit object 766 towards the player 756 (as indicated by arrow 768 in the drawing). Due to typical workout speeds, according to this system, this open portal indication feature enables the player to maintain a smooth and flowing motion during their workout with a lower risk of missing a hit object, by providing the player with "just in time" knowledge of the location of the (left or right) opening up of the portal that appears. Any number of new portals may open and close, and preferably, as explained above, only two portals are open simultaneously during portal transitions. While a hit direction indicator 760 and an additional direction indicator 764 may appear on the user's display, the arrows 762 and 768 in (M) of FIG. 5f are for illustrative purposes only and do not appear on the user's display.

[0093] As described herein, while the hit direction indicator and the additional direction indicator may be used to indicate a portal opening, one of ordinary skill in the art, upon reading this description, will understand that they may be used whether or not another portal is open or another portal is already open.

[0094] A screen shot showing the hit direction indicator and the additional direction indicator is also shown in FIG. 9E. [User Information]

[0095] As described, the training system 106 database may include a user database that stores and maintains information about the system's users. The training system 106 may use information about the user to customize and / or modify the user's routine. The user data may include some or all of the data shown in the user data structure 430 in FIG. 4B. In particular, the user data may include the user's height 432, vertical reach 433, age 434, gender 436, mass / weight 438, left reach 440, and right reach 442. The user may enter some of the information (e.g., their age and gender). As described below, some of the user information (e.g., height 432, vertical reach 433, left reach 440, and right reach 442) may be determined dynamically by the training system 106, for example, during a calibration phase.

[0096] The user data may include user statistics 444 from previous and current training sessions. The user statistics 444 may include a score 446 that includes a hit score 448, a squat score 450, a range score 452, and other scores 454. The statistics 444 may also include some of the user's physiological information 456 (e.g., the user's heart rate, temperature, sweating level, respiratory rate, etc., as determined from one or more sensors such as the sensor 121 shown in FIG. 1).

[0097] The user data may also include various other data 458. [Example]

[0098] Various exemplary interactions are shown with reference to FIGS. 6A-6H. These examples are shown from the user's perspective as presented on the user's display 124. The user is considered to be at the lower center of the display, and images corresponding to their controllers 614-1 and 614-2 may be visible in some cases.

[0099] Controllers 614-1 and 614-2 may be rendered on the user's display in different ways (e.g., a stick, a baton, a paddle, a bat, a racket, a glove, etc.), and the rendering may vary depending on the activity and / or the user's skill. For example, a less experienced user may be given a larger (or wider) paddle to use when hitting an object. As should be recognized, the system may also need to adjust the hit accuracy threshold to accommodate less experienced users, whereby more hits may be achieved. FIG. 6A shows a user's view of his own controllers 614-1 and 614-2 in this case as a stick or a baton. As used herein, the user's controllers 614-1 and 614-2 are, in some cases, used synonymously with their representations shown on the display. Therefore, for example, when the controller is shown as a stick, a description that an object was hit by the stick may be given even if the object was hit by the controller.

[0100] In the example of FIG. 6B, the system sends a hit object 612A from portal 602B towards the user as indicated by arrow A. Portal 602B is the source of the hit object. Hit object 612A has a hit direction indicated by triangle shape 624A. Hit object 612A has the same color as the user's left controller 614-1. The user should attempt to hit hit object 612A using his left controller 614-1 in the direction of arrow B. FIG. 9B shows a screenshot of an exemplary hit object.

[0101] When the user hits an object (in the VR space), the object will move in the hit direction and / or break or disintegrate (see, for example, FIGS. 10A - 10E, FIGS. 11A - 11C). Various VR effects may be used after the object is hit. For example, if the user misses the object, the object may appear to move / fly through the user or bounce to the side.

[0102] If the user successfully hits a hit object in the correct direction using the correct controller (baton), the user's hit score 448 may increase. The user may be given a higher score based on how hard the object was hit.

[0103] In the example in FIG. 6C, the system sends a hit object 612C from portal 602C towards the user. Since the object has the same color as the user's right baton (corresponding to the right controller 614 - 2), the user should attempt to hit the object 612C in the direction indicated by the triangular shape 624C.

[0104] Note that in the example in FIG. 6C, portal 602C is not in the same location as portal 602B in FIG. 6B. The portal corresponds to the source of the object, and a particular routine may use multiple portals at a number of separate locations.

[0105] In the example in FIG. 6D, the system sends a hit object 612D from the portal 602D towards the user. The hit object 612D has a tail (or arc) 625D. In order to successfully interact with this hit object, the user should attempt to hit the hit object 612D using their left baton in the direction of the triangular shape 624D (based on the matching colors of the object and the baton). The tail 625D preferably indicates that the user should follow through with the hit in a flow motion generally following a particular tail shape. FIGS. 9B and 9C show exemplary hit objects having tails.

[0106] In the example in FIG. 6E, the system sends a squat shape 614E from the portal 602E towards the user. In response, the user should attempt to squat into the object such that it appears to the user that they are passing through the object in the VR space. For example, by determining the height and position of the user's head, the system can determine how well the user has squatted and the system may adjust the user's squat score 450 accordingly. FIGS. 12A - 12C show examples of the user squatting to pass through a squat object.

[0107] In the example in FIG. 6F, the system sends a right lunge shape 614F from the portal 602F towards the user. In response, the user should attempt to lunge to the right such that it appears to the user that they are passing through the shape 614F in the VR space. For example, by determining the height and position of the user's head, the system can determine how well the user has lunged, and the system may adjust the user's lunge score 452 accordingly. Similarly, in the example in FIG. 6G, the system sends a left lunge shape 614G from the portal 602G towards the user. In response, the user should attempt to lunge to the left such that it appears to the user that they are passing through the shape 614G in the VR space. For example, by determining the height and position of the user's head, the system can determine how well the user has lunged, and the system may adjust the user's lunge score 452 accordingly. FIGS. 13A - 13C show an example where the user squats to pass through a squat object.

[0108] In the example in FIG. 6H, the system sends a squat and hold shape 622H from the portal 602H towards the user. In response, the user should attempt to squat into the shape 622H such that it appears to the user that they are passing through the shape 622H in the VR space, and the user should hold the squat until the hold portion 604H has passed. For example, by determining the height and position of the user's head, the system can determine how well and how long the user has squatted, and the system may adjust the user's squat score 450 accordingly. FIGS. 15A - 15B show a view of the user (on their own display 124) of the squat and hold shape.

[0109] As should be appreciated, each of the shapes and / or objects discussed in these examples corresponds to an event in a routine. A routine may include a number of events, and a routine may include a number of simultaneous events. For example, a routine may send a number of hit objects to a user simultaneously from the same or different sources (see, e.g., FIGS. 9D, 9E, and 9F). [System in operation]

[0110] The operation of the system will be described with reference to the flowcharts in FIGS. 7A-7F.

[0111] When a user starts, the system calibrates for that user (see also FIGS. 8A-8D at 702 and FIG. 7B). Based on that calibration and optionally other information about the user (e.g., their age, gender, weight, previous system use, etc.), the system may determine an initial routine (at 704).

[0112] The user then starts their routine when the system generates the next event in the routine (at 706 and FIG. 7C). The event may include any of the events described above (e.g., a hit object, a squat, a lunge, etc.). The system analyzes the user's interaction with the object (at 708 and FIG. 7D) and determines one or more performance levels for the user (at 710 and FIG. 7E). The system may determine an overall performance level and / or performance levels for different types of events or objects. For example, a user may be very good at hitting an object but not as good at squatting or lunging. The user's performance level may be reflected in and / or derived from their score 446 (e.g., their previous score). The user's performance level may include the user's physiological data 456 (e.g., their heart rate).

[0113] Depending on the user's performance, the routine may be changed to be easier or more difficult for the user (in 712 and FIG. 7F).

[0114] The system continues to generate and present events until the routine is performed.

[0115] The manner of user calibration (in 702) is shown in FIG. 7B, where the system determines the user's height (in 714), vertical reach (in 715), left range (in 716), right range (in 718), and squat range (in 720). The system may use sensors within the headset 104 and controllers 114-1, 114-2 to determine these values. Examples of user calibration are shown in FIGS. 8A-8D.

[0116] The manner of the system generating the next event (in 706) is shown in FIG. 7C, where the system obtains the next event from the routine (in 722), renders the event (in 724), and provides feedback to the user if necessary. The feedback may be in the form of verbal encouragement and / or training. For example, the feedback may include a video of an avatar or person indicating how the routine or event should be (or should have been) performed.

[0117] The manner of the system analyzing the user's interaction with an object (in 708) is shown in FIG. 7D, where the system recognizes and analyzes the user's movement (in 727), maps the user's movement from the real world to the virtual world (in 728), and renders the interaction in the VR space (on the user's display 124) (in 730).

[0118] Aspects of a system for determining one or more performance levels for a user are shown in FIG. 7E, where the system compares (at 732) the user's performance to a desired performance. This comparison may be used to determine the user's score (446 in FIG. 4B).

[0119] Aspects of a system for determining the next event for a routine are shown in FIG. 7F, where the system generates one or more next events based on the user's actual performance (such as represented by, for example, the user score (446 in FIG. 4B) and / or physiological data (456 in FIG. 4B)) compared to its desired performance. [Feedback]

[0120] As can be seen, the system thus creates and potentially modifies the user's exercise / activity routine based on real-time feedback based on the user's performance. [Re - calibration]

[0121] Most of the user's attributes are unlikely to change over time, but their own squat and range can change over time. For this reason, the system should re - calibrate the user each time it is used. [Real - time]

[0122] One of ordinary skill in the art, upon reading this description, will recognize and understand that, as used herein, the term "real time" means near real time or sufficiently real time. It should be recognized that inherent delays exist in electronic components and in network-based communications (e.g., based on network traffic and distance), and that these delays can cause delays when data reaches various components. The inherent delays in a system do not change the real-time nature of the data. In some cases, the term "real-time data" can refer to data that is acquired within a time sufficient to be useful for its intended purpose.

[0123] The term "real time" may be used herein, but it should be recognized that the system is not limited by this term or by how much time it actually takes. In some cases, real-time computing can refer to online computing, i.e., generating an answer as data arrives and generally keeping up with data that arrives continuously without delay. The term "online" computing is contrasted with "offline" or "batch" computing. [Example]

[0124] Exemplary screenshots of aspects of the system implementation are shown in FIGS. 8A - 8D, FIGS. 9A - 9F, FIGS. 10A - 10H, FIGS. 11A - 11D, FIGS. 12A - 12C, FIGS. 13A - 13C, FIGS. 14A - 14C, and FIGS. 15A - 15B.

[0125] Some of these screenshots show views that a VR user 102 would view on their display 124 (Figs. 9A-9F, Figs. 14A-14C), while some of the shots show trainers within the view and may be used for calibration (Figs. 8A-8D) or for training or tutorial purposes (Figs. 10A-10H, Figs. 11A-11D, Figs. 12A-12C, Figs. 13A-13C).

[0126] As can be seen in these figures, the VR display 124 may include any background, whereby the user can perceive that they are performing their activities in any real or virtual location. [Computing]

[0127] The applications, services, mechanisms, operations, and actions illustrated and described above are implemented, at least in part, by software executed on one or more computers.

[0128] Programs (and other types of data) implementing such methods may be stored and transmitted in a variety of media (e.g., computer-readable media) in several ways. Hardwired circuits or custom hardware may be used instead of, or in combination with, some or all of the software instructions that can implement the processes of the various embodiments. Therefore, various combinations of hardware and software may be used instead of software alone.

[0129] One of ordinary skill in the art, upon reading this description, will readily recognize and understand that the various processes described herein may be implemented, for example, by a suitably programmed general-purpose computer, a special-purpose computer, and computing devices. One or more such computers or computing devices may be referred to as a computer system.

[0130] FIG. 16 is a schematic diagram of a computer system 1600 in which embodiments of the present disclosure may be implemented and executed.

[0131] According to this example, the computer system 1600 includes a bus 1602 (i.e., an interconnect), one or more processors 1604, main memory 1606, read-only memory 1608, removable storage media 1610, mass storage 1612, and one or more communication ports 1614. The communication port 1614 may be connected to one or more networks (not shown) through which the computer system 1600 may receive and / or transmit data.

[0132] As used herein, "processor" means one or more microprocessors, central processing units (CPUs), computing devices, microcontrollers, digital signal processors, or similar devices or any combination thereof, regardless of their architecture. The apparatus for executing a process may include, for example, a processor and devices such as input devices and output devices suitable for executing the process.

[0133] The processor 1604 can be any known processor such as, but not limited to, an Intel® Itanium® or Itanium2® processor, an AMD® Opteron® or Athlon MP® processor, or a processor product line from Motorola®. The communication port 1614 can be any of an Ethernet® port, a Gigabit port using copper or fiber, or a USB port, etc. The communication port 1614 may be selected depending on a network such as a local area network (LAN), a wide area network (WAN), or any network to which the computer system 1600 is connected. The computer system 1600 may communicate with peripheral devices (e.g., a display screen 1616, an input device 1618) via an input / output (I / O) port 1620.

[0134] The main memory 1606 can be a random access memory (RAM) or any other arbitrary dynamic storage device generally known in the art. The read-only memory (ROM) 1608 can be any static storage device such as a programmable read-only memory (PROM) chip that stores static information such as instructions for the processor 1604. The mass storage 1612 can be used to store information and instructions. For example, an array of disks such as a hard disk drive, an optical disk, a redundant array of independent disks (RAID), or any other arbitrary mass storage device may be used.

[0135] Bus 1602 communicatively couples processor 1604 to other memory blocks, storage blocks, and communication blocks. Bus 1602 can be a PCI / PCI-X, SCSI, Universal Serial Bus (USB)-based system bus (or others), depending on the memory devices used. Removable storage medium 1610 can be any kind of external storage, including a hard drive, floppy drive, USB drive, Compact Disc-Read Only Memory (CD-ROM), Compact Disc-Rewritable (CD-RW), Digital Versatile Disc-Read Only Memory (DVD-ROM), and the like.

[0136] Embodiments herein may be provided as one or more computer program products including a machine-readable medium storing instructions that can be used to program a computer (or other electronic device) to perform a process. As used herein, the term "machine-readable medium" refers to any medium, multiple same media, or combination of different media that participates in providing data (e.g., instructions, data structures) that can be read by a computer, processor, or similar device. Such media may take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks and other permanent memories. Volatile media typically includes dynamic random access memory that constitutes a computer's main memory. Transmission media includes coaxial cables, copper wire, and fiber optics, including wires that comprise a system bus coupled to a processor. Transmission media may include or convey acoustic waves, light waves, and electromagnetic radiation such as those generated during radio frequency (RF) and infrared (IR) data communications.

[0137] A machine-readable medium may include, but is not limited to, a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a ROM, a RAM, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic or optical card, a flash memory, or other types of media / machine-readable media suitable for storing electronic instructions. Additionally, embodiments herein may be downloaded as a computer program product, where the program may be transferred from a remote computer to a requesting computer as a data signal embodied in a carrier wave or other propagated medium via a communication link (e.g., a modem or network connection).

[0138] Various forms of computer-readable media may be involved in carrying data (e.g., a sequence of instructions) to a processor. For example, the data may be (i) delivered from RAM to the processor, (ii) carried via a wireless transmission medium, (iii) formatted and / or transmitted according to a number of formats, standards or protocols, and / or (iv) encrypted in any of a variety of ways well known in the art.

[0139] A computer-readable medium can store program elements (in any suitable format) that are appropriate for performing a method.

[0140] As shown, main memory 1606 is encoded using application 1622 that supports functions as discussed herein (application 1622 may be an application that provides some or all of the functions of the services / mechanisms described herein, e.g., the VR sharing application 230 of FIG. 2). Application 1622 (and / or other resources described herein) may be embodied as software code such as data and / or logical instructions (e.g., code stored on another computer-readable medium such as a memory or disk) that support processing functions according to different embodiments described herein.

[0141] During operation of one embodiment, processor 1604 accesses main memory 1606 via the use of bus 1602 to launch, run, execute, interpret, or otherwise perform the logical instructions of application 1622. Execution of application 1622 generates processing functions of services associated with the application. In other words, process 1624 represents one or more portions of application 1622 that are executed within or on processor 1604 in computer system 1600.

[0142] For example, process 1624 may include an AR application process corresponding to VR sharing application 230.

[0143] In addition to process 1624 that performs operations as discussed herein, other embodiments herein may include application 1622 itself (i.e., logical instructions and / or data that do not perform or execute). Application 1622 may be stored on a computer-readable medium such as a disk (e.g., a repository) or an optical medium. According to other embodiments, application 1622 may be stored in a memory type system such as firmware, read-only memory (ROM), or, in this example, within main memory 1606 (e.g., within random access memory or RAM) as executable code. For example, application 1622 may be stored on removable storage medium 1610, read-only memory 1608, and / or mass storage device 1612.

[0144] One of ordinary skill in the art will understand that computer system 1600 can include other processes and / or software and hardware components such as an operating system that controls the allocation and use of hardware resources.

[0145] As discussed herein, embodiments of the present invention include various things or operations or actions. These various things may be performed by hardware components or may be embodied in machine-executable instructions that can be used to cause a general-purpose processor or a dedicated processor programmed with the instructions to perform the operations. Alternatively, the things may be performed by a combination of hardware, software, and / or firmware. The term "module" refers to a self-contained functional component that can include hardware, software, firmware, or any combination thereof.

[0146] One of ordinary skill in the art, upon reading this description, will readily recognize and understand that embodiments of the apparatus may include a computer / computing device operable to perform some (but not necessarily all) of the processes described.

[0147] Embodiments of a computer-readable medium storing a program or data structure include a computer-readable medium that, when executed, stores a program that can cause a processor to perform some (but not necessarily all) of the processes described.

[0148] When a process is described herein, one of ordinary skill in the art will recognize that the process may operate without any user intervention. In another embodiment, the process includes some human intervention (e.g., something is performed by or with the assistance of a human).

[0149] Embodiments of the present invention are described using an integrated device (e.g., a smartphone), but one of ordinary skill in the art, upon reading this description, will recognize and understand that the techniques described herein may be used on any computing device including a display and at least one camera capable of capturing a real-time video image of a user. For example, the system may be integrated into a vehicle's head-up display or the like. In such a case, the rear camera may be omitted. [Conclusion]

[0150] As used herein, including in the claims, the phrase "at least some" means "one or more," including the case of only one. Thus, for example, the phrase "at least some ABCs" means "one or more ABCs," including the case of only one ABC.

[0151] The term "at least one" should be understood to mean "one or more", and thus includes both embodiments that include one or multiple components. Further, dependent claims that refer to an independent claim that recites a feature having "at least one" have the same meaning where the feature is referred to as both "said" and "said at least one".

[0152] As used in this description, the term "portion" means part or all. Thus, for example, "portion A of X" may include part or all of "X". In the context of conversation, the term "portion" means some or all of the conversation.

[0153] As used in this specification including the claims, the phrase "based on" means "partially based on" or "at least partially based on" and is not exclusive. Thus, for example, the phrase "based on factor X" means "partially based on factor X" or "at least partially based on factor X". Unless specifically stated by use of the word "only", the phrase "based on X" does not mean "based only on X".

[0154] As used in this specification including the claims, the phrase "using" means "at least using" and is not exclusive. Thus, for example, the phrase "using X" means "at least using X". Unless specifically stated by use of the word "only", the phrase "using X" does not mean "using only X".

[0155] As used in this specification including the claims, the phrase "corresponding to" means "partially corresponding to" or "at least partially corresponding to", and is not exclusive. Therefore, for example, the phrase "corresponding to factor X" means "partially corresponding to factor X" or "at least partially corresponding to factor X". Unless specifically stated by the use of the word "only", the phrase "corresponding to X" does not mean "corresponding to only X".

[0156] Generally, as used in this specification including the claims, the word "only" should not be read into the language unless the word "only" is specifically used in the language.

[0157] As used in this specification including the claims, the phrase "different" means "at least partially different". Unless specifically stated, "different" does not mean completely different. Therefore, for example, the phrase "X is different from Y" means "X is at least partially different from Y" and does not mean "X is completely different from Y". Therefore, as used in this specification including the claims, the phrase "X is different from Y" means that X is different from Y in at least some respects.

[0158] It should be recognized that the words "first" and "second" in this specification and the claims are used to distinguish or identify and are not used to indicate a sequential or numerical limitation. Similarly, the use of letter or numerical labels (such as "(a)", "(b)", etc.) is used to assist in distinguishing and / or identifying and is not used to indicate any sequential or numerical limitation or ordering.

[0159] Unless specifically shown or stated, no order is suggested by any of the labeled boxes in any of the flowcharts. If truncated boxes are shown in the drawings, the activities associated with those boxes may be performed in any order, including fully or partially in parallel.

[0160] As used in this specification, including the claims, unless the context indicates otherwise, the singular forms of terms shall be construed to include the plural forms thereof, and vice versa. Therefore, as used in this specification, unless the context clearly dictates otherwise, it should be noted that the singular forms "a", "an", and "the" include plural referents.

[0161] Throughout this specification and the claims, the terms "comprise", "include", "have", "contain", and their derivatives should be understood to mean "including, but not limited to", and are not intended to exclude other components.

[0162] The present invention also encompasses these terms for features, values, ranges, etc. when used in conjunction with terms such as about, approximately, substantially, essentially, at least, etc. (i.e., "about 3" shall be construed to include exactly 3, or "substantially constant" shall be construed to include exactly constant).

[0163] The use of exemplary terms such as "for instance", "such as", "for example", etc. is merely intended to better illustrate the present invention and does not indicate a limitation to the scope of the present invention unless so claimed. Anything described in this specification may be performed in any order or simultaneously, unless the context clearly dictates otherwise.

[0164] All of the features and / or things disclosed in this specification can be combined in any combination, except for combinations in which at least some of the features and / or things are mutually exclusive. In particular, the preferred features of the present invention are applicable to all aspects of the present invention and may be used in any combination.

[0165] The reference signs are merely referred to for the reason of quicker understanding and are not intended to limit the scope of the present invention in any way.

[0166] Although the present invention has been described in connection with the most practical and preferred embodiments currently considered, it should be understood that the present invention should not be limited to the disclosed embodiments, but on the contrary, is intended to encompass various modifications and equivalent configurations included within the spirit and scope of the appended claims.

Claims

1. (A) determining a routine for a person wearing a user device in a real-world environment, the user device comprising a virtual reality (VR) headset worn by the person; (B) presenting an object associated with an event from the routine to the person in a virtual world via the VR headset; and (C) rendering aspects of the person's virtual interactions with the objects in the virtual world on the VR headset; and (D) analyzing the virtual interaction with the object; and (E) determining, in real time, one or more next events for the routine based on the analysis; and A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, further comprising repeating actions (B), (C), (D), and (E) until the routine is performed or the person stops.

3. The computer-implemented method of claim 1 or 2, further comprising determining calibration information for a person wearing the user device in the real-world environment, wherein the determining in (A) is based on the calibration information.

4. The computer-implemented method of claim 3 , wherein the calibration information includes one or more of height, vertical reach, left lunge information, right lunge information, and squat information.

5. The computer-implemented method of claim 1 , wherein the routine comprises a time sequence of events.

6. The computer-implemented method of claim 1 , wherein the user device includes at least one VR handheld controller worn or held by the person.

7. The computer-implemented method of claim 6 , wherein the at least one VR handheld controller is rendered in the virtual world based on an expected virtual interaction of the person with the object.

8. Analyzing the virtual interaction with the object includes: recognizing and analyzing the person's movements to determine movement data; mapping the motion data to the virtual world; The computer-implemented method of claim 1 , comprising:

9. The computer-implemented method of claim 1 , further comprising providing feedback to the person based on the analysis.

10. The computer-implemented method of claim 1 , further comprising determining a performance level of the person.

11. The computer-implemented method of claim 10 , wherein the performance level is used in (E) to determine the one or more next events for the routine.

12. The computer-implemented method of claim 10 , wherein the performance level is based on the person's virtual interaction with the object relative to an expected or desired interaction with the object.

13. The computer-implemented method of claim 10 , wherein the one or more next events for the routine are determined based on the performance level of the person.

14. The computer-implemented method of claim 1 , wherein an event has a corresponding object associated with the event.

15. The computer-implemented method of claim 1 , wherein the object comprises one of a hit object, a squat object, or a lunge object.

16. The computer-implemented method of claim 15 , wherein the hit object includes a hit direction.

17. The computer-implemented method of claim 15 , wherein the hit object includes a tail.

18. The computer-implemented method of claim 15 , wherein the squat object comprises a symmetric triangle.

19. The computer-implemented method of claim 15 , wherein the lunge object comprises an asymmetric triangle.

20. 2. The computer-implemented method of claim 1, wherein the object has a source location associated with it, and the object is presented on the VR headset as appearing in the virtual world at the source location and coming from the source location towards the person.

21. 2. The computer-implemented method of claim 1, wherein the object has a velocity associated with it, and the object is presented on the VR headset as appearing in the virtual world and coming towards the person at said velocity.

22. 16. The computer-implemented method of claim 15, wherein if the object includes a hit object, analyzing the virtual interaction with the object comprises determining whether the object was virtually hit by the person.

23. 23. The computer-implemented method of claim 22, wherein analyzing the virtual interaction with the object comprises determining whether the object was virtually hit with a virtual baton controlled by at least one VR handheld controller worn or held by the person.

24. 20. The computer-implemented method of claim 16, wherein if the object includes a hit object, analyzing the virtual interaction with the object comprises determining whether the object was virtually hit by the person in the hit direction.

25. 25. The computer-implemented method of claim 24, wherein if the object includes a hit object, analyzing the virtual interaction with the object comprises determining whether the object was virtually hit in the hit direction with a virtual baton controlled by at least one VR handheld controller worn or held by the person.

26. 26. The computer-implemented method of claim 25, wherein if the object includes a hit object, analyzing the virtual interaction with the object comprises determining whether the object was virtually hit in the hit direction with a virtual baton controlled by the at least one VR handheld controller and whether the person virtually followed through the hit.

27. The computer-implemented method of claim 12 , wherein the performance level is also based on physiological data associated with the person.

28. 16. The computer-implemented method of claim 15, wherein if the object includes a squat object, analyzing the virtual interaction with the object comprises determining an amount of squatting by the person while the squat object passes by the person in the virtual world.

29. 16. The computer-implemented method of claim 15, wherein if the object includes a lunge object, analyzing the virtual interaction with the object comprises determining an amount and direction of a lunge by the person while the lunge object passes by the person in the virtual world.

30. 30. The computer-implemented method of claim 29, wherein the lunge object comprises a triangle, and determining the amount and / or direction of the lunge is based on a position of the person's head in the virtual world relative to vertices of the triangle.

31. 16. The computer-implemented method of claim 15, wherein the squat object or the lunge object includes a hold shape, and analyzing the virtual interaction with the object comprises determining an amount and duration of a squat or lunge by the person while the object passes through the person in the virtual world.

32. 2. The computer-implemented method of claim 1, wherein the object is presented in the virtual world as coming from a first location in the virtual world, and the object includes an indication of a second location in the virtual world for a subsequent object to be presented.

33. 33. The computer-implemented method of claim 32, wherein the indication of the second location includes an arrow.

34. The computer-implemented method of claim 1 , wherein the routine comprises a fitness or exercise routine.

35. (A) providing a user device in a real-world environment, the user device comprising a virtual reality (VR) headset worn by a person, the VR headset capable of providing images of scenes and objects to the person through the VR headset to generate a visual representation of a virtual world; (B) determining a visual routine for the person to view in the virtual world using the VR headset, the visual routine including a plurality of objects with which the person may interact using specified actions; (C) presenting a first object of the plurality of objects to the person in the virtual world; and (D) enabling the person to virtually interact with the first object in the virtual world; and (E) measuring aspects of a virtual interaction between the person and the presented first object; and (F) analyzing the measured aspects of the virtual interaction; and (G) presenting, in real time, a second object of the plurality of objects to the person in the virtual world based on the analysis. A computer-implemented method comprising:

36. 36. The computer-implemented method of claim 35, further comprising repeating operations (C), (D), (E), (F), and (G) for multiple objects in the plurality of objects of the vision routine.

37. 36. The computer-implemented method of claim 35, wherein the plurality of objects comprises a sequence of objects.

38. 36. The computer-implemented method of claim 35, wherein the specified action by a user includes a swing motion to hit the first object, and the measured aspect includes a direction of the swing motion relative to the first object.

39. 36. The computer-implemented method of claim 35, wherein the specified action by a user includes a swing motion to hit the first object, and the measured aspect includes a speed of the swing motion relative to the first object.

40. 36. The computer-implemented method of claim 35, wherein the specified action by a user includes a swing motion to hit the first object, and the measured aspect includes a length of the swing motion relative to the first object.

41. 36. The computer-implemented method of claim 35, wherein the first object comprises a triangular shape, the specified action by the user comprises a squat motion, and the measured aspect comprises a vertical displacement of the squat motion relative to a vertex of the triangular shape.

42. 36. The computer-implemented method of claim 35, wherein the first object includes a triangular shape, the specified action by the user includes a lunging motion, and the measured aspect includes a relative position of the user's head with respect to locations of vertices of the triangular shape.

43. 36. The computer-implemented method of claim 35, wherein the first object includes a visual marker that indicates to a user a direction of a swing action relative to the first object.

44. 36. The computer-implemented method of claim 35, wherein the first object is generated at a first portal within the virtual world, the first object including a visual marker that indicates to a user a direction of a swing action relative to the first object.

45. (A) providing a user device in a real-world environment, the user device comprising a virtual reality (VR) headset worn by a person, the VR headset capable of providing images of scenes and objects to the person through the VR headset to generate a visual representation of a virtual world; (B) determining a visual routine for the person to view in the virtual world using the VR headset, the visual routine including a plurality of triangles, each of the triangles having a defined shape and vertices; (C) determining a position of the VR headset in the virtual world during the vision routine; and (D) presenting a first triangle of the plurality of triangles to the person in the virtual world based on the determining in (C); and (E) enabling the person to virtually interact with the presented first triangle in the virtual world; and (F) comparing the position of the VR headset to the locations of the vertices of the first triangle; (G) recording the results of said comparing in (F); and A computer-implemented method comprising:

46. 46. ​​The computer-implemented method of claim 45, wherein the enabling in (E) comprises enabling the person to virtually interact with the presented first triangle in the virtual world by positioning his or her body in the real world until the VR headset is positioned below and adjacent to a location of the vertex of the presented first triangle.

47. (A) providing a user device in a real-world environment, the user device comprising a virtual reality (VR) headset worn by a person, the VR headset capable of providing images of scenes and objects to the person through the VR headset to generate a visual representation of a virtual world; (B) determining a visual routine for the person to view in the virtual world using the VR headset, the visual routine including a series of objects generated from one of a first portal and a second portal, each of the objects moving toward the person in the virtual world; (C) presenting a first object from the first portal to the person in the virtual world; and (D) providing an indication of a location of the second portal relative to the first object; and (E) presenting a second object from the second portal to the person in the virtual world. A computer-implemented method comprising:

48. 48. The computer-implemented method of claim 47, further comprising repeating operations (C)-(E) for multiple objects of the visual routine.

49. 48. The computer-implemented method of claim 47, wherein in (D), the indication for the first object comprises an arrow shape.

50. 48. The computer-implemented method of claim 47, wherein in (D), the indication for the first object includes a triangular shape.

51. (a) hardware having a memory and at least one processor; (b) a service running on said hardware, said service being configured to perform a computer-implemented method according to any one of claims 1 to 50; and A device comprising:

52. 51. An article of manufacture comprising a non-transitory computer readable medium having computer readable instructions stored thereon, the computer readable instructions having instructions for implementing a computer implemented method, the computer implemented method operable on a device comprising hardware having a memory and at least one processor, and executing a service on the hardware, the computer implemented method comprising the computer implemented method of any one of claims 1 to 50.

53. One or more computer programs which, when executed, cause one or more processors to perform at least the computer-implemented method of any one of claims 1 to 50.

Images