Augmented Reality-based Motion Training Method, Device, Server and Terminal Device

Abstract

The present invention relates to a motion training method, device, server and terminal device based on augmented reality. The method includes: acquiring image data of an athlete in a first sports field during a motion; generating an augmented reality (AR) image of the athlete based on the image data of the athlete during the motion; and displaying the augmented reality (AR) image of the athlete on an augmented reality (AR) device, where the augmented reality (AR) device is used by a trainer during training in a second sports field, and the augmented reality (AR) image of the athlete is used to be superimposed in the second sports field so that the trainer can observe the augmented reality (AR) image of the athlete during training; wherein at least part of the first sports field and at least part of the second sports field can implement the same type of motion training.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and particularly to a motion training method, device, server, terminal device, readable storage medium and computer program product based on augmented reality. Background Art

[0002] In the field of modern sports competitions, the performance gap between athletes in some competitive events has reached the millisecond level, and the competition has become increasingly fierce. To reduce the gap and improve their own performance, in addition to daily training, athletes also participate in various competitions to improve their sports performance and exercise their stress resistance in actual combat. In addition, athletes and coaches will also watch video recordings of past competitions to analyze techniques and tactics, and so on.

[0003] However, due to the influence of some force majeure factors, such as the global pandemic, various sports events have been postponed or cancelled, and athletes have been unable to participate in person for a long time, let alone obtain video files of the competition process. Many objective factors have made the training rhythm, sports state, competitive state, etc. of many athletes unstable, affecting the improvement of athletes' competitive levels. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a motion training method, device, server, terminal device, readable storage medium and computer program product based on augmented reality, which can break through the limitations of time and space, provide as realistic a training or competition scenario as possible for athletes, and assist athletes in improving training effects; the embodiments of the present application can be extended to the daily physical exercise of the general public.

[0005] The present application provides a motion training method based on augmented reality, which is applied to a server. The method includes: obtaining image data of an athlete during exercise in a first sports venue; generating an augmented reality (AR) image of the athlete based on the image data of the athlete during exercise; sending the augmented reality (AR) image of the athlete to an augmented reality (AR) device, where the augmented reality (AR) device is used by a trainer during training in a second sports venue, and the augmented reality (AR) image of the athlete is used to be superimposed and displayed in the second sports venue so that the trainer can observe the augmented reality (AR) image of the athlete during training; where at least part of the area of the first sports venue and at least part of the area of the second sports venue can implement the same type of motion training.

[0006] Optionally, the size of at least part of the area of the first sports venue is the same as the size of at least part of the area of the second sports venue.

[0007] Optionally, the method further includes: providing a first training mode in the augmented reality (AR) device, where the first training mode is used for the trainer to observe the athlete's movement posture by following the athlete's AR image.

[0008] Optionally, the method further includes: providing a second training mode in the AR device, where the second training mode is used for the trainer to compete with the athlete's AR image in the second sports venue.

[0009] Optionally, the method further includes: displaying the athlete's motion state information in a region near the athlete's AR image.

[0010] Optionally, the method further includes: displaying the environmental information of the first sports venue and / or the second sports venue in a region far from the athlete's AR image.

[0011] Optionally, obtaining the image data of the athlete in the first sports venue during the movement process includes: obtaining the holographic information of the athlete in the first sports venue during the movement process; or, obtaining the moving image data of the athlete in the first sports venue tracked and captured by a plurality of cameras in the first sports venue, where the number of the cameras is multiple and is used to capture the moving image data of the athlete from one or more of the following perspectives: front view, rear view, left view, right view, top view, bottom view.

[0012] Optionally, the method further includes: obtaining the image data of the athlete in the third sports venue during the movement process; generating the AR image of the athlete in the third sports venue based on the image data of the athlete in the third sports venue during the movement process; superimposing the AR image of the athlete in the third sports venue on the second sports venue; where at least part of the third sports venue and at least part of the second sports venue can implement the same type of sports training.

[0013] Optionally, the method further includes: obtaining the image data of the trainer during the training process in the second sports venue; sending the image data of the trainer during the training process in the second sports venue to a specified server for the server to generate the AR image of the trainer.

[0014] Optionally, the first sports venue includes at least one of the following: swimming pool, runway, ice rink, snow field.

[0015] The present application also provides an augmented reality-based motion training method, which is applied to a terminal device. The method includes: receiving an augmented reality (AR) image of an athlete, where the augmented reality (AR) image is generated based on image data of the athlete during movement in a first sports venue; displaying the augmented reality (AR) image of the athlete on the terminal device, where the terminal device is used by a trainer during training in a second sports venue, and the augmented reality (AR) image of the athlete is used to be superimposed and displayed in the second sports venue so that the trainer can observe the augmented reality (AR) image of the athlete during training; where at least part of the area of the first sports venue and at least part of the area of the second sports venue can implement the same type of motion training.

[0016] Optionally, the size of at least part of the area of the first sports venue is the same as the size of at least part of the area of the second sports venue.

[0017] Optionally, the method further includes: providing a first training mode, where the first training mode is used for the trainer to observe the motion posture of the athlete by following the augmented reality (AR) image of the athlete.

[0018] Optionally, the method further includes: providing a second training mode, where the second training mode is used for the trainer to compete with the augmented reality (AR) image of the athlete in the second sports venue.

[0019] Optionally, the method further includes: displaying the motion state information of the athlete in a nearby area of the augmented reality (AR) image of the athlete.

[0020] Optionally, the method further includes: displaying the environmental information of the first sports venue and / or the second sports venue in an area far from the augmented reality (AR) image of the athlete.

[0021] Optionally, the first sports venue includes at least one of the following: a swimming pool, a running track, an ice rink, a snow field.

[0022] Optionally, the terminal device includes AR glasses or an AR helmet.

[0023] The present application also provides an augmented reality-based sports training device, including: an image data acquisition module for acquiring image data of an athlete during movement in a first sports venue; an AR image generation module for generating an augmented reality (AR) image of the athlete based on the image data of the athlete during movement; an AR image sending module for sending the augmented reality AR image of the athlete to an augmented reality AR device, where the augmented reality AR device is used by a trainer during training in a second sports venue, and the augmented reality AR image of the athlete is used to be superimposed on the second sports venue so that the trainer can observe the augmented reality AR image of the athlete during training; wherein at least part of the area of the first sports venue and at least part of the area of the second sports venue can carry out the same type of sports training.

[0024] The present application also provides an augmented reality-based sports training device, including: an AR image receiving module for receiving an augmented reality AR image of an athlete, where the augmented reality AR image is generated based on the image data of the athlete during movement in a first sports venue; an AR image display module for displaying the augmented reality AR image of the athlete on the terminal device, where the terminal device is used by a trainer during training in a second sports venue, and the augmented reality AR image of the athlete is used to be superimposed and displayed on the second sports venue so that the trainer can observe the augmented reality AR image of the athlete during training; wherein at least part of the area of the first sports venue and at least part of the area of the second sports venue can carry out the same type of sports training.

[0025] The present application also provides a server, which includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method described in any one of the foregoing is implemented.

[0026] The present application also provides a terminal device, which includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method described in any one of the foregoing is implemented.

[0027] The present application also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described in any one of the foregoing is implemented.

[0028] The present application also provides a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the method described in any one of the foregoing is implemented.

[0029] Using the augmented reality-based motion training solution provided by the embodiments of the present application, the trainer can compare their own movements with those of athletes to correct their own movements, can truly restore the game scene, break through the time and space barriers of the game, enabling athletes located at different times or venues to compete on the same field, achieving the purpose of improving the trainer's sports competition level. Description of the Drawings

[0030] Next, the preferred embodiments of the present invention will be further described in detail with reference to the drawings, where:

[0031] Figure 1 is a schematic diagram of the AR system architecture based on the server and terminal devices in the embodiments of the present application;

[0032] Figure 2 is a schematic diagram of the virtual-real fusion image for AR navigation using a typical mobile phone APP;

[0033] Figure 3 is a flowchart of the augmented reality-based motion training method on the server side in the embodiments of the present application;

[0034] Figure 4 is a flowchart of the augmented reality-based motion training method on the terminal side in the embodiments of the present application;

[0035] Figure 5 is a flowchart of an implementation of the augmented reality-based motion training method in the embodiments of the present application;

[0036] Figure 6A is a schematic diagram of the effect of setting multiple cameras around the swimming pool based on the embodiments of the present application;

[0037] Figure 6B is a schematic diagram of the effect of providing a training mode for the trainer based on the embodiments of the present application;

[0038] Figure 6C is a schematic diagram of the effect of providing a game mode for the trainer based on the embodiments of the present application;

[0039] Figure 7 is an effect diagram of the superimposed display of virtual athletes in the AR glasses, i.e., swimming goggles, in the embodiments of the present application;

[0040] Figure 8 is a structural block diagram of the augmented reality-based motion training device on the server side in the embodiments of the present application;

[0041] Figure 9 is a structural block diagram of the augmented reality-based motion training device on the terminal side in the embodiments of the present application;

[0042] Figure 10It is a schematic diagram of an electronic device for implementing the augmented reality-based motion training method according to an embodiment of the present application;

[0043] Figure 11 It is a schematic diagram of the software structure of a terminal device according to an embodiment of the present application. Detailed implementation manners

[0044] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that the purpose of providing these embodiments is to make the principles and spirit of the present application clearer and more thorough, so that those skilled in the art can better understand and then implement the principles and spirit of the present application. The exemplary embodiments provided herein are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments herein, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.

[0045] Those skilled in the art know that the embodiments of the present application can be implemented as a system, device, equipment, method, computer-readable storage medium or computer program product. Therefore, the present application can be specifically implemented in at least one of the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0046] According to the embodiments of the present application, the present application claims protection for an augmented reality-based motion training method, device, server, terminal device, readable storage medium and computer program product.

[0047] In this document, terms such as first, second, and third are only used to distinguish one entity (or operation) from another entity (or operation), rather than requiring or implying any order or association between these entities (or operations).

[0048] The embodiments of the present application can be applied to servers and terminal devices. Please refer to Figure 1 , which schematically shows an AR system architecture diagram based on a server and terminal devices. The AR system architecture includes a server 10 and several terminal devices 20. In some examples, the terminal device 20 is an AR device, which can be a dedicated AR device, such as a head-mounted display (HMD), smart gloves, wearable electronic devices such as clothing. In some examples, the terminal device 20 can be a general AR device, such as a mobile phone, portable computer, laptop computer, tablet computer, virtual reality (VR) device or in-vehicle device installed with AR function software, and so on.

[0049] Taking an AR helmet or AR glasses as an example, a head-mounted display, a machine vision system, a mobile computer, etc. can be integrated and set in a device that can be bound and worn. The device has a display similar to glasses in shape and is worn on the user's head during operation. The device can transmit augmented reality information to the display or project it onto the user's eyeballs, thereby enhancing the user's visual immersion. In some examples, the AR device also has a camera, which can be a wide-angle camera, a telephoto camera, or a structured light camera (also known as a point cloud depth camera, a 3D structured light camera, or a depth camera). Among them, the structured light camera is based on 3D vision technology and can obtain the plane and depth information of an object. The structured light camera can project light with certain structural features onto the object to be photographed through a near-infrared laser, and then the reflected light is collected by an infrared camera and processed by a processor chip. Its calculation principle is to calculate the position and depth information of the object according to the change of the optical signal caused by the object and present a 3D image. Usually, a two-dimensional image is presented on a terminal device such as a mobile phone, and the depth of different positions on the image cannot be displayed. Using a structured light camera, 3D image information data can be captured, that is, not only the information such as the color of different positions in the image can be obtained, but also the depth information of different positions can be obtained, which can be used for AR ranging. Of course, an ordinary terminal device can also obtain the depth information of a 2D image based on an optical camera to capture a 2D image and combine it with a deep learning algorithm, etc., and finally a 3D image can also be presented.

[0050] In some examples, software or an application program APP with AR functions is installed in the terminal device 20. The server 10 can be the management server or application server of the software or APP. The server 10 can be a single server, a server cluster composed of multiple servers, or a cloud server or a cloud server, etc. The terminal device 20 is integrated with a module with networking functions, such as a Wireless-Fidelity (Wifi) module, a Bluetooth module, a 2G / 3G / 4G / 5G communication module, etc., so as to connect to the server 10 through the network.

[0051] Taking an APP with AR navigation function as an example, the APP can have functions such as high-precision map navigation ability, environment understanding ability, and virtual-real fusion rendering ability, etc. The APP can report the current geographical location information to the server 10 through the terminal device 20, and the server 10 provides AR navigation services for the user based on the real-time geographical location information. Exemplarily, taking the terminal device 20 as a mobile phone, in response to the user's operation of starting the APP, the mobile phone can start the camera to capture images of the real environment, and then the system performs AR enhancement on the images of the real environment captured by the camera, and incorporates or superimposes and renders AR effects (such as navigation route signs, road names, merchant information, advertisement displays, etc.) in the images of the real environment, and displays the virtual-real fusion images on the mobile phone screen.

[0052] Exemplarily, the first user can log in to the first user account through the APP installed in the mobile phone, and the second user can log in to the second user account through the software installed in the AR glasses.

[0053] Figure 2 Schematically shows a typical virtual-real fusion image for AR navigation using a mobile APP, where the indication arrow of the AR navigation is superimposed on the real road surface and space in the figure, and the electronic resources for merchant promotions are displayed in the space inside the building in the form of a parachute carrying a small gift box. When the user walks with the mobile phone, they can view the AR navigation route through the mobile phone screen and easily obtain digital information related to the environment.

[0054] Embodiments of the present application relate to terminal devices and / or servers. Hereinafter, the principles and spirits of the present application will be elaborated in detail through several exemplary embodiments or representative implementations.

[0055] First, a brief description will be given of relevant contents such as concepts and technical terms that may be involved in the embodiments of the present application.

[0056] There can be various types of AR software products. Currently, they are mainly software programs, clients, application programs APP, or mini-programs within the APP installed on hardware terminal devices, etc. Embodiments of the present application have no special requirements for the type or operating form of the AR software product. That is, various types of AR software products can be trained and used using the embodiments of the present application.

[0057] Generally, the imaging of glasses can only present a basic model of a person and simple action behaviors in the glasses. For example, a walking kitten is presented in the eyes. However, in addition to the interesting display, AR devices can achieve more practical applications. For athletes' training, they need to continuously understand and adjust their own movement conditions, which requires repeatedly observing their own or their opponents' game videos for technical comparison. This method is obviously not intuitive enough and is completely limited to the two-dimensional screen of the imaging. Similarly, there are many other pain points in athletes' training that have not been properly solved.

[0058] In view of this, embodiments of the present application propose an augmented reality-based sports training method, which is applied to the server side, referring to Figure 3 , and includes the following steps:

[0059] S101, obtaining image data of an athlete during movement in a first sports field;

[0060] S102, generating an augmented reality (AR) image of the athlete based on the image data of the athlete during movement;

[0061] S103. Send the augmented reality (AR) image of the athlete to the AR device, where the AR device is used by the trainer during training on the second sports field, and the AR image of the athlete is used to be superimposed on the second sports field so that the trainer can observe the AR image of the athlete during training. At least part of the first sports field and at least part of the second sports field can carry out the same type of sports training.

[0062] Corresponding to the processing on the server side, an embodiment of the present application further provides an augmented reality-based sports training method, which is applied to a terminal device located in the second sports field. Refer to Figure 4 The method includes the following steps:

[0063] S201. Receive the AR image of the athlete, where the AR image is generated based on the image data of the athlete in the first sports field during the movement process.

[0064] S102. Display the AR image of the athlete on the terminal device, where the terminal device is used by the trainer during training on the second sports field, and the AR image of the athlete is used to be superimposed and displayed on the second sports field so that the trainer can observe the AR image of the athlete during training.

[0065] At least part of the first sports field and at least part of the second sports field can carry out the same type of sports training.

[0066] According to the augmented reality-based sports training method of the embodiment of the present application, the server (or cloud) will first obtain the image data of the movement process of the athlete in the first sports field. For example, the image data of the athlete's movement process in his / her own field can be obtained through a camera device.

[0067] Then, based on the image data of the athlete during the movement process, the AR image of the athlete's movement process can be generated by means of 3D modeling. For example, the body modeling data and action posture data of the athlete can be extracted from the captured image data, which can be used to reproduce the AR image of the athlete's movement process.

[0068] Next, the server sends the obtained AR image to the augmented reality (AR) device. In this way, the trainer can use the AR device to superimpose the AR image on the second sports field where the trainer is located, and the trainer can adjust or play the AR image by himself / herself. Since the AR image completely restores the movement process of the athletes on the first sports field, the trainer can immerse himself / herself in observing and comparing the movement process of the athletes through the AR device on the second field, and can even train and compete together.

[0069] In this way, on the one hand, the trainer can compare his / her own movements with those of the athletes to correct his / her own movements; on the other hand, it can truly restore the game scene, break through the time and space barriers of the game, so that athletes located in different times or venues can compete on the same field, achieving the purpose of improving the sports competition level of the trainer.

[0070] According to an embodiment of the present application, the size of at least part of the first sports field is the same as the size of at least part of the second sports field. The reason for this setting is that professional athletes complete training or competitions on specific or standard sports fields. In order to accurately restore their movement process and provide a better immersive experience, the AR image should be projected onto a real field with the same size as the first sports field. For example, if it is necessary to present the movement process of a 100M sprinter, then the second sports field should include a 100M track. In some embodiments, the sizes of the fields may not be exactly the same. For example, the image of a 100M run can be projected onto a 200M ordinary road surface, and the AR image can also be projected and the 100M run training can be carried out. The solution of the present application is not limited by the field itself, and the application method is flexible and convenient, which can reduce the training cost of the trainer.

[0071] According to an embodiment of the present application, a first training mode is provided in the augmented reality (AR) device. The first training mode is used for the trainer to observe the movement posture of the athlete by following the augmented reality (AR) image of the athlete. During the training process, the trainer can move together with the AR image. For example, both are located on the same track, with the AR image in front and the trainer behind. The trainer follows the AR image to move together and observes the movements of the AR image during the movement process. This can correct the trainer's movement actions and improve the training effect.

[0072] According to an embodiment of the present application, a second training mode is provided in the augmented reality (AR) device, and the second training mode is used for a trainer to compete with the AR image of an athlete in the second sports field. Similarly, the trainer can set the AR image on other tracks and start together with the trainer for a competition on the same field. Since the AR image completely restores the movement process of the athlete, the trainer is equivalent to competing with the athlete in real time. In this way, the competition is no longer restricted by time and space. For example, during the epidemic period, it is difficult to hold large-scale cross-regional and international competitions, and athletes cannot obtain the pre-competition experience of large-scale competitions, which affects the competitive state of athletes. The present invention can truly restore the competition scene through the device, enabling athletes to obtain a better competition experience.

[0073] According to an embodiment of the present application, in the area near the AR image of the athlete, the movement state information of the athlete is displayed. Among them, the movement state information can be various. For example, when the sport is running, the movement state information can be the current speed, stride, etc. of the athlete; when the sport is swimming, the movement state information can be the current speed, SWOLF index (SWOLF index = single-lap timing t + single-lap stroke count p, and for the same swimming distance d, the smaller the number, the higher the swimming efficiency), breathing frequency, etc. of the athlete. The present application does not limit the type of sport and the movement state information, and similar types of sport and movement state information that can apply the solution of the present application can be covered within the scope of the present application. The movement state information can effectively inform the trainer of the current movement state of the athlete, so that the trainer can understand the difference in the movement states of the two.

[0074] According to an embodiment of the present application, in the area far from the AR image of the athlete, the environmental information of the first sports field and / or the second sports field is displayed. Among them, the environmental information can be various. For example, when the sport is running, the environmental information can be the current time, light, temperature, humidity, etc. of the sports field; when the sport is swimming, the movement state information can be the room temperature, water temperature, water flow velocity, water pressure, etc. of the sports field. The present application does not limit the sports field and the environmental information of the sports field. Those skilled in the art can understand that similar sports fields and environmental information of the sports field that can apply the solution of the present application can be covered within the scope of the present application. The environmental information can effectively inform the trainer of the current field situation of the athlete, so that the trainer can understand the difference in the field environments of the two.

[0075] According to an embodiment of the present application, obtaining the image data of an athlete in the first sports field during the movement process can be achieved by any of the following methods:

[0076] · Obtaining the holographic information of an athlete in the first sports field during the movement process;

[0077] · Obtain the motion image data of the athlete during the motion tracked and photographed by the camera device in the first sports field.

[0078] Wherein, if a camera device is used for photographing, the number of the camera devices is multiple, and is used for photographing the motion image data of the athlete from one or more of the following perspectives: front view perspective, rear view perspective, left view perspective, right view perspective, top view perspective, bottom view perspective.

[0079] That is to say, in this application, obtaining the motion process of the athlete can be achieved through a holographic device or through multiple cameras at different angles. This application does not limit the way of obtaining images, and similar ways of obtaining images that can be applied to the solution of this application can be covered within the scope of this application. This can make the presentation of the AR image closer to the actual motion process of the athlete.

[0080] According to an embodiment of the present application, the method further includes: obtaining the image data of the athlete in the third sports field during the motion; generating an augmented reality (AR) image of the athlete in the third sports field based on the image data of the athlete in the third sports field during the motion; superimposing the augmented reality (AR) image of the athlete in the third sports field on the second sports field; wherein, at least part of the area of the third sports field and at least part of the area of the second sports field can perform the same type of sports training.

[0081] It can be seen that the solution of this application can present multiple athletes at different times and spaces in the second sports field at the same time, making the restored scene closer to the actual competition scene. This application does not specifically limit the number of athlete characters in the AR image, and can be, for example, 10, 15, 20, more or less.

[0082] According to an embodiment of the present application, the method further includes: obtaining the image data of the trainer during the training process in the second sports field; sending the image data of the trainer during the training process in the second sports field to a designated server for the server to generate an augmented reality (AR) image of the trainer.

[0083] Therefore, in this application, the trainer can not only train or compete with the athlete, but also upload his own image to the server to form a corresponding AR image. This is not only convenient for other trainers to download and train, but also convenient for the trainer himself to observe. In this way, the trainer can understand his own motion state from a close third perspective and adjust his own motion state.

[0084] According to an embodiment of the present application, the first sports venue includes at least one of the following: a swimming pool, a running track, an ice rink, and a snow field. The present application can be applied to various competition venues such as swimming pools, running tracks, ice rinks, snow fields, track and field stadiums, and racing tracks. As an example, using the embodiment of the present application, an operator can reconstruct the 3D model of a swimming pool and a swimming pool (such as the "Water Cube" venue), realize the digital twin of the "Water Cube", and the operator wearing AR glasses can see the competitive scenes of the athletes in the "Water Cube" venue in the current swimming venue, obtaining a more real training and competition experience. Among them, the AR glasses can be integrated with swimming goggles to make the operation and training process more convenient and provide a full-range immersive experience to the greatest extent.

[0085] Figure 5 is a flowchart of a specific implementation process of the augmented reality sports training method according to an embodiment of the present application. As Figure 5 shown, the implementation process of the present application can be to first perform omnidirectional image acquisition through a high-speed camera, then quickly model to restore the 3D image, and then perform an all-round deduction of the actions recorded within a specific time through the model (that is, the omnidirectional image captured by the high-speed camera). Then, the image is restored and compared through a terminal (which can be AR goggles), and finally, omnidirectional image acquisition can be performed again through the high-speed camera to generate a new 3D image from the movements of the current game.

[0086] Next, taking the swimming event as an example, the implementation process of the embodiment of the present application will be introduced in detail in combination with specific applications.

[0087] As Figure 6A shown, multiple cameras can be set around the swimming pool (the first sports venue). There are 8 swimming lanes in the figure, and each lane is independently collected. Among them:

[0088] · 8 cameras are located on each track (exemplarily placed on the left), and as the athlete moves, the image data above the athlete (Data1) is recorded;

[0089] · 8 cameras are located in front of each track to record the image data in front of the athlete (Data2);

[0090] · 8 cameras are located at the bottom of each track (exemplarily placed on the right), and as the athlete moves, the image data below the athlete (Data3) is recorded;

[0091] · 8 cameras are located behind each track to record the image data behind the athlete (Data4).

[0092] Among them, the data of each athlete includes the data sets [Data1, Data2, Data3, Data4] collected by 4 cameras. The collected motion data can be transmitted to the cloud for storage in real time, which is used to generate 3D images of these 8 athletes.

[0093] Optionally, corresponding camera positions can be added to better restore the motion process. It can be understood that when the type of motion changes, the positions of the cameras can be adjusted adaptively. For example, when the motion is running, the upper and lower cameras can be adjusted to be set on the left and right sides of the athlete's body for following shooting.

[0094] Then, based on the collected athlete data, an athlete digital model is established, and the states during the entire competition process are simulated. In addition, the SWOLF index and breathing frequency b of the athlete can be established to form a data description of the motion state of the image. Among them, the SWOLF index = single-lap timing t + single-lap stroke count p. On the same swimming distance d, the smaller the SWOLF index, the higher the swimming efficiency.

[0095] As Figure 6B shown, the trainer can select a training mode for training. In the training mode, the user can select a learning object. After the cloud processes the selected athlete data, it is loaded in real time through the AR glasses. By observing from behind, the user can understand the movements and postures of professional athletes. The user follows behind the learning object to learn the movements of the athletes, where Athletes A - H are AR images.

[0096] As Figure 6C shown, the trainer can select a competition mode for training or competition. In the competition mode, the user selects a track and the athletes participating in the competition. Through the AR glasses, the user can see the competition data of other athletes, where Athletes C, B, E, and F are AR images. By setting the AR images on the corresponding tracks, the user (i.e., the trainer) can compete with the athletes on the same field at the same time.

[0097] In some embodiments, the trainer can select the training mode or competition mode through gestures.

[0098] In some embodiments, the trainer can adjust the distance between the AR image and himself. For example, set the AR image to be 0.5m or 1m in front of himself. In some embodiments, the trainer can set the AR image at any position within the AR visible range.

[0099] Figure 7The figure shows the effect diagram of superimposed display of virtual athletes in AR glasses, i.e., swimming goggles. As shown in the figure, multiple AR images are presented in the swimming pool where the trainer is currently located through the trainer's AR swimming goggles, and at the same time, the current swimming speed is also presented, providing an immersive training environment for the trainer and helping the trainer improve their competitive level.

[0100] In some embodiments, during the use of the system, the user's motion data, such as the SWOLF index and breathing rate, is transmitted to the cloud in real time through the network, records the current user's training status information, combines with the real-time data of the corresponding virtual athlete, and after the training ends, gives a complete set of evaluation and improvement suggestions based on the training data to optimize the training plan.

[0101] Regarding many existing problems, for example, during the epidemic period, it is difficult to hold cross-regional and international competitions, and athletes cannot obtain the pre-competition experience of large-scale competitions to improve their competitive state; when ordinary people exercise daily, there is no professional guidance (high cost) and the movements are not standard; the method of selecting or recommending talents by the sports center through competitions is single and the efficiency is low. These problems can all be optimized and solved through the solution of this application. This application can collect competition and motion data and apply it to athlete training and mass exercise. The data can be reused repeatedly, reducing the training cost and improving the training effect. This process can be paused and watched repeatedly, and the movement problems can be accurately located to prescribe the right medicine. At the same time, users can send the data to the sports center, enabling the sports center to improve the training efficiency, optimize the training plan, and select athletes targeted. Provide more athlete selection solutions for the sports center. The present invention can assist the sports center in selecting talents by user registration and uploading real motion data. The present invention can assist training through equipment to improve the exercise effect. At the same time, the present invention can truly restore the competition scene through the equipment, enabling athletes to obtain a better competition experience. It can also enhance the realism of the user's simulated competition and increase the immersion of the user's training. Enable users to train targeted. Upload real-time motion data, and artificial intelligence analyzes the massive data to give a constructive training plan. Conduct professional simulation training in more scenarios. Can collect competition information at different times and places. Reduce the training cost and the workload of coaches.

[0102] Corresponding to the method embodiment provided by the present application, the present application also provides an augmented reality-based motion training device, which is applied to the server side. As Figure 8 shown, the augmented reality motion training device 300 includes:

[0103] An image data acquisition module 310, configured to acquire image data of an athlete in a first sports field during the movement process;

[0104] The AR image generation module 320 generates an augmented reality (AR) image of the athlete based on the image data of the athlete during the movement.

[0105] The AR image sending module 330 is configured to display the augmented reality (AR) image of the athlete on the augmented reality (AR) device. The augmented reality (AR) device is used by a trainer during training on a second sports field. The augmented reality (AR) image of the athlete is used to be superimposed on the second sports field so that the trainer can observe the augmented reality (AR) image of the athlete during training. At least part of the area of the first sports field and at least part of the area of the second sports field can perform the same type of sports training.

[0106] Correspondingly, the present application further provides an augmented reality-based sports training device applied to the terminal side. As Figure 9 shown, the augmented reality sports training device 400 includes:

[0107] The AR image receiving module 410 is configured to receive the augmented reality (AR) image of the athlete. The augmented reality (AR) image is generated based on the image data of the athlete in the first sports field during the movement.

[0108] The AR image display module 420 is configured to display the augmented reality (AR) image of the athlete on the terminal device. The terminal device is used by a trainer during training on a second sports field. The augmented reality (AR) image of the athlete is used to be superimposed and displayed on the second sports field so that the trainer can observe the augmented reality (AR) image of the athlete during training. At least part of the area of the first sports field and at least part of the area of the second sports field can perform the same type of sports training.

[0109] The present application further provides an electronic device, including: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the object data processing method of any one of the above embodiments is implemented. The electronic device can be the server or the terminal device of the embodiments of the present application.

[0110] Figure 10A schematic structural diagram of an electronic device 10 provided by an embodiment of the present application. The electronic device 10 includes a processor 11, a memory 12, and a communication bus for connecting the processor 11 and the memory 12. A computer program that can run on the processor 11 is stored in the memory 12. When the processor 11 runs the computer program, it can execute or implement the steps in the methods of various embodiments of the present application. The electronic device 10 can be the server in the embodiments of the present application, or the electronic device 10 can also be a cloud server. The electronic device 10 can also be the terminal device or AR device in the embodiments of the present application. In a suitable case, the electronic device can also be referred to as a computing device. The electronic device 10 can also be a cloud server. The electronic device 10 further includes a communication interface for receiving and sending data.

[0111] In some embodiments, the processor 11 can be a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a modem processor, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), etc.; the processor 11 can also be other general-purpose processors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. Among them, by learning from the biological neural network structure, the neural-network processing unit NPU can quickly process the input information and can also continuously learn by itself. Through the NPU, the electronic device 10 can implement applications such as intelligent cognition, such as image recognition, face recognition, semantic recognition, speech recognition, text understanding, etc.

[0112] In some embodiments, the memory 12 may be an internal storage unit of the electronic device 10, such as a hard disk or memory of the electronic device 10; the memory 12 may also be an external storage device of the electronic device 10, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 10. The memory 12 may also include both an internal storage unit and an external storage device of the electronic device 10. The memory 12 can be used to store an operating system, application programs, a BootLoader, data, and other programs, such as program codes of computer programs. The memory 12 includes but is not limited to a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a compact disc read-only memory (CD-ROM). The memory 12 is used to store the program codes executed by the electronic device 10 and the data transmitted. The memory 12 can also be used to temporarily store the data that has been output or will be output.

[0113] Those skilled in the art can understand that Figure 10 merely examples of the electronic device 10, which do not constitute a limitation on the electronic device 10. The electronic device 10 may include more or fewer components than shown in the figure, or combine certain components, or include different components. For example, it may also include input / output devices, network access devices, etc.

[0114] Figure 11 is a schematic diagram of the software structure of a terminal device according to an embodiment of the present application. Taking the Android system as an example of the mobile phone operating system, in some embodiments, the Android system is divided into four layers, namely: the application layer, the application framework layer (framework, FWK), the system layer, and the hardware abstraction layer, and communication between layers is through software interfaces.

[0115] First, the application layer may include multiple application packages. The application packages may be various application programs app such as calls, cameras, videos, navigation, weather, instant messaging, education, etc., or may also be application programs app based on AR technology.

[0116] Second, the application framework layer FWK provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer may include some predefined functions, such as functions for receiving events sent by the application framework layer.

[0117] The application framework layer may include a window manager, a resource manager, a notification manager, etc.

[0118] Among them, the window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data, and make this data accessible to applications. The data may include videos, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc.

[0119] Among them, the resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, and so on.

[0120] Among them, the notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as the notification of a background-running application, or a notification that appears on the screen in the form of a dialogue window. For example, it prompts text information in the status bar, emits a prompt tone, the electronic device vibrates, the indicator light flashes, etc.

[0121] In addition, the application framework layer may also include a view system. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, on the display interface of a text message notification icon, there may be a view for displaying text and a view for displaying pictures.

[0122] Third, the system layer may include multiple functional modules, such as a sensor service module, a physical state recognition module, a 3D graphics processing library (e.g., OpenGLES), and so on.

[0123] Among them, the sensor service module is used to monitor the sensor data uploaded by various sensors in the hardware layer to determine the physical state of the mobile phone; the physical state recognition module is used to analyze and recognize user gestures, faces, etc.; the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0124] In addition, the system layer may also include a surface manager and a media library. The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications. The media library supports the playback and recording of various common audio and video formats, as well as static image files, etc.

[0125] Finally, the hardware abstraction layer is the layer between the hardware and the software. The hardware abstraction layer may include a display driver, a camera driver, a sensor driver, etc., which are used to drive the relevant hardware in the hardware layer, such as a display screen, a camera, a sensor, etc.

[0126] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program or instruction. When the computer program or instruction is executed, it is used to implement the steps in the method designed in the above embodiment.

[0127] The embodiment of the present application also provides a computer program product, including a computer program or instruction. When the computer program or instruction is executed, it is used to implement the steps in the method designed in the above embodiment. Exemplarily, the computer program product may be a software installation package.

[0128] Those skilled in the art should be aware that the methods, steps, or functions of the relevant modules / units described in the embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product, or by the way of a processor executing computer program instructions. Among them, the computer program product includes at least one computer program instruction, and the computer program instructions can be composed of corresponding software modules. The software modules can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable hard disks, CD-ROMs (Compact Disc Read-Only Memory) or any other form of storage medium well known in the art. The computer program instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer-readable storage medium can be any available medium that the computer can access, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (such as an SSD), etc.

[0129] Regarding each device / product described in the above embodiments, the modules / units included therein can be software modules / units, hardware modules / units, or partly software modules / units and partly hardware modules / units. For example, for a device / product applied or integrated on a chip, each of the modules / units included therein can be implemented in the form of hardware such as circuits, or at least some of the modules / units are implemented in the form of software programs running on a processor integrated inside the chip, and the remaining modules / units are implemented in the form of hardware such as circuits. Another example is that for a device / product applied or integrated on a terminal, each of the modules / units included therein can be implemented in the form of hardware such as circuits, or at least some of the modules / units are implemented in the form of software programs running on a processor integrated inside the terminal, and the remaining modules / units can be implemented in the form of hardware such as circuits.

[0130] It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A motion training method based on augmented reality, applied to a server, the method comprising: Reconstructing a 3D model of a first sports venue to generate a digital twin of the first sports venue and sending it to an augmented reality (AR) device; Obtaining image data of an athlete in the first sports venue during a motion; Generating an augmented reality (AR) image of the athlete based on the image data of the athlete during the motion; Sending the augmented reality (AR) image of the athlete to the augmented reality (AR) device, wherein the augmented reality (AR) device is used by a trainer during training in a second sports venue, and the augmented reality (AR) image of the athlete is used to be superimposed and displayed in the second sports venue so that the trainer can observe the augmented reality (AR) image of the athlete during training; Obtaining image data of an athlete in a third sports venue during a motion; generating an augmented reality (AR) image of the athlete in the third sports venue based on the image data of the athlete in the third sports venue during the motion; superimposing the augmented reality (AR) image of the athlete in the third sports venue in the second sports venue; wherein at least part of the third sports venue and at least part of the second sports venue can perform the same type of motion training; Wherein at least part of the first sports venue and at least part of the third sports venue and at least part of the second sports venue can perform the same type of motion training; Wherein the dimensions of at least part of the first sports venue and at least part of the third sports venue are the same as the dimensions of at least part of the second sports venue.

2. The method according to claim 1, further comprising: Providing a first training mode in the augmented reality (AR) device, the first training mode being used for the trainer to observe the motion posture of the athlete by following the augmented reality (AR) image of the athlete.

3. The method according to claim 1, further comprising: Providing a second training mode in the augmented reality (AR) device, the second training mode being used for the trainer to compete with the augmented reality (AR) image of the athlete in the second sports venue.

4. The method according to claim 1, further comprising: Displaying the motion state information of the athlete in a region near the augmented reality (AR) image of the athlete.

5. The method according to claim 1, further comprising: Displaying the environmental information of the first sports venue and / or the second sports venue in a region far from the augmented reality (AR) image of the athlete.

6. The method according to claim 1, wherein The obtaining of the image data of the athlete in the first sports venue during the motion comprises: Obtaining the holographic information of the athlete in the first sports venue during the motion; Or, Obtaining the motion image data of the athlete in the first sports venue tracked and captured by a camera device in the first sports venue, wherein the number of the camera devices is multiple and is used to capture the motion image data of the athlete from one or more of the following perspectives: front view perspective, rear view perspective, left view perspective, right view perspective, top view perspective, bottom view perspective.

7. The method according to claim 1 further comprises: Obtaining image data of the trainer during the training process on the second sports ground; Sending the image data of the trainer during the training process on the second sports ground to a designated server for the server to generate an augmented reality (AR) image of the trainer.

8. The method according to any one of claims 1-7, wherein The first sports ground comprises at least one of the following: a swimming pool, a running track, an ice rink, a snow field.

9. An augmented reality-based sports training method applied to a terminal device, the method comprising: Receiving an augmented reality (AR) image of an athlete and digital twins of a first sports ground and a third sports ground, wherein the augmented reality (AR) image is generated based on image data of the athlete during the movement process in the first sports ground and the third sports ground, the digital twin of the first sports ground is generated by performing 3D model reconstruction on the first sports ground, and the digital twin of the third sports ground is generated by performing 3D model reconstruction on the third sports ground; Displaying the augmented reality (AR) image of the athlete on the terminal device, wherein the terminal device is used by a trainer during the training on the second sports ground, and the augmented reality (AR) image of the athlete is used for superimposed display in the second sports ground so that the trainer can observe the augmented reality (AR) image of the athlete during the training; Wherein at least part of the first sports ground and the third sports ground can perform the same type of sports training as at least part of the second sports ground; Wherein the size of at least part of the first sports ground and the third sports ground is the same as the size of at least part of the second sports ground.

10. The method according to claim 9 further comprises: Providing a first training mode for the trainer to observe the movement posture of the athlete by following the augmented reality (AR) image of the athlete.

11. The method according to claim 9 further comprises: Providing a second training mode for the trainer to compete with the augmented reality (AR) image of the athlete in the second sports ground.

12. The method according to claim 9 further comprises: Displaying the movement state information of the athlete in the vicinity of the augmented reality (AR) image of the athlete.

13. The method according to claim 9 further comprises: Displaying the environmental information of the first sports ground and / or the second sports ground in an area far from the augmented reality (AR) image of the athlete.

14. The method according to any one of claims 10-13, wherein The first sports ground comprises at least one of the following: a swimming pool, a running track, an ice rink, a snow field.

15. The method according to any one of claims 10-13, wherein The terminal device comprises an AR glasses or an AR helmet.

16. An augmented reality-based motion training device, characterized in that, Comprising: A 3D model reconstruction module, which is used to reconstruct the 3D models of the first sports field and the third sports field to generate digital twins of the first sports field and the third sports field, and send them to the augmented reality (AR) device; An image data acquisition module, which is used to acquire the image data of the athletes in the first sports field and the third sports field during the movement process; An AR image generation module, which is used to generate the augmented reality (AR) images of the athletes based on the image data of the athletes during the movement process; An AR image sending module, which is used to send the augmented reality (AR) images of the athletes to the augmented reality (AR) device. The augmented reality (AR) device is used by the trainer during the training in the second sports field. The augmented reality (AR) images of the athletes are used to be superimposed on the second sports field so that the trainer can observe the augmented reality (AR) images of the athletes during the training. At least part of the areas of the first sports field and the third sports field can carry out the same type of sports training as at least part of the areas of the second sports field. At least part of the areas of the first sports field and the third sports field have the same size as at least part of the areas of the second sports field.

17. An augmented reality-based motion training device, characterized in that, It includes: An AR image receiving module, which is used to receive the augmented reality (AR) images of the athletes. The augmented reality (AR) images are generated based on the image data of the athletes in the first sports field and the third sports field during the movement process; An AR image display module, which is used to display the augmented reality (AR) images of the athletes on the terminal device. The terminal device is used by the trainer during the training in the second sports field. The augmented reality (AR) images of the athletes are used to be superimposed and displayed on the second sports field so that the trainer can observe the augmented reality (AR) images of the athletes during the training. At least part of the areas of the first sports field and the third sports field can carry out the same type of sports training as at least part of the areas of the second sports field; Among them, the 3D models of the first sports field and the third sports field are reconstructed to generate digital twins of the first sports field and the third sports field. At least part of the areas of the first sports field and the third sports field have the same size as at least part of the areas of the second sports field.

18. A server, characterized in that, It includes: A processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method described in any one of claims 1-8 is implemented.

19. A terminal device, characterized in that, It includes: A processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method described in any one of claims 9-15 is implemented.

20. A computer-readable storage medium, characterized in that, Computer program instructions are stored on the computer-readable storage medium. When the computer program instructions are executed by the processor, the method described in any one of claims 1-15 is implemented.

21. A computer program product, characterized in that, It includes computer program instructions. When the computer program instructions are executed by the processor, the method described in any one of claims 1-15 is implemented.

Images