An adaptive synchronization method and electronic device for motion

Abstract

The present application relates to the field of computer application technology, and discloses an adaptive motion synchronization method, wherein the number of frames of each fitness action in a fitness video displayed on a display screen of an electronic device corresponds to the movement distance of each skeletal node when the user completes each fitness action, and the movement distance of each fitness action completed by the user corresponds to the number of frames matched with the movement distance, so that the playback speed of the fitness action demonstrated by the coach in the fitness video is synchronized with the speed of the user performing the fitness action, thereby ensuring the continuity and rhythm of the user performing the fitness action along with the fitness video, and improving the user's fitness experience.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to an action adaptive synchronization method and electronic device. Background Art

[0002] As people's life pressure continues to increase in contemporary society, fitness and leisure time are seriously insufficient. In order to reduce the time users spend traveling to the gym, more and more users choose indoor fitness. When users exercise indoors, they often learn fitness movements through fitness video tutorials provided by fitness software on electronic devices such as mobile phones, computers, and smart screens. However, current fitness software can only provide fitness video tutorials. It is difficult to match the speed at which users perform the same fitness movements in fitness video tutorials with the speed at which coaches perform fitness movements. This leads to the problem of asynchrony between the fitness movements of coaches and users in fitness video tutorials, which destroys the continuity and rhythm of users' fitness training and causes users to have a poor fitness experience. Summary of the invention

[0003] The purpose of the present invention is to provide a method for adaptively synchronizing an action, which ensures the synchronization of fitness actions of a coach and a user in a fitness video tutorial and improves the user's fitness experience.

[0004] In a first aspect, an embodiment of the present invention discloses a motion adaptive synchronization method, which is applied to an electronic device, and the motion adaptive synchronization method includes: the electronic device plays a fitness video, wherein the fitness video contains information such as the movement posture of a fitness coach, the guidance voice of the fitness video, and text information.

[0005] The electronic device collects the user's motion and posture. The image acquisition module of the electronic device can be used on the electronic device to collect image information such as the posture and motion of the user in front of the electronic device. The image acquisition module can be embedded in the housing of the electronic device, such as embedded in the upper frame of the electronic device, or can be set independently of the electronic device, and the image acquisition module is set above the display screen of the electronic device. The image acquisition module can be a camera (such as an infrared camera) and the like.

[0006] The electronic device matches the fitness action currently being played in the fitness video with the user's action posture. The electronic device determines the key skeletal nodes involved in the fitness action demonstrated by the coach in the fitness video and the key skeletal nodes involved in the user's current action posture captured by the camera of the smart screen, and determines whether the fitness action of the user and the coach in the fitness video is synchronized based on the key skeletal nodes.

[0007] According to the embodiment of the first aspect of the present application, at least one fitness coach may demonstrate standard fitness movements in a fitness video.

[0008] According to the embodiment of the first aspect of the present application, when determining the key skeletal nodes, the following steps may be performed:

[0009] The electronic device obtains one or more skeleton nodes of at least one fitness coach in the fitness video;

[0010] The electronic device obtains one or more skeleton nodes of the user;

[0011] The electronic device determines key skeletal nodes from one or more skeletal nodes of at least one fitness trainer and one or more skeletal nodes of the user in the fitness video.

[0012] According to an embodiment of the first aspect of the present application, determining the key skeletal nodes includes:

[0013] The electronic device selects key skeletal nodes of a currently demonstrated fitness action from skeletal nodes of a user and at least one fitness coach, wherein the key skeletal nodes are skeletal nodes involved in the current fitness action.

[0014] According to an embodiment of the first aspect of the present application, the electronic device matches the currently played fitness action in the fitness video with the user's action posture, including:

[0015] The electronic device extracts key skeletal nodes from the fitness action currently demonstrated by at least one fitness coach and the action posture of the user in the current fitness video based on LSTM, obtains the third posture feature of the currently demonstrated fitness action and the fourth posture feature of the user, and matches the third posture feature with the fourth posture feature based on the key skeletal nodes. The third posture feature and the fourth posture feature are the action postures of the key skeletal nodes.

[0016] According to the embodiment of the first aspect of the present application, the electronic device extracts key skeletal nodes from the fitness action currently demonstrated by at least one fitness coach and the action posture of the user in the current fitness video based on LSTM, obtains the first posture feature of the currently demonstrated fitness action and the second posture feature of the user, and matches the first posture feature with the second posture feature. The first posture feature and the second posture feature are the action postures of the key skeletal nodes.

[0017] According to the embodiment of the first aspect of the present application, the fitness video is framed to obtain frame images of continuous moments of the fitness video. The electronic device reads each fitness action in the fitness video in sequence according to the playback time sequence, and performs video frame division on each fitness action. The video frame division is to divide each fitness action in the fitness video into frames, and each frame in the video is a still picture, so that each fitness action is divided into multiple frames of image pictures.

[0018] If the fitness action does not match the user's current action posture, the electronic device adjusts the playing speed of the fitness action in the fitness video to synchronize the fitness action with the user's action posture.

[0019] According to an embodiment of the first aspect of the present application, the electronic device determines a target frame image corresponding to a second posture feature of the user. Based on the time relationship between the frame image corresponding to the first posture feature and the target frame image, the playback speed of the fitness video is adjusted so that the fitness action corresponding to the target frame image is synchronized with the action posture of the user.

[0020] According to an embodiment of the first aspect of the present application, determining a target frame image corresponding to a second posture feature of the user includes:

[0021] The electronic device determines the target frame image based on the total number of frames and the average moving distance of the complete fitness action corresponding to the second posture feature, wherein the average moving distance is the average moving distance of the key skeletal nodes corresponding to the second posture feature from the initial position to the end position of the complete fitness action. The electronic device calculates the speed of the user when doing the fitness action based on the average moving distance and the number of frames of the fitness action. The camera of the electronic device collects the user's action posture in real time. When the user repeats the same fitness action, the real-time average moving distance of all skeletal nodes involved in the fitness action is calculated, and the picture frame corresponding to the average moving distance is played accordingly, so as to achieve the purpose of controlling the playback speed of the fitness video.

[0022] According to the embodiment of the first aspect of the present application, the average moving distance is the average value of the moving distances of the key skeletal nodes corresponding to the second posture feature from the initial position to the end position in the three-dimensional coordinate system in the X-axis, Y-axis and Z-axis directions.

[0023] The first aspect of the present application discloses an adaptive motion synchronization method, in which the number of frames of each fitness action in a fitness video displayed on a display screen of an electronic device corresponds to the movement distance of each skeletal node collected when the user completes each fitness action, and the movement distance occurred when the user completes each fitness action corresponds to the number of frames matched with the movement distance, so that the playback speed of the fitness action demonstrated by the coach in the fitness video is synchronized with the speed of the user performing the fitness action, thereby ensuring the continuity and rhythm of the user's fitness action following the fitness video and improving the user's fitness experience.

[0024] According to an embodiment of the first aspect of the present application, when the time point corresponding to the target frame image exceeds the time point of the frame image corresponding to the first posture feature, the electronic device adjusts the playback speed of the fitness video to the target speed, wherein the target speed is greater than the current playback speed of the fitness video.

[0025] The time point corresponding to the target frame image refers to the value of the time corresponding to the target frame image, and the time point of the user's current fitness action refers to the time point corresponding to the first posture feature of the user's current fitness action. The relationship between the time point of the user's current fitness action and the time point corresponding to the target frame image is called a time relationship.

[0026] According to an embodiment of the first aspect of the present application, when the time point corresponding to the target frame image is greater than the time point of the frame image corresponding to the first posture feature, the electronic device controls the fitness video to jump directly from the currently played fitness action to the target frame image. The time relationship between the time point of the frame image of the fitness action of the currently played fitness video and the time point of the target frame image is: the time of the frame image of the fitness action of the currently played fitness video is earlier than the time of the target frame image.

[0027] According to an embodiment of the first aspect of the present application, when the time point corresponding to the target frame image is less than the time point of the frame image corresponding to the first posture feature, the electronic device controls the playback of the fitness video and jumps directly from the currently playing fitness action to the target frame image.

[0028] According to an embodiment of the first aspect of the present application, when the fitness action is not synchronized with the action posture of the user, the electronic device performs a voice prompt and / or a text prompt. When the fitness action is not synchronized with the action posture of the user, the electronic device displays words such as "action is not synchronized" on the display screen or prompts "action is not synchronized" through a speaker.

[0029] According to an embodiment of the first aspect of the present application, the electronic device displays the fitness video in full screen.

[0030] According to the embodiments of the first aspect of the present application, in some embodiments of the present application, when the video interface is displayed in full screen, the video interface can occupy the entire display area of ​​the display screen. In one possible implementation, displaying the video interface in full screen means that only the video interface is displayed in the display screen without displaying other content. In another possible implementation, the video interface can also occupy only a portion of the display area of ​​the display screen, for example, when the video interface is displayed in the middle part of the display screen and one or both sides of the edge are white or black, it can also be regarded as the display screen displaying the video interface in full screen.

[0031] According to an embodiment of the first aspect of the present application, the user's action gesture is displayed in a floating window on a display screen of the electronic device.

[0032] According to the embodiment of the first aspect of the present application, the electronic device displays a real-time fitness window of the user on the full screen of the display screen, and displays a fitness video window in the lower left corner of the display screen in a floating manner, and displays a text prompt of the current fitness action on the display screen.

[0033] According to an embodiment of the first aspect of the present application, the electronic device stores the action posture of the user; the electronic device determines that the action posture standard of the user is not matched based on the standard fitness action corresponding to the action posture of the user;

[0034] The user's movement posture is marked as a non-standard movement and the standard fitness movement corresponding to the non-standard movement in the fitness video is replayed; the electronic device replays the standard fitness movement in the fitness video at the fitness speed of the user performing the fitness movement corresponding to the non-standard movement, and the standard fitness movement corresponds to the fitness movement corresponding to the non-standard movement.

[0035] The electronic device provides text and / or voice explanation of the non-standard action.

[0036] According to the embodiments of the first aspect of the present application, the electronic device determines that the user has not completed the fitness action based on the user's movement posture and movement trajectory captured in real time by the camera, then the electronic device pauses playing the fitness video, and displays a text prompt of the fitness action currently being performed by the user on the display screen of the electronic device until the camera captures in real time that the user has completed the fitness action that has been played and the user's current fitness action matches the movement posture corresponding to the currently paused screen, then cancels the pause and resumes the video playback from the current pause moment, or replays the fitness video to the screen corresponding to the user's current movement posture to perform the fitness action synchronously again.

[0037] According to the embodiments of the first aspect of the present application, a fitness video is played on a display screen of an electronic device and the user's movements and postures are collected in real time. The electronic device determines whether the user's fitness movements are standard. If they are not standard, the electronic device regards the fitness movements as non-standard fitness movements and calculates the fitness speed of the user's non-standard fitness movements. The electronic device plays the standard fitness movements demonstrated by the coach corresponding to the non-standard fitness movements in the fitness video at the fitness speed to correct the non-standard fitness movements performed by the user.

[0038] According to the embodiment of the first aspect of the present application, when playing the non-standard action video, voice explanations and voice prompts are given to the user to correct the user's incorrect fitness actions.

[0039] It is obvious that the electronic device stores the video of the user's non-standard fitness movements, and after the entire fitness video on the display screen is played, it plays the non-standard fitness video of the user's mistakes, and compares and explains it with the standard fitness video demonstrated by the coach, correcting the user's mistakes in fitness movements, which is beneficial to the user's physical health and further enhances the user's experience.

[0040] In a second aspect, an embodiment of the present application discloses an electronic device, including:

[0041] A memory storing synchronization instructions;

[0042] A processor, when executing the synchronization instruction, implements the steps of any one of the above-mentioned action adaptive synchronization methods.

[0043] In a third aspect, an embodiment of the present application discloses a motion adaptive synchronization device, the motion adaptive synchronization device comprising:

[0044] Play module, used to play fitness videos;

[0045] A collection module, used to collect user's motion gestures;

[0046] The matching module is used to match the fitness action currently played in the fitness video with the action posture of the user.

[0047] The adjustment module is used to adjust the playing speed of the fitness action in the fitness video so that the fitness action is synchronized with the action posture of the user.

[0048] Other features and corresponding beneficial effects of the present invention are described in the latter part of the specification, and it should be understood that at least part of the beneficial effects become obvious from the description in the specification of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] FIG. 1( a ) is a schematic diagram of the structure of a smart screen in an application scenario shown in an embodiment of the present application;

[0050] FIG1( b ) is a schematic diagram of the positional relationship between the smart screen and the image acquisition module shown in an embodiment of the present application;

[0051] FIG1( c ) is a schematic diagram of a use state of a motion adaptive synchronization device shown in an embodiment of the present application;

[0052] FIG1( d ) is a schematic diagram of a human skeleton model shown in an embodiment of the present application;

[0053] FIG1( e ) is a schematic diagram of a UI interface of a smart screen according to an embodiment of the present application;

[0054] FIG1( f ) is a schematic diagram of a fitness page after a user clicks on a fitness APP, shown in an embodiment of the present application;

[0055] Figure 2(a) to Figure 2(d) This is a UI interface diagram of an action adaptive synchronization method shown in an embodiment of the present application;

[0056] FIG3( a ) is a schematic diagram of a flow chart of an action adaptive synchronization method according to an embodiment of the present application;

[0057] FIG3( b ) is a schematic diagram of an action adaptive synchronization state shown in an embodiment of the present application;

[0058] Figure 4(a) to Figure 4(c) This is a UI interface diagram of another action adaptive synchronization method shown in an embodiment of the present application;

[0059] Figure 5 A flowchart of another method for adaptive synchronization of actions according to an embodiment of the present application is shown;

[0060] Figure 6(a) to Figure 6(c) This is a UI interface diagram of another action adaptive synchronization method shown in an embodiment of the present application;

[0061] Figure 7 A flowchart of another method for adaptive synchronization of actions according to an embodiment of the present application is shown;

[0062] Figure 8 A schematic diagram of the structure of an electronic device shown in an embodiment of the present application;

[0063] Fig. 9 This is a schematic diagram of the structure of the SOC shown in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following specific examples illustrate the implementation methods of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application will be introduced in conjunction with the examples, this does not mean that the features of this application are limited to the implementation methods. On the contrary, the purpose of introducing the application in conjunction with the implementation methods is to cover other options or modifications that may be extended based on the claims of the present application. In order to provide a deep understanding of the present application, the following description will include many specific details. The present application can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0065] It will be understood that, as used herein, the term "module" may refer to or include a dedicated integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and / or memory that executes one or more software or firmware programs, a combinational logic circuit, and / or other appropriate hardware components that provide the described functionality, or may be part of these hardware components.

[0066] It is understood that in each embodiment of the present application, the processor may be a microprocessor, a digital signal processor, a microcontroller, etc., and / or any combination thereof. According to another aspect, the processor may be a single-core processor, a multi-core processor, etc., and / or any combination thereof.

[0067] According to some embodiments of the present application, an electronic device for providing a fitness video tutorial is described below. The electronic device includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, virtual reality or augmented reality devices, smart screens and other electronic devices. For the above electronic devices, in order to implement the technical solutions provided in the embodiments of the present application, the electronic device includes but is not limited to a display screen and an image acquisition device. One of the purposes of setting a display screen on the electronic device is to display a fitness video. One of the purposes of setting a video acquisition device on the electronic device is to capture the user's movements and postures within the acquisition range of the video acquisition device of the electronic device. The electronic device uses the number of frames of each fitness action in the fitness video displayed on the display screen and the movement distance of each skeletal node when the user completes each fitness action, and uses the movement distance of each fitness action completed by the user and the total number of frames of the fitness action in the fitness video to control the playback speed of the fitness video, so that the playback speed of the fitness action in the fitness video is synchronized with the speed at which the user performs the fitness action.

[0068] Figure 1(a) is a schematic diagram of the structure of the smart screen of the application scenario shown in the embodiment of the present application, Figure 1(b) is a schematic diagram of the positional relationship between the smart screen and the image acquisition module shown in the embodiment of the present application, Figure 1(c) is a schematic diagram of the use status of the motion adaptive synchronization device shown in the embodiment of the present application, Figure 1(d) is a schematic diagram of the human skeleton model shown in the embodiment of the present application, Figure 1(e) is a schematic diagram of the UI interface of the smart screen shown in the embodiment of the present application, and Figure 1(f) is a schematic diagram of the fitness page after the user clicks on the fitness APP shown in the embodiment of the present application.

[0069] In some embodiments of the present application, the motion adaptive synchronization method provided in the embodiments of the present application is described by taking the electronic device as a smart screen as an example. As shown in FIG1(a), the smart screen 10 includes: a processor 101, a power module 102, a memory 103, a wireless communication module 104, a display screen 105, an image acquisition module 106, etc.

[0070] In some embodiments of the present application, it is understood that the structure shown in FIG. 1(a) of the embodiment of the present application does not constitute a specific limitation on the smart screen 10. In other embodiments of the present application, the smart screen 10 may include more or fewer components than those shown in FIG. 1(a), or combine certain components, or split certain components, or arrange the components differently. The components shown in FIG. 1(a) may be implemented in the form of hardware, software, or a combination of software and hardware.

[0071] In some embodiments of the present application, the processor 101 may provide one or more processing units applied to the smart screen 10. For example, the processor 101 includes but is not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a microprocessor (MCU), an artificial intelligence (AI) processor, or a programmable logic device (Field Programmable Gate Array, FPGA) and the like. Processing modules or processing circuits. Among them, different processing units may be independent devices or integrated in one or more processors. A storage unit may be provided in the processor 101 for storing instructions and data. In some embodiments, the storage unit in the processor 101 is a cache memory. It should be noted that the processor 101 may also be used as a controller.

[0072] In some embodiments of the present application, the power module 102 may provide a battery, a battery management component, etc. applied to the smart screen 10. The power management component is used to manage the charging of the power supply and the power supply to other modules. The battery may be a rechargeable battery such as a lithium battery or a lead-acid battery. The shape of the battery may be, but is not limited to, a button type, a square type, etc.

[0073] In some embodiments of the present application, the memory 103 may include one or more tangible, non-temporary computer-readable media for storing data and / or instructions. In some embodiments, the memory 103 includes but is not limited to any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD) drive, and at least one of a digital versatile disc (DVD) drive. The fitness video tutorial can be cached in the memory 103. When the user determines to play the fitness video tutorial, the fitness video tutorial can be directly extracted from the memory 103 and played on the display screen 105. In addition, the memory 103 can also pre-store a human skeleton model, and each bone node of the human skeleton model is numbered or defined by name. The processor 101 can directly extract the human skeleton model from the memory 103 for application.

[0074] In some embodiments of the present application, the wireless communication module 104 can provide wireless communication solutions for application on the smart screen 10, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), etc. The smart screen 10 can communicate with the network, mobile phones, remote controls, etc. through wireless communication technology. In some embodiments of the present application, the wireless communication module 104 can be a Bluetooth chip, and the communication method is Bluetooth communication. In order to reduce the power consumption of the Bluetooth chip to the smart screen 10, the Bluetooth chip can be a low-power Bluetooth chip, and its model can be selected as the NRF52810-QFAA-R Bluetooth 5.0 low-power Bluetooth chip.

[0075] In some embodiments of the present application, the display screen 105 can provide a display for displaying a human-computer interaction interface, an image, an interactive interface of a fitness APP, a fitness video, etc. on the smart screen 10. The display screen 105 includes but is not limited to a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0076] In some embodiments of the present application, the image acquisition module 106 can be provided on the smart screen 10 for collecting image information such as the posture and action of the user in front of the smart screen 10. The image acquisition module can be embedded in the housing of the smart screen 10, such as embedded in the upper frame of the smart screen, or can be set independently of the smart screen 10. As shown in FIG. 1(b), the image acquisition module 106 is set above the display screen of the smart screen 10, and the image acquisition module 106 is connected to the smart screen 10 via a video transmission line. The image acquisition module 106 can be a camera (such as an infrared camera) or the like.

[0077] In some embodiments of the present application, the image acquisition module 106 includes but is not limited to a lens part, a chipset part, a PCB and a component part, etc. The lens part includes but is not limited to a lens, which is composed of a lens (convex lens, concave lens). It can be classified into a plastic lens and a glass lens based on the material; according to the color classification, the lens can be divided into a color lens and a black and white lens; according to the function classification, the lens can be divided into a fixed lens and a zoom lens; according to the size classification, the lens can be divided into a single-board lens, a pinhole lens, and a CS lens; according to the infrared classification, the lens can be divided into an 850nm lens, a 940nm lens, and a 650nm lens. The chipset part is the heart of the camera and is the main component that converts the optical signal into an electrical signal. According to the characteristics of the photosensitive chip in the chipset, the main chipsets are divided into CCD chips and CMOS chips. The PCB and the component part are important components of the camera and are the carriers of all devices. Their main function is to provide electrical connections between various parts.

[0078] In some embodiments of the present application, as shown in Figure 1(c), the power supply supplies power to the smart screen. After the smart screen is powered, the user operates the smart screen and performs fitness through the fitness video tutorial played by the smart screen shown in the embodiment of the present application.

[0079] The image acquisition module 106 is set at the upper edge of the display screen 105 of the smart screen 10. The image acquisition module 106 can be connected to the smart screen 10 through a video transmission line. The smart screen 10 plays and displays the fitness video of the fitness APP through the display screen 105.

[0080] The fitness video includes the movements and postures of the fitness coach, the guiding voice of the fitness video and text information. The fitness video can be pre-stored in the memory 103 or downloaded and played through the network in real time. The memory 103 also pre-stores a human skeleton model, and each bone node of the human skeleton model is numbered or named.

[0081] In some embodiments of the present application, as shown in FIG1( d ), the skeletal points of the human body are set to 15 skeletal points, which are the center of the head 1, the center of the neck (e.g., the spine center of the neck) 2, the left shoulder joint 3, the right shoulder joint 4, the left elbow joint 5, the right elbow joint 6, the left wrist joint 7, the right wrist joint 8, the left hip joint 9, the left knee joint 10, the left ankle joint 11, the right hip joint 12, the right knee joint 13, the right ankle joint 14 and the trunk center (e.g., the spine center of the trunk) 15, and each bone point is a plurality of bone points. The body can be divided into five parts, namely the trunk, left arm, right arm, left leg and right leg, based on the skeleton points between the skeleton nodes, wherein the trunk includes, for example, the spine center of the trunk and the trunk center (for example, the spine center of the trunk) 15, the left arm includes the left shoulder joint point 3, the left elbow joint point 5 and the left wrist joint point 7, the right arm includes the right shoulder joint point 4, the right elbow joint point 6 and the right wrist joint point 8, the left leg includes the left hip joint point 9, the left knee joint point 10 and the left ankle joint point 11, and the right leg includes the right hip joint point 12, the right knee joint point 13 and the right ankle joint point 14. In addition, it should be noted that in addition to numbering the skeleton nodes for distinction, the names of the skeleton nodes can also be defined for distinction, such as the head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, left knee, left ankle, right hip, right knee, right ankle and trunk, and the embodiment of the present application does not limit the way of defining and distinguishing the skeleton nodes. When the user performs fitness exercises, the trunk, left arm, right arm, left leg and right leg coordinate with each other to complete a series of fitness exercises.

[0082] In some embodiments of the present application, the displacement distance and displacement direction of each skeletal node of the human skeletal model can be mapped to the positive direction of the X-axis, negative direction of the X-axis, positive direction of the Y-axis, negative direction of the Y-axis, positive direction of the Z-axis, and negative direction of the Z-axis of the three-dimensional coordinate system. When each skeletal node is displaced, calculations are performed through the three-dimensional coordinate system.

[0083] In some embodiments of the present application, the fitness movements and voice text demonstrated by the coach in the fitness video displayed on the display screen 105 are used to prompt the user to perform fitness movements synchronously. The image acquisition module 106 arranged on the upper edge of the display screen 105 of the smart screen 10 collects the posture information of the user performing fitness movements in real time and transmits it to the processor 101 of the smart screen 10 through a video transmission line or wireless communication. The processor 101 reads the fitness video and performs frame processing on the fitness video, and segments all the fitness movements involved in the fitness video (for example, a fitness movement in the fitness video is to raise the left leg from an initial position to the left side and then restore the left leg from the raised position to the initial position. The fitness movement can be divided into two sections, namely, the left leg is raised from the initial position to the left side as the first section, and the left leg is restored from the raised position to the initial position as the second section), and then the video of each fitness movement is subjected to video frame processing. The video frame processing is to divide each fitness movement in the fitness video into frames of pictures, and each frame in the video is a still picture, so as to obtain frame images of continuous moments of the fitness movement. After video framing, the total number of frames for each fitness action after framing is counted (taking raising the left leg to the left side as an example, the total number of frames after framing is 40 frames); the smart screen 10 collects the user's initial fitness action posture information collected by the camera, and counts the numbers or names of the bone nodes involved when the user initially performs a certain fitness action (taking raising the left leg to the left side as an example) (left hip joint 9, left knee joint 10 and left ankle joint 11). When the user initially performs the fitness action of raising the left leg to the left side, the distances moved by the bone nodes involved, the left hip joint 9, the left knee joint 10 and the left ankle joint 11, from the original position when standing on both feet to the final position after the left knee joint of the left leg is raised are respectively: the left hip joint moving distance, the left knee joint moving distance and the left ankle joint moving distance.

[0084] The smart screen 10 calculates the average of the moving distances of the left hip joint, the left knee joint and the left ankle joint when the user performs the fitness action of raising the left leg to the left side, to obtain the average moving distance of the joints involved in completing the fitness action of raising the left leg to the left side, and then calculates the ratio of the average moving distance to the total number of frames of the fitness action of raising the left leg to the left side, to obtain the number of frames that should be played for the fitness action of raising the left leg to the left side corresponding to the unit distance of movement of the joints involved in completing the fitness action of raising the left leg to the left side. When the user performs the fitness action of raising the left leg to the left side again, the average moving distance of the real-time movement of the left hip joint 9, the left knee joint 10 and the left ankle joint 11 of the user when raising the left leg to the left side is calculated in real time, and the fitness video is controlled to play the number of frames corresponding to the average distance of real-time movement of the fitness action of raising the left leg to the left side, so as to achieve the purpose of synchronization between the user performing the fitness action and the fitness video.

[0085] Example 1

[0086] FIG. 1( e ) exemplarily shows an exemplary user interface 10 on a smart screen 10 for displaying applications installed on the smart screen 10 .

[0087] The user interface 100 may include: a status bar 101, and application icons 102. Among them:

[0088] The status bar 101 may include: one or more signal strength indicators 101C of wireless fidelity (Wi-Fi) signals, a battery status indicator 101D, and a time indicator 101E.

[0089] In some embodiments, the status bar 101 may also include: one or more signal strength indicators of a mobile communication signal (also referred to as a cellular signal), and an operator name (eg, "China Mobile").

[0090] The application icons 102 may be: a fitness icon 102A, a gallery icon 102B, a music icon 102C, an application icon 102D, a contact icon 102E, a mailbox icon 102F, a cloud sharing icon 102G, a memo icon 102H, a setting icon 102I, and a camera icon 102J. The user interface 10 may also include a page indicator 103. Other application icons may be distributed on multiple pages, and the page indicator 103 may be used to indicate which page the user is currently browsing. The user may slide the area of ​​other application icons left and right to browse application icons on other pages.

[0091] In some embodiments of the present application, the user interface 100 may further include: a navigation bar 104 .

[0092] The navigation bar 104 may include system navigation keys such as the return key 104A, the home screen key 104B, and the multitasking key 104C. When it is detected that the user clicks the return key 104A, the smart screen 100 can display the previous page of the current page. When it is detected that the user clicks the home screen key 104B, the smart screen 100 can display the main interface. When it is detected that the user clicks the multitasking key 104C, the smart screen 100 can display the tasks that the user has recently opened. The naming of each navigation key can also be other, and this application does not limit this. Not limited to virtual buttons, each navigation key in the navigation bar 104 can also be implemented as a physical button.

[0093] In some embodiments, the user interface 100 exemplarily shown in FIG. 1( e ) may be a home screen.

[0094] In some other embodiments, the smart screen 100 may also include a front camera, which may also be called a sub-camera, and is mainly located above the screen of the smart screen 100. The front camera serves as the image acquisition module 106 in Figure 1(a) in the embodiment of the present application, and it can collect image information such as the posture and movements of the user in front of the smart screen 10, as well as selfies, etc.

[0095] In some other embodiments, the smart screen 10 may also include a home screen key. The home screen key may be a physical key or a virtual key. The home screen key may be used to receive instructions from the user and return the currently displayed UI to the main interface, so that the user can view the home screen at any time. The above instructions may specifically be an operation instruction for the user to press the home screen key once, or an operation instruction for the user to press the home screen key twice in a short period of time, or an operation instruction for the user to long press the home screen key within a predetermined time. In some other embodiments of the present application, the home screen key may also be integrated with a fingerprint reader so that fingerprint collection and identification can be performed when the home screen key is pressed.

[0096] It should be understood that Figure 1(e) merely illustrates the user interface on the smart screen 10 and should not constitute a limitation on the embodiments of the present application.

[0097] Exemplarily, as shown in FIG. 1( f ), the user may click on the fitness icon 102A on the user interface 100 , and the smart screen 10 detects the user operation. In response to the user operation, the smart screen 10 displays the interface 11 of the fitness app.

[0098] The user interface 11 may include: an application title bar 201, a search box 202, a setting 203, a function bar 204, and a display area 205. Among them:

[0099] The application title bar 201 can be used to indicate that the current page is used to display the setting interface of the smart screen 10. The application title bar 201 can be expressed in the form of text information "Smart Fitness", an icon or other forms.

[0100] The search box 202 may be used to search for fitness courses matching characters input by the user.

[0101] Setting 203 can receive user operations (such as touch operations). In response to the detected user operations, the smart screen 10 can display the settings interface of smart fitness.

[0102] The function bar 204 may include: a user center control 204A, a course recommendation control 204B, and multiple course classification controls. The multiple course classification controls may include but are not limited to: a fat burning area control 204C, a body shaping area control 204D, and a body shaping area control 204E.

[0103] The user center control 204A can receive user operations (such as touch operations). In response to the detected user operations, the smart screen 10 can display the interface content of the user's personal center in the display area 205.

[0104] The course recommendation control 204B can receive a user operation (e.g., a touch operation). In response to the detected user operation, the smart screen 10 can display one or more recommended fitness courses in the display area 205 of the display screen 105. For example, as shown in FIG1(e), the display area 205 displays course covers of multiple recommended courses, as well as the course category, duration, and name of each recommended course.

[0105] The controls in the above-mentioned multiple course classification controls can receive user operations (such as touch operations). In response to the detected user operations, the smart screen 10 can display the course covers of one or more fitness courses corresponding to the course classification controls in the display area 205 of the display screen 105.

[0106] In some embodiments of the present application, the course cover or name of the above-mentioned fitness course can receive a user's playback operation (such as a touch operation). In response to the detection of the above-mentioned playback operation, the smart screen 10 can display the specific content of the fitness course on the display screen.

[0107] The fitness app can be launched in response to a user's touch operation on the icon of the fitness app, such as a single click, double click, long press, etc. In some embodiments of the present application, the display screen is configured with a touch panel, which can be used to receive the user's touch operation, and the touch operation refers to the operation of the user's hand, elbow, stylus, etc. touching the display screen. In a specific implementation, there can also be other ways to open the interface 11 of the fitness app. This is not limited here.

[0108] For example, the user can open the fitness app interface 11 by pressing a button to start the first control mode; or, the user can open the fitness app interface 11 by detecting the voice input by the user; or, the user can open the fitness app interface 11 by drawing a specific shape (such as a Z shape, a rectangle, a circle, etc.) with the knuckles. This embodiment of the application does not specifically limit this.

[0109] In addition, the user can also control the smart screen 10 to display the interface 11 of the fitness app through the remote control; the user can also control the smart screen 10 to display the interface 11 of the fitness app through specific gestures. The user operation is not specifically limited in the embodiments of the present application.

[0110] It should be understood that Figure 1(e) merely illustrates an exemplary fitness app interface on the smart screen 10 and should not constitute a limitation on the embodiments of the present application.

[0111] Fitness courses usually include multiple movements, and there can be a preset rest time between two consecutive movements. Fitness courses can be recommended by the smart screen based on the user's historical fitness data, or selected by the user based on actual needs. Fitness courses can be played locally or online. There is no specific limitation here.

[0112] In some embodiments of the present application, the smart screen 10 can receive a user's playback operation. In response to detecting the above playback operation, the smart screen 10 can display the specific content of the fitness course in the fitness course display area of ​​the display screen, and at the same time display the user's body posture in real time in the user fitness display area of ​​the display screen.

[0113] In some embodiments of the present application, the smart screen 10 displays the user's fitness video in real time in the above-mentioned user fitness display area based on the video data collected by the camera.

[0114] The camera is used as the image acquisition module 106 of the smart screen. It can be the front camera of the smart screen 10, or the camera can be an external camera of the smart screen 10, which is not specifically limited here.

[0115] The following uses the fitness action of "standing straight with back and looking forward until the left knee is raised to the left" as an example to illustrate an adaptive motion synchronization method provided in an embodiment of the present application.

[0116] Exemplarily, as shown in FIG. 2( a ), the user may click on the recommended course displayed in the display area 205 , and the smart screen 10 detects the above user operation. In response to the above user operation, the smart screen 10 displays the training interface 12 .

[0117] The training interface 12 includes a fitness course window 301 and a user fitness window 302. Among them:

[0118] The fitness course window 301 is used to display the specific content of the fitness course.

[0119] The user fitness window 302 is used to display the user's fitness movements captured by the camera in real time.

[0120] The fitness video is played on the display screen, and the camera of the smart screen collects the user's movements and postures in real time. Before the display screen starts playing the fitness video, the movements of the coach and the user in the fitness video are in the initial state. The text description of the upcoming fitness movement is displayed on the initial UI interface, the fitness movement demonstrated by the coach is displayed on the right, the user's current movement posture is displayed on the left, and the fitness movement essentials with text prompts of the fitness movement are displayed in the middle of the display screen 105, and then the fitness video starts playing on the display screen.

[0121] In some embodiments of the present application, the smart screen 10 can display the user's fitness video in full screen and display a small window of the fitness course floating on the display screen.

[0122] For example, as shown in FIG2( b ), the smart screen 10 displays the user fitness window 302 in full screen on the training interface 12 , and displays the fitness course window 301 in the lower left corner of the display screen. In addition, the text prompt of the current fitness action is displayed on the training interface 12 .

[0123] In some embodiments of the present application, the smart screen 10 displays the fitness course in full screen, and a small window of the user's fitness video is suspended on the display screen.

[0124] In some embodiments of the present application, when a video interface is displayed in full screen, the video interface may occupy the entire display area of ​​the display screen. In one possible implementation, displaying a video interface in full screen means that only the video interface is displayed in the display screen without displaying other content. In another possible implementation, the video interface may also occupy only a portion of the display area of ​​the display screen, for example, when the video interface is displayed in the middle portion of the display screen and one or both edges are partially white or black, the display screen may also be regarded as displaying the video interface in full screen.

[0125] In other embodiments of the present application, displaying the video interface in full screen may mean that, while the video interface is displayed on the display screen, system-level interface elements, such as a status bar, a floating shortcut menu, etc., may also be displayed.

[0126] In addition to the window display methods shown in FIG. 2( a ) and FIG. 2( b ), the fitness course window 301 and the user fitness window 302 may also be displayed in other display methods, which are not specifically limited in the embodiments of the present application.

[0127] After the user performs the fitness movement shown in Figure 2(a), the smart screen 10 determines the key skeletal nodes involved in the fitness movement demonstrated by the coach in the fitness video and the key skeletal nodes involved in the user's current movement posture captured by the camera of the smart screen, and determines whether the fitness movement of the user and the coach in the fitness video is synchronized based on the key skeletal nodes. If not synchronized, the smart screen 10 controls and adjusts the playback speed of the fitness video.

[0128] In some embodiments of the present application, as shown in FIG2(c), the display screen of the smart screen displays in real time the coach's action posture of "keeping your back straight and looking forward until your left knee is raised to the left side", and at the current moment, the display screen displays the coach's action posture of "raising your left knee to the left side" (the display screen displays the 50th frame of the left knee raised to the standard position), and a text prompt of the fitness action is also displayed in the upper left corner of the display screen: "Keep your back straight and look forward until your left knee is raised to the left side". The user performs fitness accordingly according to the fitness actions and text prompts demonstrated by the coach on the display screen.

[0129] The lower left corner of the display screen displays in real time the user's movement posture of "raising the left knee to the left" (the display screen displays the 25th frame when the left knee is raised to half the distance of the standard position). It is obvious that when the user initially does the fitness movement of "keeping the back straight and looking forward to raising the left knee to the left", the user's fitness movement is not consistent with the fitness movement demonstrated by the coach in the fitness video. In other words, the user's fitness speed is slower than the fitness speed demonstrated by the coach in the fitness video. In order to remind the user, text prompts can be given on the display screen, such as the text prompt "Please note that the movements do not match".

[0130] In some embodiments of the present application, as shown in FIG. 2( d), the playback speed of the fitness video is adjusted, that is, the playback speed of the fitness video is adjusted to 0.5 times the speed and a text prompt "Currently playing at 0.5 times the speed" is displayed on the display screen, so that the next time the user performs the action, the posture of "raising the left knee to half the distance of the standard position" is used, and the 25th frame of the fitness action "keeping the back straight and looking forward until the left knee is raised to the left" in the fitness video is played, thereby ensuring that the playback speed of the fitness action demonstrated by the coach in the fitness video is synchronized with the speed at which the user performs the fitness action.

[0131] As shown in FIG. 3( a ), according to some embodiments of the present application, the process of performing motion adaptive synchronization through the present application is as follows:

[0132] Step S30: The power module supplies power to the smart screen, and the user operates the remote control to turn on the smart screen, and clicks on the fitness APP on the interactive interface of the display screen of the smart screen to open the fitness video on the fitness APP and play it. When the smart screen plays the fitness video, if the fitness video is pre-stored in the memory of the smart screen, the fitness video is directly extracted from the memory and played on the display screen. If it is not pre-stored in the memory, the fitness video is downloaded through the network and played on the display screen and cached in the memory. The fitness video contains multiple fitness movements demonstrated by a fitness coach, voice guidance, text prompts and other information. For example, in chronological order, the first fitness movement is to straighten the back and look forward to raise the left knee to the left side, the second fitness movement is to bend the body to the left with the elbow touching the knee, and the third movement is to raise the legs high (both the left leg and the right leg complete a high leg raise for a complete high leg raise), etc. When the coach demonstrates the fitness movement, a voice prompt will be given and the text of the voice will be displayed on the display screen of the smart screen.

[0133] Step S31: The smart screen reads each fitness movement in the fitness video in sequence according to the playback time sequence, and divides the video into frames for each fitness movement. Video framing is to divide each fitness movement in the fitness video into frames, and each frame in the video is a still picture, so that each fitness movement is divided into multiple frames of images, and each frame of each continuous fitness movement in the fitness video is a still image. For example, the first fitness movement of straightening the back, looking forward and raising the left knee to the left is divided into 50 frames, the second fitness movement of bending the body to the left with the elbow touching the left knee is divided into 25 frames, and the third movement of raising the leg high is divided into 50 frames.

[0134] Step S32: The smart screen identifies the motion trajectory of each fitness movement demonstrated by the coach in the fitness video, extracts the text description and voice description of each fitness movement, and parses the keywords therefrom to determine the information of the skeletal nodes involved in each fitness movement demonstrated by the coach. Among them, the motion trajectory of the fitness movement and the parsing of keywords can be achieved through machine learning, such as long short-term memory network (LSTM), support vector machine (SVM), etc. Take the first fitness movement played in the fitness video as an example, "straighten your back and look forward until your left knee is raised to the left side." While the coach is demonstrating the fitness movements, the coach voice broadcasts the prompt voice "straighten your back, look forward until your left knee is raised to the left side" and displays it on the display screen in text form. The smart screen recognizes the fitness movements based on LSTM and extracts the keywords "back", "knee", "left side" and "look forward" from the prompt voice and text to divide the body of the fitness movement into four parts: head, neck, trunk and left leg, and identifies the bone nodes involved in the above body parts as head center 1, neck center 2, trunk center 15, left hip joint point 9, left knee joint point 10 and left ankle joint point 11. In addition, based on machine learning, each fitness movement can also be segmented. For example, the first fitness movement played in the fitness video is "straighten your back, look forward until your left knee is raised to the left side". Based on LSTM, the fitness movement is divided into two sections, the first section is "straighten your back, look forward", and the second section is "raise your left knee to the left side". In this way, dividing a fitness movement into two sections can better synchronize with the user's fitness speed.

[0135] Step S33: When the display screen of the smart screen plays a fitness video, the user stands within the camera range facing the smart screen, and performs corresponding fitness movements following the fitness movements demonstrated by the coach, text prompts, etc. displayed on the display screen. When the user performs a certain fitness movement for the first time, the camera can collect the user's movement posture within the camera range at a sampling frequency of 5 times / second and transmit it to the processor. The processor extracts the fitness movement currently playing on the display screen from the memory based on the collected movement posture, and identifies the user's skeletal nodes that match the coach's skeletal nodes involved in the fitness movement currently playing on the display screen through machine learning (such as LSTM), and calculates the average moving distance of the skeletal nodes of all users involved in the fitness movement from the initial position to the end position.

[0136] In some embodiments of the present application, taking the first fitness action played in the fitness video as "straighten the back, look forward until the left knee is raised to the left" as an example, the fitness action has a total of 50 frames, and the skeletal nodes involved in the action are the head center 1, the neck center 2, the trunk center 15, the left hip joint 9, the left knee joint 10 and the left ankle joint 11, as shown in Figure 3(b), in the three-dimensional coordinate system, when the user first performs the action of raising the left knee to the left, the displacement directions of the left knee joint 10 and the left ankle joint 11 are the positive direction of the Z axis and the positive direction of the Y axis. Assuming that the displacement distance of the left knee joint 10 from the initial position of this fitness action to the positive direction of the Z axis is 80 cm, the displacement of the left knee joint 10 to the positive direction of the Y axis is The distance is 50cm, the displacement distance of the left ankle joint point 11 from the initial position of this fitness action to the positive direction of the Z axis is 60m, the displacement distance of the left ankle joint point 11 to the positive direction of the Y axis is 40cm, the displacement of the head center 1, the neck center 2, the left hip joint point 9, and the trunk center 15 is zero, the displacement distance of the left knee joint point 10 is about 90.2cm, and the displacement distance of the ankle joint point 11 is about 72.1cm. Then, the average displacement distance of the left knee joint point 10 and the left ankle joint point 11 when the user completes the fitness action of "straightening the back and looking forward to raising the left knee to the left side" is the average of the displacement distance of the left knee joint point 10 and the displacement distance of the ankle joint point 11, which is about 81.2cm.

[0137] Step S34: The smart screen calculates the speed of the user when doing the fitness action based on the average moving distance and the number of frames of the fitness action. In some embodiments of the present application, the fitness action of "straightening the back and looking forward to raising the left knee to the left" is still taken as an example. The fitness action has a total of 50 frames. When the user completes the fitness action of "straightening the back and looking forward to raising the left knee to the left", the displacement of the head center 1, the neck center 2, the left hip joint point 9, and the trunk center 15 is zero, and the average displacement distance of the left knee joint point 10 and the left ankle joint point 11 in the positive direction of the Z axis and the positive direction of the Y axis from the initial position of this fitness action is 81.2cm. Then the user's fitness speed when doing the fitness action of "straightening the back and looking forward to raising the left knee to the left" is

[0138] Step S35: The camera of the smart screen collects the user's movements and postures in real time at a sampling frequency of 5 times / second. When the user repeats the same fitness action, the real-time average movement distance of all skeletal nodes involved in the fitness action is calculated, and the picture frame corresponding to the average movement distance is played accordingly, so as to achieve the purpose of controlling the playback speed of the fitness video. In some embodiments of the present application, the fitness action of "straightening the back and looking forward to raising the left knee to the left" is still taken as an example. When the user is doing the fitness action of "straightening the back and looking forward to raising the left knee to the left", the displacement of the head center 1, the neck center 2, the left hip joint point 9, and the torso center 15 is zero, and the real-time average displacement distance of the left knee joint point 10 and the left ankle joint point 11 in the positive direction of the Z axis and the positive direction of the Y axis from the initial position of this fitness action is 40cm. The user's first time doing "straightening the back and looking forward to raising the left knee to the left" The speed of the "side raise" fitness action is 0.62 frames / cm. At this time, according to the time playback order, the fitness action of the fitness video is "keep your back straight, look forward and raise your left knee to the left side" and should be played to the 0.62×40 frame, which is about the 25th frame. In other words, when the real-time average displacement distance of the user's left knee joint 10 and left ankle joint 11 in the positive direction of the Z axis and the positive direction of the Y axis is 40cm, the fitness video played on the display screen of the smart screen is "keep your back straight, look forward and raise your left knee to the left side" and this fitness action should be played to the 25th frame. The user continues to perform the fitness action until the completion moment (the real-time average displacement distance of the left knee joint 10 and the left ankle joint 11 in the positive direction of the Z axis and the positive direction of the Y axis is 81.2cm), and the fitness action "keep your back straight, look forward and raise your left knee to the left side" played on the display screen of the smart screen should be played to the 50th frame.

[0139] Among them, when the real-time average displacement distance of the user's left knee joint 10 and left ankle joint 11 in the positive direction of the Z axis and the positive direction of the Y axis is 40cm, the normal playback frame number of the fitness action of "straighten the waist and back, look forward to raise the left knee to the left side" played on the display screen is 30 frames (as the target frame image). In this way, according to the actual distance moved by the user in a certain fitness action, the smart screen controls the playback speed of the fitness action of "straighten the waist and back, look forward to raise the left knee to the left side" accordingly, so that the more ideal frame number of the fitness action currently performed by the user in the fitness video corresponds to the actual distance moved by the user. As described above, when the real-time average displacement distance of the user's left knee joint 10 and left ankle joint 11 in the positive direction of the Z axis and the positive direction of the Y axis from the initial position of this fitness action is 40cm, the smart screen controls the fitness video "straighten the waist and back, look forward to raise the left knee to the left side" to play the 25th frame (obviously, the target frame image The time point corresponding to the image (the value of the time corresponding to the target frame image) is less than the picture of the user's current fitness action (the time point corresponding to the first posture feature)) instead of playing the 30th frame at the normal playback speed, that is, the smart screen controls the playback speed of the fitness action of "standing straight, looking forward to raising the left knee to the left" to be slower than the normal playback speed (0.8 times the speed) of the fitness action of "standing straight, looking forward to raising the left knee to the left"; in this way, the playback speed of the fitness action of "standing straight, looking forward to raising the left knee to the left" played on the display screen matches the actual speed of the user doing "standing straight, looking forward to raising the left knee to the left"

[0140] An embodiment of the present application discloses an adaptive motion synchronization method, in which the number of frames of each fitness action in the fitness video displayed on the display screen of the smart screen corresponds to the movement distance of each skeletal node collected when the user completes each fitness action, and the movement distance occurred when the user completes each fitness action corresponds to the number of frames that match the movement distance, so that the playback speed of the fitness action demonstrated by the coach in the fitness video is synchronized with the speed of the user performing the fitness action, thereby ensuring the continuity and rhythm of the user's fitness actions following the fitness video and improving the user's fitness experience.

[0141] Example 2

[0142] Figure 4(a) to Figure 4(c) is a UI interface diagram of another action adaptive synchronization method shown in an embodiment of the present application,

[0143] In some embodiments of the present application, as shown in FIG. 4( a), a fitness video is played on a display screen and the user's movements and postures are collected in real time. Before the fitness video starts playing, the movements of the coach and the user in the fitness video are in an initial state. A text description of the fitness movement that is about to begin is displayed on the initial UI interface, the fitness movement demonstrated by the coach is displayed on the right, the user's current movement posture is displayed in the lower left corner, and the fitness movement essentials with text prompts of the fitness movement are displayed in the upper left corner, and then the fitness video starts playing.

[0144] The smart screen determines the skeletal nodes involved in the fitness movements demonstrated by the coach in the fitness video and the skeletal nodes involved in the user's current movement posture. The smart screen determines whether the fitness movements played in the fitness video match the fitness movements performed by the user. If they match, the fitness video is played normally on the display screen. If they do not match, the smart screen adjusts the playback status of the fitness video on the display screen.

[0145] In some embodiments of the present application, as shown in FIG4(b), taking "body bent to the left with elbows and knees touching" as an example, the display screen displays in real time the coach's action posture of "body bent to the left with elbows and knees touching", and at the current moment the display screen displays the coach's action posture of "body bent to the left with elbows and knees touching" (the display screen displays the 25th frame of the elbows and knees touching), and a text prompt of the fitness action "body bent to the left with elbows and knees touching" is also displayed in the upper left corner of the display screen. The user performs fitness according to the fitness actions and text prompts demonstrated by the coach, and the user's real-time action posture is collected and displayed in real time in the lower left corner of the display screen as the third fitness action "high leg raise", and it is determined that the user has completed the second fitness action at this time, and a text prompt "Please note that the actions do not match" is displayed on the display screen to prompt the user.

[0146] In some embodiments of the present application, as shown in FIG4(c), the processor controls the fitness video to directly skip the second fitness action and displays a text prompt on the display screen "The previous action has been skipped for you", and switches to the third fitness action "High Leg Raise" currently being performed by the user. At this time, as shown in FIG4(c), the right side of the display screen displays the high leg raise action demonstrated by the coach, the upper left corner of the display screen displays the text prompt of the high leg raise, and the lower left corner of the display screen displays the real-time action posture of the user doing the high leg raise at this time. In this way, the synchronization of the fitness action of the fitness video currently playing on the display screen and the fitness action currently being performed by the user is ensured.

[0147] Figure 5 FIG. 1 is a flow chart of another method for adaptive synchronization of actions according to an embodiment of the present application. Figure 5 As shown, in one embodiment, the process of performing interactive fitness through this application is as follows:

[0148] Step S50: The user clicks on the fitness APP on the interactive interface of the display screen of the smart screen, and selects a fitness plan or fitness course in the fitness APP (specifically, the fitness plan may include one or more fitness courses, which is not limited in this application). If the smart screen has pre-stored fitness courses, the fitness video is directly extracted and played on the display screen. The smart screen can also obtain fitness courses through the Internet and play and cache them in the memory. The fitness course includes multiple fitness movements demonstrated by fitness coaches, voice guidance, text prompts and other information. For example, in chronological order, the first fitness movement is to straighten the back and look forward until the left knee is raised to the left side, the second fitness movement is to bend the body to the left with the elbow touching the knee, and the third movement is to raise the legs high (both the left leg and the right leg complete a high leg raise for a complete high leg raise), etc. When the coach demonstrates the fitness movement, a voice prompt will be given and the text of the voice will be displayed on the display screen of the smart screen. It is worth noting that fitness videos can also be played in other ways (such as users sending voice commands to the smart screen through voice interaction (such as playing videos), for example, the user says fitness, and the smart screen automatically opens the fitness APP). The embodiments of the present application are not limited to the user clicking on a fitness video to play the fitness video.

[0149] Step S51: The smart screen reads each fitness movement in the fitness video in sequence according to the playback time. Each continuous fitness movement in the fitness video can be composed of multiple static picture frames. For example, the first fitness movement is to straighten the back and look forward to raise the left knee to the left, which is divided into 50 frames, the second fitness movement is to bend the body to the left with the elbow touching the left knee, which is divided into 25 frames, and the third movement is to raise the leg high, which is divided into 50 frames.

[0150] Step S52: The smart screen identifies the motion trajectory of each fitness action demonstrated by the coach in the fitness video, extracts the text description and voice description of each fitness action, and parses keywords from them to determine the information (number or name) of the skeletal nodes involved in each fitness action. Among them, the motion trajectory of the fitness action and the analysis of keywords can be achieved through machine learning, such as Long Short-Term Memory (LSTM), Support Vector Machine (SVM), etc. Taking the first fitness movement played in the fitness video as an example, "Keep your back straight, look forward and raise your left knee to the left", while the coach demonstrates the fitness movement, the coach voice broadcasts the prompt "Keep your back straight, look forward and raise your left knee to the left" and displays it in text form on the display screen. The smart screen recognizes fitness movements based on LSTM and extracts the keywords "back", "knee", "left side" and "look forward" to divide the body into four parts: head, neck, torso and left leg, and identifies the bone nodes involved in the above body parts as head center 1, neck center 2, torso center 15, left hip joint point 9, left knee joint point 10 and left ankle joint point 11.

[0151] Step S53: When the display screen of the smart screen plays a fitness video, the user stands within the camera range facing the camera of the smart screen, and performs corresponding fitness movements following the fitness movements demonstrated by the coach, text prompts, etc. displayed on the display screen. When the user performs a certain fitness movement, the camera can collect the user's movement posture within the camera range at a sampling frequency of 5 times / second and transmit it to the processor. The processor extracts the fitness movement currently playing on the display screen from the memory based on the collected movement posture, and identifies the user's bone nodes corresponding to the coach's bone nodes involved in the fitness movement currently playing on the display screen through machine learning (such as LSTM). Taking the fitness movement "straighten your back and look forward until the left knee is raised to the left" as an example, the user's bone nodes are identified by LSTM as head center 1, neck center 2, torso center 15, left hip joint point 9, left knee joint point 10 and left ankle joint point 11.

[0152] Step S54: The smart screen captures in real time the motion trajectory and posture of the skeletal nodes corresponding to the fitness movements in the fitness video currently played on the display screen, as well as the motion trajectory and posture of the skeletal nodes involved in the user's current fitness movements collected through the camera. The smart screen determines whether the fitness movements played in the fitness video match the fitness movements performed by the user. If they match, it enters S55; if not, it enters S56.

[0153] Among them, it is possible to judge whether the fitness movements played in the fitness video match the fitness movements performed by the user (action posture) by the coach's skeletal node number or name involved in the fitness movements currently played on the display screen and the user's skeletal node name or number involved in the fitness movements currently performed by the user, and judge whether the fitness movements played in the fitness video match the fitness movements performed by the user by the motion trajectory. For example, in chronological order of playback, the first fitness movement of the fitness video displayed on the display screen is to straighten the back and look forward until the left knee is raised to the left, and the second fitness movement is to bend the body to the left with the elbow touching the knee. The camera of the smart screen captures the user performing the second fitness movement of "bending the body to the left with the elbow touching the knee" and transmits it to the processor.

[0154] The processor uses LSTM to learn that the skeletal nodes involved in the second fitness action "body flexed to the left with elbows and knees touching" currently performed by the user are head center 1, neck center 2, trunk center 15, left hip joint 9, left knee joint 10, left elbow joint 5, left wrist joint 7, and left ankle joint 11. The smart screen analyzes the user's posture and motion trajectory captured by the camera, that is, the user's posture is body flexion to the left, elbows and knees touching, and the motion trajectory is: the body changes from straight to left flexion, and the elbows and knees are close, that is, the head center 1, neck center 2 and trunk center 15 tilt to the left, the left knee joint 10 is lifted up, the left elbow joint 5 and the left wrist joint are 7 approaches the left knee joint 10, so the smart screen determines that the user has completed the first fitness action; the fitness action currently played on the display screen extracted by the smart screen from the memory is "raise the left knee to the left side", and the smart screen analyzes the motion trajectory of the fitness action demonstrated by the coach currently playing on the display screen, that is, the coach's action posture is to raise the left knee, and the motion trajectory is: the left knee is raised from the initial position to the left side, that is, the left knee joint 10 and the left ankle joint 11 are lifted from bottom to top, so the smart screen determines that the current display screen is playing the first fitness action of the fitness video. At this time, the smart screen determines that the fitness action currently performed by the user does not match the fitness action currently played on the display screen.

[0155] S55: The fitness video is played normally on the display.

[0156] S56: The smart screen adjusts the playing status of the fitness video on the display screen, as follows: According to the time playback order, if the user has completed the fitness action of the fitness video currently playing on the display screen (the moment point of the frame image of the fitness action of the currently playing fitness video (the time of the fitness action of the currently playing fitness video) does not exceed the moment point of the target frame image (the time of the fitness action of the target frame image), wherein the moment relationship between the moment point of the frame image of the fitness action of the currently playing fitness video and the moment point of the target frame image is: the time of the frame image of the fitness action of the currently playing fitness video is earlier than the time of the target frame image), then the smart screen controls the fitness video to play the fitness action currently playing on the display screen at a high speed (such as 4 times the speed as the target speed) or directly skips the fitness action currently playing on the display screen (the fitness action currently playing is the frame image corresponding to the first posture feature) and plays the next fitness action (target frame image) that the user is performing. If the display screen has completed playing the fitness movements of the fitness video, and the smart screen determines that the user has not completed the fitness movement based on the user's movement posture and movement trajectory captured in real time by the camera, the smart screen will pause playing the fitness video and display a text prompt of the current user's fitness movement on the display screen of the smart screen until the camera captures in real time that the user has completed the fitness movement that has been played and the user's current fitness movement matches the movement posture corresponding to the currently paused screen. Then, the pause is canceled and the video playback is resumed from the current pause moment, or the fitness video is replayed to the screen corresponding to the user's current movement posture to perform the fitness movement synchronously again.

[0157] In some embodiments of the present application, such as in chronological order of playback, the first fitness movement of the fitness video displayed on the display screen is to straighten the back, look forward and raise the left knee to the left, and the second fitness movement is to bend the body to the left with the elbow and knee touching. The camera of the smart screen captures the user's second fitness movement of "bending the body to the left with the elbow and knee touching" and transmits it to the processor.

[0158] The processor uses LSTM to learn that the skeletal nodes involved in the second fitness action currently performed by the user are head center 1, neck center 2, trunk center 15, left hip joint point 9, left knee joint point 10, left elbow joint point 5, left wrist joint point 7, and left ankle joint point 11. The smart screen analyzes the user's posture and motion trajectory captured by the camera, that is, the user's posture is left flexion, the elbow and knee are touching, and the motion trajectory is: the body changes from straight to left flexion, the elbow and knee are close, that is, the head center 1, neck center 2 and trunk center 15 are tilted to the left, the left knee joint point 10 is lifted up, and the left elbow is closed. Node 5 and left wrist joint 7 approach left knee joint 10, so the smart screen determines that the user has completed the first fitness action; the fitness action currently played on the display screen extracted from the memory by the smart screen is "raise the left knee to the left side", and the smart screen analyzes the motion trajectory of the fitness action currently played by the coach on the display screen, that is, the coach's action posture is to raise the left knee, and the motion trajectory is: the left knee is raised from the initial position to the left side, that is, the left knee joint 10 and the left ankle joint 11 are raised from bottom to top, so the smart screen determines that the current display screen is playing the first fitness action of the fitness video. At this time, the camera of the smart screen will collect the user's current fitness action in real time. The smart screen can play the fitness video at 4 times the speed until the fitness action played on the display screen matches the user's current fitness action, and then resume playing the fitness video at normal speed.

[0159] An adaptive motion synchronization method disclosed in an embodiment of the present application can recognize in real time the motion posture and motion trajectory of a coach in a fitness video played on a display screen, as well as the motion posture and motion trajectory of a user's fitness movements captured by a camera of a smart screen. When the two do not match, the playback status of the fitness video played on the display screen is adjusted so that the fitness movements in the fitness video currently played on the display screen are synchronized with the fitness movements currently performed by the user, thereby ensuring the continuity and fluency of the user when performing fitness.

[0160] Example 3

[0161] Figure 6(a) to Figure 6(c) is a UI interface diagram of another action adaptive synchronization method shown in an embodiment of the present application,

[0162] In some embodiments of the present application, as shown in FIG6(a), a fitness video is played on the display screen and the user's movements and postures are collected in real time. Before the fitness video starts playing, the movements of the coach and the user in the fitness video are in an initial state. A text description of the fitness movement that is about to begin is displayed on the initial UI interface, the fitness movement demonstrated by the coach is displayed on the right, the user's current movement posture is displayed in the lower left corner, and the fitness movement essentials with text prompts of the fitness movement are displayed in the upper left corner, and then the smart screen controls to start playing the fitness video.

[0163] The smart screen determines whether the fitness movement performed by the user is standard. If it is not standard, the smart screen will treat the fitness movement as a non-standard fitness movement and calculate the fitness speed of the user performing the non-standard fitness movement. The smart screen will play the standard fitness movement demonstrated by the coach corresponding to the non-standard fitness movement (incorrect fitness movement) in the fitness video at the fitness speed to correct the incorrect fitness movement performed by the user.

[0164] In some embodiments of the present application, as shown in FIG6(b), taking "back straight, eyes looking forward until left knee is raised to the left" as an example, the display screen displays in real time the coach's action posture of "back straight, eyes looking forward until left knee is raised to the left", and at the current moment the display screen displays the coach's action posture of "left knee raised to the left" (the display screen displays the 50th frame of the left knee raised to the standard position), and a text prompt of the fitness action "back straight, eyes looking forward until left knee is raised to the left" is also displayed in the upper left corner of the display screen, and the user Comparing the fitness movements demonstrated by the coach and the text prompts on the display screen, the user performs fitness accordingly, collects the user's real-time movement posture and displays the user's movement posture as "raise the right knee to the right" in real time in the lower left corner of the display screen. It is obvious that when the user is doing the fitness movement of "straighten the back, look forward to raise the left knee to the left", it does not match the fitness movement demonstrated by the coach of "straighten the back, look forward to raise the left knee to the left", indicating that there is an error in the fitness movement performed by the user at this time, and a text prompt "Please note that the movements do not match" is displayed on the display screen.

[0165] After the entire fitness video is played, the standard fitness action "raise the left knee to the left side" demonstrated by the coach and the fitness action "raise the right knee to the right side" performed incorrectly by the user are played synchronously on the display screen. In some embodiments of the present application, as shown in FIG6(c), a video of the fitness action performed incorrectly by the user can be displayed on the right side of the display screen, the standard fitness action demonstrated by the coach can be displayed in the lower left corner of the display screen, and a text prompt of the standard fitness action demonstrated by the coach can be displayed in the upper left corner to compare and explain with the standard fitness video demonstrated by the coach, and a text prompt "wrong action occurred, please watch your action playback" can be displayed on the display screen to remind the user and correct the fitness action performed incorrectly by the user.

[0166] Figure 7 FIG. 1 is a flow chart of another method for adaptive synchronization of actions according to an embodiment of the present application. Figure 7 As shown, in one embodiment, the process of performing interactive fitness through this application is as follows:

[0167] Step S70: The power module supplies power to the smart screen, the user operates the remote control to turn on the smart screen, and clicks on the fitness APP on the interactive interface of the display screen of the smart screen to open the fitness video on the fitness APP and play it. When the smart screen plays the fitness video, if the fitness video is pre-stored in the memory of the smart screen, the fitness video is directly extracted from the memory and played on the display screen. If it is not pre-stored in the memory, the fitness video is downloaded through the network and played on the display screen and cached in the memory. The fitness video contains multiple fitness movements demonstrated by a fitness coach, voice guidance, text prompts and other information. For example, in chronological order, the first fitness movement is to straighten the back and look forward to raise the left knee to the left side, the second fitness movement is to bend the body to the left with the elbow touching the knee, and the third movement is to raise the legs high (both the left leg and the right leg complete a high leg raise for a complete high leg raise), etc. When the coach demonstrates the fitness movement, a voice prompt will be given and the text of the voice will be displayed on the display screen of the smart screen.

[0168] Step S71: The processor of the smart screen reads each fitness movement in the fitness video in sequence according to the playback time sequence. Each continuous fitness movement in the fitness video can be composed of multiple still picture frames. For example, the first fitness movement is straightening the back and looking forward to raising the left knee to the left, which is divided into 50 frames, the second fitness movement is bending the body to the left with the elbow touching the left knee, which is divided into 25 frames, and the third movement is raising the leg high, which is divided into 50 frames.

[0169] Step S72: The smart screen identifies the motion trajectory of each fitness action demonstrated by the coach in the fitness video, extracts the text description and voice description of each fitness action, and parses keywords from them to determine the information (number or name) of the skeletal nodes involved in each fitness action in the fitness video. Among them, the motion trajectory of the fitness action and the parsing of keywords can be achieved through machine learning, such as Long Short-Term Memory (LSTM), Support Vector Machine (SVM), etc. Taking the first fitness movement played in the fitness video as an example, "Keep your back straight, look forward and raise your left knee to the left", while the coach demonstrates the fitness movement, the coach voice broadcasts the prompt "Keep your back straight, look forward and raise your left knee to the left" and displays it in text form on the display screen. The smart screen recognizes fitness movements based on LSTM and extracts the keywords "back", "knee", "left side" and "look forward" to divide the body into four parts: head, neck, torso and left leg, and identifies the bone nodes involved in the above body parts as head center 1, neck center 2, torso center 15, left hip joint point 9, left knee joint point 10 and left ankle joint point 11.

[0170] Step S73: When the display screen of the smart screen plays a fitness video, the user stands within the camera range of the camera facing the smart screen, and performs fitness movements corresponding to the fitness movements, text prompts, etc. demonstrated by the coach displayed on the display screen. When the user performs a certain fitness movement for the first time, the camera can collect the user's action posture within the camera range at a sampling frequency of 5 times / second and transmit it to the processor. The processor extracts the fitness movement currently played on the display screen from the memory based on the collected action posture, and identifies the user's bone nodes that match the coach's bone nodes involved in the fitness movement currently played on the display screen through machine learning (such as LSTM), and calculates the average movement distance of the bone nodes of all users involved in the fitness movement from the initial position to the end position. Take the first fitness movement played in the fitness video as "straighten the waist and back, look forward to the left knee and raise it to the left side" as an example. The fitness movement has a total of 50 frames, and the bone nodes involved in the movement are the head center 1, the neck center 2, the trunk center 15, the left hip joint point 9, the left knee joint point 10 and the left ankle joint point 11.

[0171] In some embodiments of the present application, as shown in FIG3( b), in a three-dimensional coordinate system, when the user first raises the left knee to the left, the displacement directions of the left knee joint 10 and the left ankle joint 11 are the positive direction of the Z axis and the positive direction of the Y axis. Assuming that the displacement distance of the left knee joint 10 in the positive direction of the Z axis is 80 cm, the displacement distance of the left knee joint 10 in the positive direction of the Y axis is 50 cm, the displacement distance of the left ankle joint 11 in the positive direction of the Z axis is 60 m, and the displacement distance of the left ankle joint 11 in the positive direction of the Y axis is The distance between the left knee joint point 10 and the left ankle joint point 11 is 40 cm, the displacement of the head center 1, the neck center 2, the left hip joint point 9 and the trunk center 15 is zero, the displacement distance of the left knee joint point 10 is about 90.2 cm, and the displacement distance of the ankle joint point 11 is about 72.1 cm. The average displacement distance of the left knee joint point 10 and the left ankle joint point 11 when the user completes the fitness action of "straightening the back and looking forward to raising the left knee to the left" is the average of the displacement distances of the left knee joint point 10 and the ankle joint point 11, which is about 81.2 cm.

[0172] After the user completes the first fitness exercise, he / she immediately performs the second fitness exercise of bending the body to the left with the elbow touching the left knee, until the user completes all fitness exercises instructed in the fitness video.

[0173] Step S74: The camera transmits the user's fitness movement posture, movement trajectory and other information collected in real time to the processor and memory of the smart screen. The memory stores the video of each fitness movement of the user in chronological order. The processor calculates the speed of each fitness movement of the user in real time and stores the speed of each fitness movement calculated in real time to the memory.

[0174] Step S75: The smart screen extracts from the memory the standard fitness movements demonstrated by the coach corresponding to each fitness movement performed by the user, identifies and matches them to determine whether the fitness movements performed by the user are standard. When the fitness movements performed by the user are not standard, step S751 is executed to store the fitness movement as an erroneous movement in the memory. When the fitness movement performed by the user is standard, step S752 is executed to define the fitness movement as a standard movement, and the standard movement is optionally stored in the memory.

[0175] In some embodiments of the present application, the fitness action of "straightening the back and looking forward to raising the left knee to the left" is still taken as an example. The fitness action has a total of 50 frames. When the user completes the fitness action of "straightening the back and looking forward to raising the left knee to the left", the displacement of the head center 1, the neck center 2, the left hip joint point 9, and the trunk center 15 is zero, and the average displacement distance of the left knee joint point 10 and the left ankle joint point 11 in the positive direction of the Z axis and the positive direction of the Y axis is 81.2 cm. The speed of the user when doing the fitness action of "straightening the back and looking forward to raising the left knee to the left" is The smart screen stores the speed of the fitness action of 0.62 frames / cm in the memory. When the user starts to do the fitness action of "straighten the back and look forward to raise the left knee to the left", the user's right knee is raised to the left or the right, the camera captures the user's action posture at this time (right knee raised to the right) and transmits it to the processor. The processor recognizes the user's right knee joint point 13 and right ankle joint point 14 in the action posture based on the LSTM network. The processor recognizes the coach's bone node of the fitness action of "straighten the back and look forward to raise the left knee to the left" in the fitness video as the left knee joint point 10 based on the LSTM network. and left ankle joint 11. In this fitness action, the smart screen will identify the key skeletal nodes of the user, right knee joint 13 and right ankle joint 14, and the key skeletal nodes of the coach, left knee joint 10 and left ankle joint 11, and compare and match the skeletal node numbers or names. At this time, the skeletal node numbers of the user compared by the smart screen are inconsistent with the skeletal node numbers of the coach in the fitness video. The user's fitness action of "straightening the back, looking forward to the left knee and raising it to the left side" is judged to be an incorrect action, and the incorrect fitness action of "straightening the back, looking forward to the left knee and raising it to the left side" recorded by the camera is stored in the memory. It is worth noting that the method for calculating the speed of the user's fitness action is the same as that in the above embodiment, and the embodiments of the present application will not be repeated here.

[0176] Step S76: After the user has completed all the fitness movements instructed in the fitness video, the smart screen calculates the average speed of each fitness movement. Taking the fitness movement of "straighten your back and look forward to raise your left knee to the left" as an example, the user needs to do "straighten your back and look forward to raise your left knee to the left" 3 times in total in the entire fitness video. The smart screen extracts the speed of the user's first fitness movement of "straighten your back and look forward to raise your left knee to the left" from the memory, and the speed is The smart screen extracts from the memory the speed of the user's second fitness movement of "standing straight, looking forward to raising the left knee to the left" and the speed is 0.66 frames / cm. The smart screen extracts from the memory the speed of the user's third fitness movement of "standing straight, looking forward to raising the left knee to the left" and the speed is 0.64 frames / cm. Therefore, the average speed of the user when performing the fitness movement of "standing straight, looking forward to raising the left knee to the left" is 0.64 frames / cm.

[0177] Step S77: After the fitness video on the display screen of the smart screen is completed, the camera on the smart screen stores the recorded user's fitness video to the memory, and the smart screen extracts the non-standard fitness action that the user has done wrong and the average speed of the user doing the non-standard fitness action from the memory, and the smart screen plays the non-standard fitness action and the standard fitness action of the coach corresponding to the non-standard fitness action on the display screen at the same screen, wherein, when playing the standard fitness action of the coach corresponding to the non-standard fitness action, the standard fitness action demonstrated by the coach is played at the average speed of the non-standard fitness action done by the user. Taking step S44 as an example, when the user does the fitness action of "straightening the waist and back, looking forward to the left knee to the left side", it is a non-standard fitness action, and the average speed of this fitness action is 0.64 frames / cm, then the smart screen controls the non-standard action video of the user "straightening the waist and back, looking forward to the left knee to the left side" stored in the memory and the standard action video of "straightening the waist and back, looking forward to the left knee to the left side" in the fitness video to be played at a speed of 0.64 frames / cm. When playing the non-standard action video, voice explanations and voice prompts are given to the user to correct the user's incorrect fitness actions.

[0178] An embodiment of the present application discloses an adaptive motion synchronization method, in which a smart screen stores a video of a user's incorrect fitness movements in a memory. After the entire fitness video on the display screen is played, the smart screen extracts the user's incorrect fitness videos from the memory and compares and explains them with standard fitness videos demonstrated by a coach to correct the user's incorrect fitness movements, which is beneficial to the user's physical health and further enhances the user's sense of experience.

[0179] In some embodiments of the present application, an electronic device is also provided. Figure 8 The electronic device in the embodiment of the present application is introduced. Figure 8 A schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application.

[0180] For at least one embodiment, the controller hub 804 communicates with the processor 801 via a multi-drop bus such as a front-side bus (FSB), a point-to-point interface such as a QuickPath Interconnect (QPI), or a similar connection. The processor 801 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 804 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which can be on a separate chip) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0181] The electronic device 800 may also include a coprocessor 806 and a memory 802 coupled to the controller hub 804. Alternatively, one or both of the memory 802 and the GMCH may be integrated within the processor 801 (as described in this application), with the memory 802 and the coprocessor 806 being directly coupled to the processor 801 and the controller hub 804, with the controller hub 804 being in a single chip with the IOH.

[0182] In one embodiment, the memory 802 may be, for example, a dynamic random access memory (DRAM), a phase change memory (PCM), or a combination of the two. The memory 802 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. The computer-readable storage medium stores instructions, and more specifically, stores temporary and permanent copies of the instructions.

[0183] In one embodiment, coprocessor 806 is a special purpose processor, such as, for example, a high throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, or an embedded processor, etc. The optional nature of coprocessor 806 is indicated by dashed lines in FIG. Figure 8 middle.

[0184] In one embodiment, the electronic device 800 may further include a network interface (NIC) 803. The network interface 803 may include a transceiver for providing a radio interface for the device 800 to communicate with any other suitable device (such as a front-end module, an antenna, etc.). In various embodiments, the network interface 803 may be integrated with other components of the electronic device 800. The network interface 803 may implement the functions of the communication unit in the above-mentioned embodiments.

[0185] In one embodiment, Figure 8 As shown, the electronic device 800 may further include an input / output (I / O) device 805. The input / output (I / O) device 805 may include: a user interface designed to enable a user to interact with the electronic device 800; a peripheral component interface designed to enable peripheral components to interact with the electronic device 800; and / or a sensor designed to determine environmental conditions and / or location information related to the electronic device 800.

[0186] It is worth noting that Figure 8 This is for illustrative purposes only. Figure 8 It is shown that the electronic device 800 includes multiple devices such as a processor 801, a controller hub 804, and a memory 802. However, in actual applications, the devices using the methods of the present application may only include a part of the devices of the electronic device 800, for example, it may only include the processor 801 and the NIC 803. Figure 8The properties of optional devices are shown with dotted lines.

[0187] In some embodiments of the present application, the instructions stored in the computer-readable storage medium of the electronic device 800 may include: instructions that, when executed by at least one unit in the processor, cause the device to implement the motion adaptive synchronization method mentioned in Embodiment 1, Embodiment 2, and Embodiment 3. When the instructions are executed on the computer, the computer executes the motion adaptive synchronization method mentioned in Embodiment 1, Embodiment 2, and Embodiment 3.

[0188] Reference now Fig. 9 , Fig. 9 1 is a schematic diagram of a structure of a SOC disclosed in an embodiment of the present application, showing a block diagram of a SoC (System on Chip) 1000 according to an embodiment of the present application. Fig. 9 In the figure, similar components have the same reference numerals. In addition, the dotted box is an optional feature of a more advanced SoC. The SoC can be used in an electronic device according to an embodiment of the present application, and corresponding functions can be implemented according to the instructions stored therein.

[0189] exist Fig. 9 In the embodiment, SoC 1000 includes: an interconnect unit 1002, which is coupled to processor 1001; a system agent unit 1006; a bus controller unit 1005; an integrated memory controller unit 1003; a group or one or more coprocessors 1007, which may include integrated graphics logic, image processors, audio processors, and video processors; a static random access memory (SRAM) unit 1008; and a direct memory access (DMA) unit 1004. In one embodiment, coprocessor 1007 includes a special-purpose processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0190] One or more computer-readable media for storing data and / or instructions may be included in the static random access memory (SRAM) unit 1008. The computer-readable storage medium may store instructions, and more specifically, temporary and permanent copies of the instructions.

[0191] When the SoC 1000 is applied to an electronic device according to the present application, the instructions stored in the computer-readable storage medium may include: instructions that, when executed by at least one unit in the processor, cause the electronic device to implement the motion adaptive synchronization method mentioned in Embodiment 1, Embodiment 2, and Embodiment 3. When the instructions are executed on a computer, the computer executes the motion adaptive synchronization method mentioned in Embodiment 1, Embodiment 2, and Embodiment 3.

[0192] In addition, an embodiment of the present application further discloses a computer-readable storage medium, on which a processing program is stored. When the processing program is executed by a processor, the method for adaptive synchronization of actions as mentioned in Embodiment 1, Embodiment 2, and Embodiment 3 is implemented.

[0193] The computer-readable storage medium may be a read-only memory, a random access memory, a hard disk, or an optical disk.

Claims

1. A motion adaptive synchronization method, characterized in that: Applied to electronic equipment, the motion adaptive synchronization method comprises: The electronic device plays a fitness video, wherein the fitness video is a fitness video in which a fitness coach demonstrates fitness movements; The electronic device collects the user's motion posture, wherein the user's motion posture is the user's motion posture when currently performing a fitness action; The electronic device matches the currently played fitness action in the fitness video with the action posture of the user, and matches the motion trajectory corresponding to the fitness action with the motion trajectory of the user performing the fitness action; If the fitness action does not match the action posture of the user, and the motion trajectory corresponding to the fitness action does not match the motion trajectory of the user performing the fitness action, the electronic device adjusts the playback speed of the fitness action in the fitness video to synchronize the fitness action with the action posture of the user.

2. The motion adaptive synchronization method according to claim 1, characterized in that: The electronic device matches the fitness action currently played in the fitness video with the action posture of the user, including: The electronic device obtains one or more skeleton nodes of at least one fitness coach in the fitness video; The electronic device obtains one or more skeleton nodes of the user; The electronic device determines key skeletal nodes from one or more skeletal nodes of at least one fitness coach in the fitness video and one or more skeletal nodes of the user.

3. The method according to claim 2, characterized in that Determining the key skeletal nodes includes: The electronic device selects key skeletal nodes of the currently playing fitness action from the skeletal nodes of the user and the at least one fitness coach, wherein the key skeletal nodes are skeletal nodes involved in the currently playing fitness action.

4. The motion adaptive synchronization method according to claim 3, characterized in that: The electronic device matches the fitness action currently played in the fitness video with the action posture of the user, including: The electronic device extracts key skeletal nodes of the fitness action currently demonstrated by at least one fitness coach in the current fitness video and the action posture of the user based on LSTM, obtains the third posture feature of the currently demonstrated fitness action and the fourth posture feature of the user, and matches the third posture feature with the fourth posture feature based on the key skeletal nodes.

5. The motion adaptive synchronization method according to claim 2, characterized in that: The electronic device matches the fitness action currently played in the fitness video with the action posture of the user, including: The electronic device extracts key skeletal nodes of a fitness movement currently demonstrated by at least one fitness coach in the fitness video and the movement posture of the user based on LSTM, obtains a first posture feature of the currently demonstrated fitness movement and a second posture feature of the user, and matches the first posture feature with the second posture feature.

6. The motion adaptive synchronization method according to claim 5, characterized in that: The fitness video is divided into frames to obtain frame images of the fitness video at consecutive moments.

7. The motion adaptive synchronization method according to claim 6, characterized in that: The electronic device determines a target frame image corresponding to a second posture feature of the user; Based on the time relationship between the frame image corresponding to the first posture feature and the target frame image, the playback speed of the fitness video is adjusted so that the fitness action corresponding to the target frame image is synchronized with the action posture of the user.

8. The motion adaptive synchronization method according to claim 7, characterized in that: The determining of the target frame image corresponding to the second posture feature of the user comprises: The electronic device determines the target frame image based on the total number of frames and the average moving distance of the complete fitness action corresponding to the second posture feature, wherein the average moving distance is the average moving distance of the key skeletal nodes corresponding to the second posture feature from the initial position to the end position of the complete fitness action.

9. The motion adaptive synchronization method according to claim 8, characterized in that: The average moving distance is the average value of the moving distances of the key skeletal nodes corresponding to the second posture feature from the initial position to the end position in the three-dimensional coordinate system in the X-axis, Y-axis and Z-axis directions.

10. The motion adaptive synchronization method according to claim 7, characterized in that: When the time point corresponding to the target frame image is greater than the time point of the frame image corresponding to the first posture feature, the electronic device adjusts the playback speed of the fitness video to the target speed, wherein the target speed is greater than the current playback speed of the fitness video.

11. The motion adaptive synchronization method according to claim 7, characterized in that: When the time point corresponding to the target frame image is greater than the time point of the frame image corresponding to the first posture feature, the electronic device controls the fitness video to jump directly from the currently playing fitness action to the target frame image.

12. The motion adaptive synchronization method according to claim 7, characterized in that: When the time point corresponding to the target frame image is less than the time point of the frame image corresponding to the first posture feature, the electronic device controls the playback of the fitness video and directly jumps from the currently playing fitness action to the target frame image.

13. The motion adaptive synchronization method according to any one of claims 1 to 5, characterized in that: When the fitness action is not synchronized with the user's action posture, the electronic device performs voice prompts and / or text prompts.

14. The motion adaptive synchronization method according to claim 1, characterized in that: The electronic device stores the user's motion gestures; The electronic device determines, based on a standard fitness action corresponding to the action posture of the user, that the standard fitness action does not match the action posture standard of the user; then marking the user's action posture as a non-standard action and replaying the standard fitness action corresponding to the non-standard action in the fitness video; The electronic device replays the standard fitness action in the fitness video at the fitness speed of the user performing the fitness action corresponding to the non-standard action, wherein the standard fitness action corresponds to the fitness action corresponding to the non-standard action; The electronic device provides text and / or voice explanation of the non-standard action.

15. An electronic device, characterized in that: include: A memory storing synchronization instructions; A processor, wherein when executing the synchronization instruction, the processor implements the steps of the action adaptive synchronization method according to any one of claims 1 to 14.

Images