Display device and device control method

Abstract

The present disclosure provides a display device and a device control method. The method comprises: in response to an operation of starting a workout program video, acquiring an action group list associated with the workout program video, wherein the workout program video comprises an action group clip corresponding to at least one standard action, and the action group list comprises action information and action start and end time of the standard action; displaying the workout program video in a first window of a follow-along interface, and displaying follow-along images of a user in a second window; when the workout program video is played back to a first action group clip corresponding to a first standard action, comparing reference action images comprised in the first action group clip with aligned first follow-along images, and calculating workout scores corresponding to the first follow-along images; and accumulating the workout scores of multiple consecutive frames of first follow-along images to obtain a first follow-along score, and displaying the first follow-along score on the follow-along interface. Therefore, the present disclosure can allow a user to clearly know a follow-along score of each individual action in an entire workout program video.

Description

Display device and device control method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese applications filed on September 27, 2023, with application number 202311270483.2; filed on December 27, 2023, with application number 202311820390.2; and filed on February 28, 2024, with application number 202410219360.4, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the technical field of display devices, and in particular to a display device and a device control method. Background Art

[0004] The display device can be installed with a fitness app offering various training programs, such as yoga, aerobics, and Tai Chi. After the user selects a program, the display device plays a training video on the training interface. The training video includes multiple standard movements, and the user adjusts their body posture to practice the corresponding movements. An image collector simultaneously captures the user's training images, and the display device calculates the degree of match between the user's training images and the standard movement images, scoring the user's training movements. This allows users to train without leaving their homes.

[0005] After the user finishes training, the display device will give the final cumulative training score. Based on the training score, the user can understand his or her comprehensive follow-up training of a whole set of movements in the training project, but cannot understand the user's follow-up training of each individual standard movement. For example, the user cannot know which specific movement is not standard, and cannot strengthen training for low-scoring movements, which affects the user's follow-up training experience.

[0006] Summary of the Invention

[0007] In a first aspect, an embodiment of the present disclosure provides a display device, which may include: an image collector, which may be configured to collect a user's follow-up training images; a display, which may be configured to display a follow-up training interface; a user input interface, which may be configured to receive instructions from the user; a memory, which may be configured to store computer instructions; at least one processor, connected to the image collector, the display, the user input interface and the memory, and may be configured to execute computer instructions to cause the display device to execute: in response to the operation of starting a training project video, obtaining an action group list associated with the training project video; wherein the training project video contains an action group segment corresponding to at least one standard action, the action group segment includes a continuous multi-frame reference action image corresponding to the standard action, and the action group list includes The system includes the action information and start and end time of the standard action; controls the display to display the training project video in the first window of the follow-up training interface, and displays the follow-up training image in the second window of the follow-up training interface; when it is detected that the training project video is played to the first action group segment corresponding to the first standard action, based on the alignment mechanism of the playback timestamp of the reference action image and the acquisition timestamp of the follow-up training image, compares the reference action image contained in the first action group segment with the aligned first follow-up training image, and calculates the training score corresponding to the first follow-up training image; accumulates the training scores corresponding to multiple frames of the first follow-up training image to obtain and store the first follow-up training score corresponding to the first standard action; controls the display to display the first follow-up training score on the follow-up training interface.

[0008] In a second aspect, an embodiment of the present disclosure further provides a device control method, which may include: in response to an operation of starting a training project video, obtaining an action group list associated with the training project video; wherein, the training project video contains an action group segment corresponding to at least one standard action, the action group segment includes a continuous multi-frame reference action image corresponding to the standard action, and the action group list includes action information and action start and end time of the standard action; displaying the training project video in a first window of a follow-up training interface, and displaying a follow-up training image captured by an image collector in a second window of the follow-up training interface; when it is detected that the training project video is played to the first action group segment corresponding to the first standard action, based on the alignment mechanism of the playback timestamp of the reference action image and the capture timestamp of the follow-up training image, comparing the reference action image contained in the first action group segment with the aligned first follow-up training image, and calculating the training score corresponding to the first follow-up training image; accumulating the training scores corresponding to multiple frames of the first follow-up training image to obtain and store the first follow-up training score corresponding to the first standard action; and displaying the first follow-up training score on the follow-up training interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG1 illustrates an operation scenario of a display device and a control apparatus according to some embodiments;

[0010] FIG2 is a block diagram of a hardware configuration of a control device according to some embodiments;

[0011] FIG3 is a block diagram of a hardware configuration of a display device according to some embodiments;

[0012] FIG4 is a diagram illustrating a software configuration of a display device according to some embodiments;

[0013] FIG5 is a schematic diagram of a fitness application homepage according to some embodiments;

[0014] FIG6 is a schematic diagram of a training item list page according to some embodiments;

[0015] FIG7 is a schematic diagram of a training mode selection page according to some embodiments;

[0016] FIG8 is a schematic diagram of a portrait recognition interface according to some embodiments;

[0017] FIG9 is a schematic diagram of a training interface according to some embodiments;

[0018] FIG10 is a schematic diagram of upper limb key points according to some embodiments;

[0019] FIG11 is a schematic diagram of a training report page according to some embodiments;

[0020] FIG12 is a schematic diagram of a dot configuration for a training program video according to some embodiments;

[0021] FIG13 is another schematic diagram of a training interface according to some embodiments;

[0022] FIG14 is another schematic diagram of a training interface according to some embodiments;

[0023] FIG15 is a flow chart of a device control method according to some embodiments;

[0024] FIG16 is a schematic diagram of a training details list page according to some embodiments;

[0025] FIG17 is another flow chart of a device control method according to some embodiments;

[0026] FIG18 is another schematic diagram of a training details list page according to some embodiments;

[0027] FIG19 is a schematic diagram of a practice playback interface according to some embodiments;

[0028] FIG20 is another schematic diagram of a practice playback interface according to some embodiments;

[0029] FIG21 is a flowchart of a display device recording a follow-up training video according to some embodiments;

[0030] FIG22 is a flowchart of a display device playing back a practice video according to some embodiments;

[0031] FIG23 is a flowchart of a display device generating a first progress bar according to some embodiments;

[0032] FIG24 is a schematic diagram of a playback interface according to some embodiments;

[0033] FIG25 is another schematic diagram of a playback interface according to some embodiments;

[0034] FIG26 is another schematic diagram of a playback interface according to some embodiments;

[0035] FIG27 is a flowchart of a display device playing a follow-up training video according to some embodiments;

[0036] FIG28 is a flowchart of a display device quickly switching to play each first action group according to some embodiments;

[0037] FIG29 is a flow chart illustrating a first set of actions that continuously play a specified score level according to some embodiments;

[0038] FIG30 is a diagram illustrating a playback interface with buttons according to some embodiments;

[0039] FIG31 is a flow chart illustrating a display device displaying a training interface according to some embodiments;

[0040] FIG32 is another schematic diagram of a training interface according to some embodiments;

[0041] FIG33 is another schematic diagram of a training interface according to some embodiments;

[0042] FIG34 is another schematic diagram of a training interface according to some embodiments;

[0043] FIG35 is a schematic diagram of a result display page according to some embodiments;

[0044] FIG36 is another flow chart of a device control method according to some embodiments;

[0045] FIG37 is a schematic diagram of obtaining the current intelligent control state according to some embodiments;

[0046] FIG38 is a schematic diagram of an image coordinate system according to some embodiments;

[0047] FIG39 is a schematic diagram illustrating the order of multiple key limb points and their meanings according to some embodiments;

[0048] FIG40 is a flowchart of determining a current target user according to some embodiments;

[0049] FIG41 is another flowchart of determining a current target user according to some embodiments;

[0050] FIG42 is another flow chart of a device control method according to some embodiments;

[0051] FIG43 is a schematic diagram illustrating left and right shoulder and left and right span key points located within a user's usage area according to some embodiments;

[0052] FIG44 is a diagram illustrating a left shoulder key point and a left hip key point located within a user usage area according to some embodiments;

[0053] FIG45 is a diagram illustrating a right shoulder key point and a right span key point located within a user usage area according to some embodiments;

[0054] FIG46 is a schematic diagram illustrating left and right span key points located within a user usage area according to some embodiments;

[0055] FIG47 is a schematic diagram illustrating the positioning of left and right shoulder key points within a user's usage area according to some embodiments;

[0056] FIG48 is another flow chart of a device control method according to some embodiments;

[0057] Figure 49 is a schematic diagram of a system architecture according to some embodiments. DETAILED DESCRIPTION

[0058] The display device of the present disclosure may have various implementation forms, for example, it may be a television, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figures 1 and 2 illustrate a specific implementation of the display device of the present disclosure.

[0059] FIG1 is a schematic diagram of an operation scenario of a display device and a control device according to some embodiments. As shown in FIG1 , a user can operate a display device 200 through a smart device 300 or a control device 100 .

[0060] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0061] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, the display device 200 can be controlled using an application running on the smart device.

[0062] In some embodiments, the display device may not use the aforementioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0063] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the display device 200 can directly receive the user's voice command control through a module for obtaining voice commands configured inside the display device 200, or it can receive the user's voice command control through a voice control device set outside the display device 200.

[0064] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0065] Figure 2 is a block diagram of a control device 100 according to some embodiments. As shown in Figure 2 , the control device 100 includes a processor 110, a communication device 130, a user input / output interface 140, a memory, and a power supply. The control device 100 receives user input commands and converts them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200.

[0066] FIG3 is a block diagram of a hardware configuration of a display device 200 according to some embodiments. As shown in FIG3 , the display device 200 may include at least one of a tuner / demodulator 210, a communication device 220, a detector 230, an external device interface 240, a processor 250, a display 260, an audio output interface 270, a memory, a power supply, and a user input interface.

[0067] In some embodiments, the processor may include a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to nth interfaces for input / output.

[0068] The display 260 includes a display screen component for presenting images, a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

[0069] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0070] Communication device 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communication device may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. Display device 200 can use communication device 220 to send and receive control signals and data signals with control device 100 or server 400.

[0071] The user input interface can be used to receive control signals from the control device 100 (such as an infrared remote controller, etc.).

[0072] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0073] The external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0074] The tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals and EPG data signals from a plurality of wireless or wired broadcast television signals.

[0075] In some embodiments, the processor 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the processor 250 is located, such as an external set-top box.

[0076] Processor 250 can control the operation of the display device and respond to user operations through various software control programs stored in the memory. Processor 250 can control the overall operation of display device 200. For example, in response to receiving a user command to select a UI object to be displayed on display 260, processor 250 can perform operations related to the object selected by the user command.

[0077] In some embodiments, the processor 250 may include at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), random access memory (RAM), read-only memory (ROM), a first interface to an nth interface for input / output, a communication bus, etc.

[0078] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific voice or gesture, and the user input interface may recognize the voice or gesture through a sensor to receive the user input command.

[0079] A user interface is the medium for interaction and information exchange between an application or operating system and the user. It converts information between its internal form and a user-friendly format. A common user interface is the graphical user interface (GUI), which refers to a graphical user interface related to computer operations. It can be an icon, window, control, or other interface element displayed on an electronic device's display. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0080] FIG4 is a diagram illustrating the software configuration of the display device 200 according to some embodiments. As shown in FIG4 , in some embodiments, the system is divided into four layers, namely, from top to bottom, the application layer (hereinafter referred to as the "application layer"), the application framework layer (hereinafter referred to as the "framework layer"), the Android runtime and system library layer (hereinafter referred to as the "system runtime library layer"), and the kernel layer.

[0081] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0082] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0083] As shown in Figure 4, in some embodiments of the present disclosure, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0084] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and backing of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, shaking the display, distorting the display, etc.).

[0085] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0086] In some embodiments, the kernel layer is a layer between hardware and software. As shown in FIG4 , the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a Wi-Fi driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power driver.

[0087] For display devices with the same or similar software and hardware configurations as above, see Figure 4. The application layer can install fitness applications. Fitness applications provide a variety of different training programs. Users can choose the training programs they are interested in and achieve training without leaving home through AI interaction with the display device.

[0088] Figure 5 is a schematic diagram of a fitness application homepage according to some embodiments. When a user opens a fitness application (e.g., triggers the application icon), the display device may display the fitness application homepage. Referring to Figure 5 , the fitness application homepage includes a selection control 51 for various training programs, including but not limited to aerobics, yoga, square dancing, parent-child exercise, Tai Chi, weight loss exercises, martial arts, etc.

[0089] In some embodiments, a user can register and log in to an account in a fitness application, so that the fitness application can record the user's historical training data under the account. Referring to Figure 5, the first area (for example, the left area) on the homepage of the fitness application displays user information, energy consumption, training duration, last training item and other information, where the user information includes but is not limited to the user name and user avatar, etc., so that the user can easily understand his or her recent training dynamics.

[0090] FIG6 is a schematic diagram of a training item list page according to some embodiments. A user can select any training item A of interest in the fitness application homepage and click the option control 51 corresponding to the training item A. In response to this operation, the display device can control the display to display the training item list page. Referring to FIG6 , the training item list page includes subordinate training items of training item A. For example, if training item A is aerobics, subordinate training items of aerobics include but are not limited to HIIT (High-Intensity Interval Training), Pamela, etc. In this way, users can more finely select the training items of interest based on their preferences and abilities, such as selecting aerobics-HIIT.

[0091] In some embodiments, the user can operate the direction keys on the control device (such as a remote control), move the selector 61, and select the training item E in the training item list page in the selector 61, see Figure 6, the display device can control the display to display relevant information of the training item E in a preset area of ​​the training item list page (such as the upper part of the page), including but not limited to: training item name, theoretical calorie consumption, training part, difficulty level (such as entry-level, easy, difficult, high difficulty, etc.), training intensity level (such as beginner, intermediate, advanced, etc.), training duration, training item introduction, etc., so that the user can decide whether to select training item E for follow-up training.

[0092] In some embodiments, operation buttons may also be set in the preset area of ​​the training item list page, including but not limited to a start training button, a like button, a favorite button, a share button, a fitness VIP activation button, etc.

[0093] Figure 7 is a schematic diagram of a training mode selection page according to some embodiments. In response to a user clicking a Start Training button associated with a training item E, the display device may control the display to display the training mode selection page. Referring to Figure 7 , the training mode selection page includes training modes provided by the fitness application, including but not limited to a follow-up training mode, a normal mode, and a movie viewing mode.

[0094] In some embodiments, the follow-up training mode means that the operator pre-marks the action key frames for the training project video. When the training project video is played to the key frame, the user refers to the standard action shown in the key frame, adjusts his or her body posture to perform the follow-up training action, and the image collector synchronously collects the user's follow-up training image. The fitness application scores according to the degree of match between the standard action and the follow-up training action, and records training data such as training time and calorie consumption.

[0095] In some embodiments, the normal mode refers to only playing the training project video for the user to watch, and does not support follow-up training functions such as key frame marking, training scoring, and recording training data, and the image collector does not collect follow-up training images. The viewing mode is to display two independent windows in the application interface. Window one is used to display the training project video (follow-up training is not supported, and the image collector does not collect follow-up training images), and window two displays other media content (such as video resources, applications, etc.), allowing users to watch other media content while freely training with reference to the training project video. The present disclosure is mainly aimed at the above-mentioned follow-up training mode and does not limit other training modes.

[0096] In some embodiments, the display device can detect whether the image collector is available in response to the user selecting the follow-up training mode. The image collector can be pre-installed in the detector 230 of the display device, or the display device can be connected to an external image collector through an interface (such as USB). The image collector is, for example, a pan-tilt camera. There are three situations in which the image collector is unavailable: the first is that the physical switch of the image collector is in the off state, in which case the user can be prompted to turn on the image collector through the UI or voice; the second is that the image collector is occupied by other processes, such as a video call application is occupying the image collector, resulting in the inability to call the fitness application, in which case the image collector can be prompted to be occupied, and the user can release the image collector, such as exiting the video call application, or switching to normal mode or viewing mode; the third is that the display device is not configured with an image collector, in which case the user can be prompted that the image collector is not detected, so that the user can switch to normal mode or viewing mode, or reconfigure and verify the image collector.

[0097] In some embodiments, a terminal device can capture a user's training image and simultaneously transmit the image to a display device, allowing the display device to complete the training process without the need for an external image capture device. Terminal devices include, but are not limited to, smartphones and smart tablets.

[0098] FIG8 is a schematic diagram of a portrait recognition interface according to some embodiments. The display device can respond to the user selecting the follow-up training mode and detect that the image collector is available, then control the display to display the portrait recognition interface. If no real portrait is captured, see FIG8 . The portrait recognition interface can display a virtual human image with an aperture area under the virtual human foot. The aperture area is the effective recognition area of ​​the image collector. The image collector can be configured as an adjustable device such as a pan / tilt to adjust the camera's shooting position and angle. When the camera moves, the aperture moves accordingly, and eventually the aperture area is adjusted to the center of the screen, and a guidance message is displayed. The guidance message, for example, is "Please keep standing alone in the circle." In this way, the user only needs to move the body into the aperture area, so that the image collector can successfully recognize the portrait and then capture the local video. The local video includes the follow-up training images captured when the user performs the follow-up training movements with reference to the standard movements.

[0099] In some embodiments, as the user moves, the image collector and aperture also move accordingly. That is, the image collector tracks the body. If it detects that the user's position is far from the center of the screen (for example, at the edge of the screen), the display device can prompt the user to stand in the center of the screen. After successful portrait recognition, the training video will start after a few seconds of countdown, and the fitness app will enter the training process of follow-up mode.

[0100] FIG9 is a schematic diagram of a training interface according to some embodiments. Referring to FIG9 , the training interface includes a first window 91 and a second window 92 , wherein the first window 91 is used to display a training program video, and the second window 92 is used to display a user's training image.

[0101] In some embodiments, referring to FIG9 , each time a key frame of a training project is played, relevant information about the standard action contained in the key frame can be displayed in the first window 91. For example, the first window 91 shown in FIG9 is a jumping jack project. When implemented, the action name, action duration and other information can be displayed in the first window 91. The second window 92 can display information related to the user's training, such as words of encouragement (Good, Great, Perfect, etc.), action standardization prompts (such as paying attention to straighten your elbows as much as possible and clamp them on both sides of the head), etc. In addition, fitness applications can also count data such as the training time, accuracy, energy consumption, etc., referring to FIG9 , and display these statistical data in a preset position (such as the top) of the training interface.

[0102] FIG10 is a schematic diagram of upper limb key points according to some embodiments. For each standard movement, the fitness application can invoke an image recognition algorithm configured to match the positions of limb key points between the standard movement and the practiced movement. Referring to FIG10 , limb key points include, but are not limited to, facial key points such as the nose, left eye, eyes, left ear, and right ear, as well as torso key points such as the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left knee, and right knee. By matching limb key points between the standard movement and the practiced movement, it is possible to determine which parts of the movement are non-standard and the degree of standardization, thereby providing movement standardization prompts and appropriate encouragement. The user's practiced movement can also be scored, where accuracy and energy consumption are comprehensive evaluation indicators of the user's practiced movement after the practice time has elapsed. For example, if a user has practiced for 25 minutes after the training program video starts, some of the practiced movements may have higher scores and some may have lower scores. Overall, the accuracy rate is 65%, and the total energy consumption in the 25-minute practice is 300 kcal.

[0103] Figure 11 is a schematic diagram of a training report page according to some embodiments. At the end of a training session video, the display device can control the display to display the training report page. Referring to Figure 11 , the training report page displays the training data for the current session, a button for another session, and a button for continuing to the next session. The training data for the current session is a comprehensive summary of the training data from the entire training session up to the time the training mode is exited. This training data includes, but is not limited to, the final training score, total energy consumption, accuracy, and total training duration.

[0104] Through the training report page, users can only understand their comprehensive practice status of a whole set of movements in the training program, but cannot understand the practice status of any single movement. For example, users cannot know which specific movement is not standard (low score), and cannot strengthen training for low-scoring movements. They also cannot know the points where points are lost when practicing each movement, which affects the user's practice experience.

[0105] In order to solve the above-mentioned related problems, the embodiments of the present disclosure provide a display device and a device control method, which can realize the display of the follow-up training score of any individual action during and after the follow-up training, so that the user can know which action has a higher score and which action has a lower score, so that the user can strengthen the training of the low-scoring actions, thereby improving the user's training enthusiasm and personal ability.

[0106] FIG12 is a schematic diagram of a dot configuration for a training project video according to some embodiments. Referring to FIG12 , an operator can perform a dot configuration according to M (M≥1) standard actions contained in any training project video. i And a single standard action iThe training duration is 100 seconds, and the training project video is divided into M action group segments ActionGroup i , i represents the sequence number of the standard action in the training project video, 1≤i≤M, M action group segments ActionGroup i The action group segments are connected in series according to the playback sequence. i Including standard actions i The corresponding continuous N (N is greater than 1) frames of reference action images may have different training durations and the number of frames included in the action group segment may be different. Assume that each frame of the reference action image P j The reference action contained in Action ij , where j represents the frame number of the reference action image, 1≤j≤N, then any standard action i The corresponding action group is {Action ij |1≤i≤M,1≤j≤N}.

[0107] In some embodiments, see FIG12 , the operator may also configure an action group list associated with the training project video, the action group list including M standard actions i Corresponding action group information ActionGroup_Info i An example of ActionGroup_Info is as follows:

[0108] In some embodiments, ActionGroup_Info i Including but not limited to Action i _config_time, Action i _start_time, Action i _end_time, Action i _id and Action i _type, etc. Action i _config_time indicates a standard action i Action configuration time; Action i _start_time indicates a standard action i Action start time; Action i _end_time indicates a standard action i Action end time; i _id represents the standard action i Action i_type indicates Action i The categories of movements in the overall training program video include warm-up, movement, stretching, and release.

[0109] In some embodiments, when you open an ActionGroup from the previous ActionGroup i-1 Switch to the current action group fragment ActionGroup i Due to the continuity of the video recording, the action demonstrator cannot i-1 Immediately mutate to Action i , often requires action demonstration personnel to end the action i-1 After that, release the limbs and gradually transition to Action i , which generates Action i _config_time, Action i _config_time is equivalent to ActionGroup i The starting point, from Action i _config_time The action demonstrator begins to adjust his or her body movements (abbreviated as: action configuration) until the body movements are adjusted to the standard action. i Consistent, at this time the time node transitions to Action i _start_time, that is, from Action i _start_time starts the standard action i The display; in the action teacher ended the action i During training, the time node reaches Action i _end_time, Action i _end_time is equivalent to ActionGroup i Therefore, the display device can obtain the M action group segments ActionGroup contained in the training project video according to the action group list. i .

[0110] In some embodiments, referring to FIG12 , the operator may also configure an action template file, which includes each standard action i Corresponding action template data Action i _template,Action i _template includes standard actions i The position coordinates, action information, and benchmark scores of the key points of the limbsi _standard_score, etc. Among them, the action information includes but is not limited to Action i _name (action name, such as scissors step), Action i _id (action identifier) ​​and Action i _type (action classification), etc. An exemplary action template data is as follows:

[0111] "Action i ":[{"Action i _name":"xxx",

[0112] "Action i _id":"0",

[0113] "Action i _type":1,

[0114] "Action i _standard_score":7.0,

[0115] "Action i _weight":[1,2,3,4,5],

[0116] "data":[

[0117] {"bbox":[832.0,188.0,1095.0,1009.0],

[0118] "keypoints":

[0119] [[961,250],[980,251],

[0120] [944,255],[1004,269],

[0121] [926,271],[1050,352],

[0122] [881,363],[1071,495],

[0123] [863,495],[1080,614],

[0124] [842,598],[998,601],

[0125] [913,604],[1022,781],

[0126] [894,770],[1037,925],

[0127] [881,935]]}

[0128] …]

[0129] Among them, Action i _standard_score indicates a standard action i The benchmark score of Action is different. Due to the different difficulty levels of the actions, different standard actions can correspond to different benchmark scores. i _score is used for standard actions i The scoring provides a reference standard; Action i _weight indicates the weight of the action; bbox is the action group fragment ActionGroup i The rectangular box used by the action demonstrator to calibrate the object, bbox is used to indicate the position and boundary of the action demonstrator, (832.0,188.0) is the coordinate of the upper left corner of the rectangular box, (1095.0,1009.0) is the coordinate of the lower right corner of the rectangular box; keypoints indicates the standard action Action i Limb key point position data, referring to FIG10 , keypoints includes the position coordinates of each limb key point.

[0130] In this way, the display device can query the limb key point position data (abbreviated as: first limb key point position data) configured for any standard action Actioni through action information, such as Actioni_id, and then compare it with the limb key point position data of the user's follow-up action (abbreviated as: second limb key point position data) to obtain the action matching degree between the standard action and the follow-up action. The action matching degree is used to characterize the similarity between the two actions. The higher the action matching degree, the more standard the user's follow-up action is, and the higher the score for the follow-up action is.

[0131] In some embodiments, the fitness application can communicate with a server. When a user selects a training program, the fitness application can send a service request to the server, which includes the identification information of the training program. In response to the service request, the server can return the video data of the training program, its associated action group list, and action template data to the fitness application.

[0132] In some embodiments, after receiving the video data, action group list and action template data of the training project, the fitness application can parse the action group list to obtain the M standard actions included in the training project video. i Action information (e.g. action name) and M action group fragments ActionGroup i, and control the display to display the practice interface.

[0133] Figure 13 is another schematic diagram of a follow-up training interface according to some embodiments. Referring to Figure 13 , the follow-up training interface may include a first window 131, a second window 132, a first progress bar 133, and a third window 134. The layout of the follow-up training interface is not limited. Referring to Figure 13 , an exemplary layout may be as follows: first window 131 is located in the upper left portion, second window 132 is located in the upper right portion and is parallel to first window 131, first progress bar 133 is located at the bottom of first window 131 and second window 132, and third window 134 is located below first progress bar 133.

[0134] In some embodiments, the first window 131 is used to display a training program video. The first window 131 may also display the current exercise name (e.g., jumping jacks), exercise instructions, playback progress information (e.g., 08:20), video duration (e.g., 30:00), a pause control 131a, a previous item control 131b, and a next item control 131c. The user can pause the training program video by triggering the pause control 131a, and can also trigger the previous item control 131b or the next item control 131c to switch training program videos.

[0135] In some embodiments, second window 132 is used to display the training images captured by the image collector in real time, or to display the training images captured and pushed by the terminal device. Second window 132 may also display the energy consumption of the current exercise group (such as 44 kcal as shown in Figure 13), the cumulative training score (such as 48 points as shown in Figure 13), encouragement, movement standardization prompts, etc., so that the user can understand the training status of each exercise group during training.

[0136] In some embodiments, the third window 134 is used to display the M standard actions included in the training program video. i Corresponding action prompt control 134a. Referring to FIG13, the action prompt control 134a is used to prompt the standard action Action i action thumbnails, action information, training status and follow-up scores, etc.

[0137] In some embodiments, fitness applications can be derived from the action group fragment ActionGroup i Get the action that best reflects the standard action i A frame of reference action image P of body posture i , and the reference action image P i Perform zoom processing to get the action thumbnail. Fitness applications can use standard action i After the configuration is completed, in [Action i_start_time,Action i _end_time) range to extract the reference action image P i .

[0138] In some embodiments, the action information displayed by the action prompt control 134a includes at least the action name, such as "scissor step". In this way, the user can intuitively understand each standard action by browsing the action thumbnail and action name. i body posture.

[0139] In some embodiments, the training status may include a trained state, a training in progress state, a training to be trained state, and a training paused state.

[0140] In some embodiments, the practiced state indicates that the standard action has been trained and the action group segment corresponding to the standard action in the practiced state has finished playing, that is, media_progress (playback progress) ≥ Action i _end_time, fitness apps can continue playing Action i+1 The corresponding action group fragment. If the next standard action is not detected i+1 , that is, i=M, indicating that the training project video has ended.

[0141] In some embodiments, the "Practicing" state indicates that the user is currently practicing the standard action. The first window 131 currently plays the action group clip corresponding to the standard action in the "Practicing" state. There may be two situations: the first one is that the action i _config_time≤media_progress<Action i _start_time, which is in the configuration stage of the current action group; the second type, Action i _start_time≤media_progress<Action i _end_time, that is, the action training stage of the current action group.

[0142] In some embodiments, the waiting-for-training state indicates waiting for future training, and the action group segment corresponding to the standard action in the waiting-for-training state has not yet been played, that is, media_progress<Action i _config_time.

[0143] In some embodiments, the fitness application can query the standard movements that have been practiced (hereinafter referred to as the target standard movements), obtain the action prompt control 134a (hereinafter referred to as the target action prompt control) corresponding to the target standard movement from the third window 134, and control the target action prompt control to display the action group practice score corresponding to the target standard movement. Referring to the example of FIG13 , the target standard movements include scissors step and cross lunge. The target action prompt control corresponding to scissors step displays a practice score of 35 points, and the target action prompt control corresponding to cross lunge displays a practice score of 85 points. In this way, the user can clearly understand the score of the previously practiced movements.

[0144] In some embodiments, the action prompt control 134a can be configured with interactive and trigger functions. See Figure 13. For example, if a user is currently practicing a jumping jack and sees that the score for practicing the scissors step is only 35 points, knowing that the training quality of the scissors step is low, the user can click the action prompt control 134a corresponding to the scissors step. In response to the user clicking the action prompt control 134a corresponding to the scissors step, the display device can pause the practice of the jumping jack action (i.e., switch to a paused practice state), record the current practice score of the jumping jack action group and the progress of the playing of the jumping jack action group segment (media_progress_kht), and according to the action configuration time (Action1_config_time) of the scissors step, jump the playback progress of the training project video to Action1_config_time, so that the user can practice the scissors step action again.

[0145] FIG14 is another schematic diagram of a follow-up training interface according to some embodiments. After a user clicks the action prompt control 134a corresponding to the scissors step, if the playback progress of the training project video reaches the action end time (Action1_end_time) of the scissors step, referring to FIG14 , the display device can update the follow-up training score displayed in the action prompt control 134a corresponding to the scissors step, for example, to 70 points, and display a first prompt pop-up window 135 on the follow-up training interface. The first prompt pop-up window 135 includes a first prompt message, a sequential training button, and a resume jumping jack training button. The first prompt message is used to prompt that the action score of the scissors step has been updated and to ask the user whether to continue the jumping jack training.

[0146] In some embodiments, if the user does not want to resume jumping jack training, he or she may click the sequential training button. The display device may control the first window 131 to continue playing the action group clip of the next action "squat" of the scissors step in response to the user clicking the sequential training button 135, so that the user can practice the squat action sequentially.

[0147] In some embodiments, if the user wants to resume jumping jack training, he can click the resume jumping jack training button. The display device can respond to the user's operation of clicking the resume jumping jack training button, and according to the media_progress_kht of the previously recorded jumping jack action group segment, automatically position the overall playback progress of the training project video to the media_progress_kht, so that the first window 131 continues to play the jumping jack action group segment, and continues the jumping jack follow-up process and scoring mechanism based on the previously recorded jumping jack action group follow-up score, thereby realizing the resumption of jumping jack follow-up practice.

[0148] In some embodiments, the user can click on the action prompt control 134a corresponding to any standard action in the third window 134 according to training interests, for example, click on the action prompt control 134a marked as having been practiced to refresh the score; for example, the user can click on the action prompt control 134a marked as to be practiced to skip the action group segment corresponding to the standard action that he does not want to practice, or practice the standard action that is ranked lower in advance.

[0149] FIG15 is a flow chart of a device control method according to some embodiments. Referring to FIG15 , the method includes the following steps:

[0150] Step S151: In response to the operation of starting a training project video, a list of action groups associated with the training project video is obtained.

[0151] In some embodiments, when the display device receives an operation from the user to start a training project video, it can send a service request to the server; the server responds to the service request and sends the training project video data, the associated action group list and the action template data to the display device.

[0152] Step S152: Control the display to display the training item video in the first window of the training interface, and to display the training image in the second window of the training interface.

[0153] The practice interface can refer to the examples of Figure 9 or Figure 13, and the practice interface can also be configured with other layouts.

[0154] Step S153: When it is detected that the training project video is played to the first action group segment corresponding to the first standard action, the reference action image contained in the first action group segment is compared with the aligned first follow-up training image, and the training score corresponding to the first follow-up training image is calculated.

[0155] Step S154: Accumulate the training scores corresponding to the multiple frames of the first follow-up training images to obtain a first follow-up training score corresponding to the first standard action.

[0156] In some embodiments, the action group list has provided each standard action Action iAction i _config_time, Action i _start_time and Action i _end_time (i.e., the configured dotted time). The display device can monitor the playback progress (media_progress) of the training project video. By traversing the action group list and comparing media_progress with the dotted time, it can determine which action group segment the training project video is playing to, and then determine the current standard action to be practiced. For ease of understanding, the action group segment corresponding to any standard action currently playing on the display device will be referred to as the first action group segment, and the standard action currently being practiced by the user will be referred to as the first standard action.

[0157] In some embodiments, if Action i _config_time≤media_progress<Action i _start_time, indicating that the training project video plays to the action group segment ActionGroup i In the action configuration phase; if Action i _start_time≤media_progress<Action i _end_time, indicating that the training project video plays to the action group segment ActionGroup i Action training phase; if media_progress≥Action i _end_time, indicating that the action group segment has ended playing ActionGroup i .

[0158] In some embodiments, the key to step S153 is to detect any action group segment (hereinafter referred to as the first action group segment) played in the training program video based on media_progress and the marking time. There are two implementation methods:

[0159] Method 1: Set media_progress=Action i The scoring mechanism is triggered at _config_time.

[0160] Method 2: Set media_progress=Action i The scoring mechanism is triggered at _config_time. Since the action group clips generally need to transition from the action configuration stage to the action training stage, during the action configuration stage, the action demonstration personnel have not yet fully performed the correct standard action. i, which cannot provide correct training guidance for the trainees, but in Action i _config_time The action demonstration personnel will perform the standard action i , so you can set media_progress=Action i The scoring mechanism is triggered at _config_time to improve the accuracy and reliability of the scoring results.

[0161] In some embodiments, the scoring mechanism can be configured as follows: based on the alignment mechanism of the playback timestamp of the reference action image and the acquisition timestamp of the training image, obtain the reference action image P in the first action group segment. 1k Time-aligned follow-up image (abbreviated as: first follow-up image P 2k ), by comparing the reference action image P 1k and the first follow-up image P 2k Calculate the matching degree between the standard action and the user's synchronized training action, and then give the training score Score ik , where k represents the frame number of the reference action image and the first training image, i represents the number of the standard action in the training project video, Score ik Indicates that the user is practicing standard actions i The first training image P 2k The corresponding training score.

[0162] In some embodiments, the server-side operator can configure a playback timestamp for the training project video. For example, the playback timestamp when the training project video starts is 0, and the timestamp when the training project video ends is t, where t represents the total playback time of the training project video. When the first window 131 starts playing the training project video, the display device can control the image collector to synchronously capture the user's follow-up training images and configure a capture timestamp for each frame of the follow-up training image. Therefore, the display device can extract the reference action image P from the video code stream captured by the image collector based on the alignment mechanism of the playback timestamp and the capture timestamp. 1k Aligned first training image P 2k , so the first training image P 2k The user refers to the reference action image P 1k The training images collected by the image collector when performing the training movements are aligned with the timestamp to ensure the accuracy of the scoring mechanism.

[0163] In some embodiments, the display device can perform standard actions according to the i Action information (such as Action i _name、Action i_id), get the standard action Action from the action template data i Benchmark score Action i _standard_score, and get standard action i The display device can display the first training image P 2k Input the pre-trained human body recognition model and obtain keypoints2 (second limb key point position data) output by the human body recognition model.

[0164] In some embodiments, the display device can calculate the action matching fitness based on keypoints1 and keypoints2. k , 0≤fitness k ≤1, and based on the action matching fitness k and benchmark score Action i _standard_score, calculate the training score Score ik , Score ik =fitness k *Action i _score. Among them, due to the different difficulty levels of the actions, different standard actions i , can have different benchmark scores Action i _standard_score.

[0165] In some embodiments, the scoring mechanism can also be configured to: calculate the training score Score corresponding to the first training image of multiple frames (Q frames, 1<Q≤N) ik Accumulate and get the standard action i Corresponding action group follow-up score ActionGroup_score i , i.e. ActionGroup_score i =SUM{Score ik ,k=1,2…,Q}. Action group training score ActionGroup_score i Used to represent the user's standard action i The comprehensive training situation of the action group.

[0166] In some embodiments, the scoring mechanism can also be configured to periodically obtain reference action images P from the first action group segment according to a preset period. 1k, and periodically obtain the first follow-up image P from the video code stream collected by the image collector or the video code stream collected and sent by the terminal 2k Since the user's body movements will not change significantly when the time interval is short, such as only a slight shake, in order to reduce the computational complexity of the scoring mechanism, the display device can obtain a frame of reference action image P with aligned timestamps at every preset interval. 1k and a frame of the first training image P 2k , so as to periodically calculate the training score Score ik , reducing the number of training score calculations. The scoring mechanism provided by the embodiment of the present disclosure can be preset in the follow-up scoring model or module.

[0167] Step S155 , storing the first follow-up practice score, and controlling the display to display the first follow-up practice score on the follow-up practice interface.

[0168] In some embodiments, see FIG13, for the standard action in the following training state, i (For example, jumping jack Action3), the display can display the corresponding action group follow-up score ActionGroup_score in the second window 132 of the follow-up interface i , due to ActionGroup_score i It is dynamically accumulated (e.g., periodically accumulated). The action group practice score ActionGroup_score displayed in the second window 132 is i It is updated dynamically.

[0169] In some embodiments, referring to FIG13 , for standard actions that have been practiced (e.g., Action 1 and Action 2), the practice score for the action group can be displayed via the action prompt control 134a. If the action group has not been practiced again, the practice score displayed by the action prompt control 134a remains unchanged. For standard actions that are in the pending practice state (e.g., Action 4), since they have not yet been practiced and scored, the practice interface does not display the corresponding practice score for the action group.

[0170] 13 , the training interface further includes an exit training button 136. The display device may respond to the user clicking the exit training button 136 or detecting that the training program video has reached the end, both of which indicate the end of the training program video.

[0171] FIG16 is a schematic diagram of a training details list page according to some embodiments. When the training program video ends, the display device can control the display to display the training details list page 160. Referring to FIG16 , the training details list page 160 includes M separate standard actions. iThe training details include but are not limited to action information (such as action name), exercise duration, energy consumption and action group training score. Taking jumping jacks as an example, when a user is practicing jumping jacks, the exercise duration is 145 seconds, the energy consumption is 74 kcal, and the score is 90 points. The display device can follow the standard action i Sort the M pieces of training details in the training details list page by their order of appearance in the training project video.

[0172] In some embodiments, see FIG16 , the training details list page 160 also includes each standard action Action i The corresponding retraining control 161. The training details information can be set in the same row as the retraining control 161, with each row corresponding to a standard action Action i After browsing the training details list page, the user can learn the training score of each standard action group and click the retraining control 161 corresponding to the low-scoring action (such as sumo squat) to retrain and refresh the score.

[0173] In some embodiments, the display device may control the display to close the follow-up details list page in response to the user clicking the re-training control 161 corresponding to the first standard action, and display the action group clip corresponding to the first standard action (referred to as: first action group clip) in the first window 131 of the follow-up interface, and the second window 132 displays the follow-up image collected when the user re-trains the first standard action.

[0174] In some embodiments, when the first action group segment ends, i.e., when the user completes the first standard action after retraining, the processor re-acquires the final accumulated first practice score in the aforementioned manner, updates the locally recorded first practice score, controls the display to display the practice details page, and updates the first practice score on the practice details page. For example, if the original action group practice score for the sumo squat is 2 points, and after the user retrains, the sumo squat action group practice score increases to 55 points, the locally recorded sumo squat action group practice score is updated, as well as the sumo squat action group practice score displayed on the practice details page, from the original 2 points to the latest 55 points.

[0175] In some embodiments, although the user can understand the action group practice score of each standard action through the third window 134 of the practice interface and the practice details list page, and know which actions have higher scores and which actions have lower scores, the user cannot understand where the points of loss are for each low-scoring action, and cannot know which specific limb movement is inaccurate and affects the score, and thus cannot strengthen and improve the training in a targeted manner. In response to this, the present disclosure records a video clip of the user practicing each standard action through screen recording. In this way, by playing back the screen recording file, the user can compare the difference between the standard action presented in the first window 131 and the practice action presented in the second window 132, and find out the points of loss during the practice process in combination with the action standardization prompt information and other content displayed in the second window 132, so that the user can correct his or her own wrong actions in a targeted manner when re-practicing.

[0176] FIG17 is another flow chart of a device control method according to some embodiments. Referring to FIG17 , the method includes the following steps:

[0177] Step S171 , in response to the operation of starting the training project video, obtaining a list of action groups associated with the training project video.

[0178] Step S172: Control the display to display the training item video in the first window of the training interface, and to display the training image in the second window of the training interface.

[0179] Step S173: When it is detected that the training program video is played to the first action group segment corresponding to the first standard action, the screen recording function is activated to start recording the screen content.

[0180] In some embodiments, the display device can use a screen recording API (such as MediaProjection) to capture screen content and generate a video of the screen content. When recording the screen, the screen recording API can obtain the screen resolution (such as 1920*1080), and based on the screen resolution, call the mMediaRecorder.setVideoSize(width,height) interface to set the size of the recorded video; set relevant recording parameters, such as the clarity, format, frame rate, image quality, bit rate, etc. of the recorded video; set the file name and preset storage location of the screen recording file. The naming rules are not limited. For example, the naming rules are configured as the training project video name + action name + start recording timestamp to avoid repeated recording of action group segments in the training project video. The preset storage location is the save location of the screen recording file (such as folder b in disk A). In this way, the screen recording API records the screen content according to the screen resolution and recording parameters, and at the end of recording, sets the file name of the screen recording file according to the preset naming rules, and saves the screen recording file to the preset storage location.

[0181] In some embodiments, referring to FIG. 13 , after the screen recording function is started, the display device may control the display to display a screen recording prompt control 137 on the training interface. The screen recording prompt control 137 is used to prompt the user that the display device is recording the screen.

[0182] Step S174 , comparing the reference action image included in the first action group segment with the aligned first follow-up training image, and calculating a training score corresponding to the first follow-up training image.

[0183] Step S175 , accumulating the training scores corresponding to the multiple frames of the first follow-up training images to obtain a first follow-up training score corresponding to the first standard action.

[0184] Step S176: store the first follow-up practice score, and control the display to display the first follow-up practice score on the follow-up practice interface.

[0185] Step S177: When the first action group segment ends playing, stop recording the screen content.

[0186] In order to record the video clip of the user practicing a single standard action, when the action group clip ends, media_progress = Action i When _end_time comes, the screen recording stops, thereby generating a screen recording file corresponding to each action group segment, that is, recording is performed in segments according to the action group segments, so that the user can review his or her own practice process when playing any action group segment first.

[0187] Step S178: storing the screen recording file of the first action group segment and the first action group recording information in a preset storage location.

[0188] In some embodiments, an exemplary action group recording information is as follows:

[0189] record_ActionGroup i _info{

[0190] Action i _name, / / Standard action i Action name

[0191] record_ActionGroup i _start_time, / / Follow standard action i The action group recording start time

[0192] record_ActionGroup i _end_time, / / Follow standard action i The action group recording end time

[0193] ActionGroup_score i , / / Action group follow-up score

[0194] Action i _standard_score, / / standard score of the action group (related to difficulty)

[0195] …

[0196] }

[0197] Action group recording information record_ActionGroup i _info can include: Standard Action Action i Action information (such as action name Action i _name), action group practice score (ActionGroup_score i ), action group standard score (Action i _standard_score), action group recording start time (record_ActionGroup i _start_time) and the action group recording end time (record_ActionGroup i _end_time).

[0198] In some embodiments, under normal circumstances record_ActionGroup i _start_time=Action i _config_time,record_ActionGroup i _end_time=Action i _end_time.

[0199] In some embodiments, referring to Figure 13, the user may click the pause control 131a in the first window 131. In response to the user clicking the pause control 131a, the processor controls the first window 131 to pause playing the training project video and pause the follow-up mode, including pausing the statistics of training duration, energy consumption, action group follow-up scores and other information to avoid ineffective follow-up training for the user, until the user clicks the pause control 131a again, and the display device continues to play the training project video and resumes the follow-up mode.

[0200] In some embodiments, if the user's body moves significantly, or the user temporarily walks out of the shooting area of ​​the image collector for some reason, the portrait cannot be detected in the follow-up training image. The display device can control the first window 131 to pause playing the training project video and suspend the follow-up training mode at the same time to avoid invalid follow-up training for the user, until the user re-enters the aperture and completes the portrait recognition verification, and then continues to play the training project video and resumes the follow-up training mode.

[0201] In some embodiments, if the image collector is occupied by other applications (such as video call applications) in the follow-up training mode, or the image collector fails, or the physical switch of the image collector is mistakenly turned off, these situations will cause the image collector to become unavailable. The display device can control the first window 131 to pause playing the training project video and suspend the follow-up training mode at the same time to avoid invalid follow-up training for the user, until the image collector becomes available and the portrait recognition is verified, and then continue to play the training project video and resume the follow-up training mode.

[0202] However, after the display device starts the screen recording function, during the process of recording the screen content by the screen recording API, once the training model is paused, the statistics of training time, energy consumption, action group training score and other information will be paused, and media_progress will also stay at the pause node and will not change. As long as media_progress does not reach the Action i _end_time, the screen recording process will continue, that is, the screen recording time includes the pause time of the follow-up mode. i _config_time, and Action i _end_time directly marks the start and end time of the action group recording, which obviously ignores the extra time consumed when the follow-up mode is paused, resulting in the video length of the recorded file being inconsistent with the marked time.

[0203] In order to solve the problem of inconsistency between the recorded video duration and the marked duration caused by the pause of the follow-up mode, in some embodiments, when the training project video is played in the first window 131 of the follow-up interface, the display device starts a timer and detects the current playing content during the timing. When it is detected that the training project video is played to the first action group segment ActionGroup i , that is, media_progress = Action i _config_time, read the timer's current elapsed time (hereinafter referred to as: first time), and set the first time to record_ActionGroup i _start_time. The timer will continue to count after that and will not stop when the follow-up mode is paused.

[0204] In some embodiments, the display device can be in the first action group fragment ActionGroup i When playback ends, media_progress = Action i _end_time, read the timer elapsed at this time (hereinafter referred to as: second time), and set the second time to record_ActionGroup i _end_time. Since the timer continues to count and will not stop due to the pause of the follow-up mode, if any of the above-mentioned follow-up mode pauses occurs, the time recorded by the timer includes the time of the follow-up mode pause, so the second time is used as record_ActionGroup i _end_time, the start and end time of the action group recording marked by this will match the actual recording process, making the start and end time of the action group recording more accurate.

[0205] FIG18 is another schematic diagram of a training details list page according to some embodiments. At the end of the training video playback, according to the above recording process, M action group segments ActionGroup have been recorded in sequence. i Video recording file Record i , see Figure 18, the training details page also includes each standard action i The corresponding playback control 181, follow-up details information, retraining control 161 and playback control 181 can be set in the same row, with each row corresponding to a standard action i correspond.

[0206] In some embodiments, the i-th playback control 181 can be associated with the video recording file Record i Create an associated link so that users can review the previous standard action i During the practice, you can click on the standard action i The corresponding i-th playback control 181.

[0207] FIG19 is a schematic diagram of a follow-up playback interface according to some embodiments. In response to the user clicking the i-th playback control 181, the display device can obtain the recording segment Record linked to the i-th playback control 181 from a preset storage location. i , and control the display to display the practice playback interface, see Figure 19, the practice playback interface plays the video recording file Record i The follow-up playback interface also includes but is not limited to a second progress bar 191, recording duration information 192, a pause control 193, a previous control 194, a next control 195 and a score prompt information 196.

[0208] In some embodiments, the second progress bar 191 is a video recording file Record i The video playback progress bar; wherein the recording time information 192 can be determined according to the start and end time of the action group recording. For example, the recording time T record =record_ActionGroup i _end_time-record_ActionGroup i _start_time; users can pause the playback of the video recording file Record by using the pause control 193 i When the user clicks the previous control 194, the display device can control the display to display the video recording file Record on the follow-up playback interface. i-1 , and refresh the recording duration information 192 and other information; when the user clicks the next control 195, the display device can control the display to display the video recording file Record in the follow-up playback interface i+1 , and refresh the recording time information 192 and other information; users can understand the standard action of the previous practice through the score prompt information 196 i The accumulated action group practice scores.

[0209] In some embodiments, it is assumed that the starting point of the training video is T0 (usually 00:00), and the ending point of the training video is T S , T S It can be equal to the total duration t1 of the training program video, or T S The display device can control the screen recording API to start recording the screen content when media_progress=T0, and start the timer. i _config_time, read the timer when the first time T is recorded i1 , the first time T i1 Set to record_ActionGroup i _start_time, the timer will continue to count after that, and will not stop counting due to the pause of the training mode. i _end_time, read the timer when the second time T is recorded i2 , the second time T i2 Set to record_ActionGroup i _end_time. And so on, until media_progress = TS , the display device can control the screen recording API to stop recording, thereby recording the user's complete practice process in a complete screen recording file Record.

[0210] In some embodiments, the display device can obtain the complete screen recording file Record from the preset storage location in response to the user clicking the i-th playback control 181, and obtain the Action i The first time of mapping T i1 and the second time T i2 , according to the first time T i1 and the second time T i2 , find the recording segment Record_part from the complete screen recording file Record i , record segment Record_part i The starting point is T i1 , Record_part i The end point is T i2 , so the recording segment Record_part i Recorded user actions i The practice process.

[0211] FIG20 is another schematic diagram of a follow-up playback interface according to some embodiments. Referring to FIG20 , the display device can jump the playback progress of the complete screen recording file Record directly to the starting point T in response to the user clicking the i-th playback control 181. i1 To make the practice playback interface from the recording segment Record_part i Start playing until the playback progress of the complete screen recording file Record reaches the end point T i2 When recording the segment Record_part i When the playback is finished, you can stop playing the complete screen recording file Record. i When the playback progress is T i1 Gradually transition to T i2 .

[0212] In some embodiments, the display device can also perform actions according to each standard action. i The first time of mapping T i1 and the second time T i2 , split the recording segment Record_part from the complete screen recording file Record i , and save each recording segment Record_part independently i The video file is created, and the i-th playback control 181 and the recording segment Record_part are created.i Thus, the display device can control the display to display the recording segment Record_part in the follow-up playback interface in response to the user clicking the i-th playback control 181. i , record segment Record_part i Please refer to Figure 19 for the follow-up playback interface, which will not be repeated here.

[0213] Through the above process, users can refer to the training project video and perform follow-up exercises. The image collector can collect follow-up images containing the follow-up exercises. The display device can display the training project video and the follow-up images in different windows, respectively, so as to provide users with visible action comparisons. Operators can mark the training project video according to the standard actions contained in the training project video, thereby dividing the entire training project video into action group segments corresponding to different standard actions, and configuring an action group list. Since each standard action has a certain continuous training duration, the action group segment can contain multiple frames of continuous reference action images. The action group list includes the action group information corresponding to each standard action. The action group information includes action information (such as action name, ID, etc.) and action start and end time.

[0214] In this way, the display device can monitor the playback progress of the training project video, locate the current action group segment based on the playback progress and the action group list, and determine which standard action the user is practicing. Through the alignment mechanism of the reference action image and the practice image on the timestamp, the two aligned reference action images and the practice image are compared to obtain the training score of the user's current single practice action. Similarly, at the end of the playback of the action group segment, the training scores corresponding to multiple consecutive frames of practice images are accumulated to obtain a comprehensive practice score for the action group. The action group practice score can be used to evaluate the practice of a single standard action. In addition, the present disclosure also displays the action group practice score on the practice interface to enable the user to know his or her practice score for each standard action, so as to understand which action in the entire training project video has a high score and which action is not standard, and to strengthen the training of low-scoring actions in a targeted manner, thereby enhancing the user's training enthusiasm, making up for the deficiencies in training, and improving the user's practice experience.

[0215] In order to further improve the user experience, the display device of the embodiment of the present disclosure can also record the practice video when the user is practicing the training project video, and score each practice action group that the user is practicing, and obtain a data file of the practice video, which includes group information of each practice action group, and the group information of the practice action group includes the second practice score of the practice action group. When the display device plays back the practice video, it also displays a first progress bar of the practice video, and the first progress bar is used to show the score level of each practice action group. In addition, when the user is practicing the training project video, the display device can display the filmed practice action group and the training project video, as well as the second progress bar of the training project video. The second progress bar is used to show the score level of the practice action group corresponding to the standard action group that has been practiced. In this way, the user can quickly determine the practice status of each action group by browsing the progress bar. Specifically, as shown in Figure 21, Figure 21 is another flow chart of the device control method according to some embodiments, including the following steps:

[0216] S2101: In response to a user's follow-up instruction, filming a follow-up movement group of a training program video in which the user is following the training program, displaying a follow-up training interface, and recording the follow-up training interface to obtain a follow-up training video. The follow-up training interface displays the follow-up movement group and a second movement group in the training program video in correspondence.

[0217] As mentioned above, a training program video may include action group clips corresponding to at least one standard action, and all action group clips corresponding to any standard action are the second action group.

[0218] In some embodiments, each second action group corresponds to a time period. For ease of distinction, the time period corresponding to the second action group can be referred to as the second time period. For example, a training program video may include three second time periods, and the second action groups corresponding to the three second time periods are warm-up exercises, sit-ups, and plank support.

[0219] In some embodiments, a fitness application may include a follow-up training button on the application's main interface, application menu, or playback interface of a workout video. After the user opens the fitness application on a display device, the user may input a follow-up training instruction using the follow-up training button to instruct the display device to enable the follow-up training function of the fitness application.

[0220] In some embodiments, the display device can turn on the shooting function of its own configured camera or an external camera to shoot the user's follow-up movements when playing the training program video.

[0221] In other embodiments, the display device may enable a function of receiving shooting data from a camera device, so as to receive shooting data transmitted by the camera device when playing a training program video, where the shooting data is a video stream including the user's follow-up movements.

[0222] For ease of distinction, a group of follow-up exercises performed by a user following a video can be referred to as a follow-up exercise group. The follow-up exercise group has a corresponding relationship with the second exercise group. The second exercise group has a corresponding relationship with the follow-up exercise group captured during the time period of playing the second exercise group. For example, the second exercise group in the training program video is played (during which time the playback can be paused without affecting the user's follow-up exercise), such as the squatting exercise group, and the user's squatting exercise is captured at the same time. The entire set of follow-up exercises captured is the follow-up exercise group corresponding to the squatting exercise group serving as the second exercise group.

[0223] The display device may display a follow-up training interface, and display the follow-up training movement group and the second movement group in the training program video in a corresponding manner in the follow-up training interface, wherein the follow-up training movement group and the second movement group have a corresponding relationship. The display device may display the follow-up training movement group in a first window and display the second movement group in a second window.

[0224] In some embodiments, the first window and the second window can be displayed in a split-screen manner, such as the first window and the second window can be arranged in a left-right direction or in an up-down direction. In some embodiments, the first window can be displayed on the second window in a floating or overlay manner, such as when the first window is a projection window. In some embodiments, the first window can be displayed as a picture-in-picture (PIP) window of the second window.

[0225] The display device can record the practice interface to obtain a practice video, such as the video file ai_video.mp4. In this way, the practice video includes the practice action group and the second action group. The total length of the practice video is the recording time.

[0226] S2102: After playing a second action group, scoring the follow-up action group according to the second action group and the follow-up action group corresponding to the second action group to obtain group information of the follow-up action group.

[0227] The training program video may include dot data, which may include the start time and end time of each second period in the training program video. The display device may determine whether the second action group has been played based on the dot data. For example, if the display device recognizes dot data for the start time, it may determine that the second action group has started playing. If the display device recognizes dot data for the end time, it may determine that the second action group has ended playing.

[0228] The training program video may also include the name of each second action group, the corresponding second time period, and the standard score (action_standard_score) of the corresponding follow-up action group. The standard score of the follow-up action group is used to evaluate the score level of the follow-up action group. The standard score of the follow-up action group is related to the difficulty of the corresponding second action group. For example, if the difficulty of the second action group is high, the standard score of the corresponding follow-up action group is low; if the difficulty of the second action group is low, the standard score of the corresponding follow-up action group is high.

[0229] The group information of the follow-up action group may include a second follow-up score for the follow-up action group. The display device may score the user's follow-up action for each second action group (i.e., the corresponding follow-up action group) to obtain the second follow-up score for the follow-up action group. It should be understood that the second follow-up score for the follow-up action group is equivalent to the action group score mentioned above.

[0230] In some embodiments, after the display device recognizes the dot data of the start time, it can determine that a second action group has started to be played, and the display device can mark the current time in the follow-up video accordingly, and the current time is the start time of the follow-up action group corresponding to the second action group. After the display device recognizes the dot data of the end time, it can determine that a second action group has been played, and the display device can mark the current time in the follow-up video accordingly, and the current time is the end time of the follow-up action group corresponding to the second action group. The start time and end time of the follow-up action group correspond to the first time period of the follow-up action group in the follow-up video. After determining that the second action group has been played, the display device can extract the video data corresponding to the first time period from the follow-up video according to the start time and end time of the follow-up action group. The video data is the video data containing the follow-up action group corresponding to the second action group, and the display device can score the follow-up action group in the video data.

[0231] In some embodiments, the display device can use the NUP chip, based on frameworks such as Caffe and TensorFlow, to use an AI algorithm module to score the follow-up action group in the extracted video data to obtain a follow-up score, which represents the similarity between the follow-up action group and the second action group. Among them, if the similarity between the follow-up action group and the second action group is higher, the second follow-up score of the follow-up action group will be higher; if the similarity between the follow-up action group and the second action group is lower, the second follow-up score of the follow-up action group will be lower. In some embodiments, the follow-up score can be based on a percentage system or a ten-point system.

[0232] In some embodiments, the group information of the follow-up action group may further include the start time and the end time of the follow-up action group.

[0233] In some embodiments, the group information of the follow-up action group may further include a name of the follow-up action group. The display device may obtain the name of a second action group corresponding to the follow-up action group and use the name as the name of the follow-up action group.

[0234] In some embodiments, the group information of the follow-up action group may further include a standard score of the follow-up action group. The display device may obtain the standard score of the follow-up action group corresponding to the second action group corresponding to the follow-up action group.

[0235] S2103, storing the follow-up training video and data file, where the data file includes group information of each follow-up training action group.

[0236] The display device can store the group information and the follow-up score of the follow-up action group as a data file, such as ai_data.config. The data file can use a recording method similar to that of Figure 12 to record the data results, and the recorded data results can be:

[0237] record_action_group_info{

[0238] action_name, / / name of the action group to follow

[0239] action_start_time, / / Start time of the action group

[0240] action_end_time, / / end time of the action group

[0241] action_score, / / The second follow-up score of the action group

[0242] action_standard_score, / / Standard score for the action group

[0243] …

[0244] }

[0245] The display device can refer to the above steps S2101-S2103 to obtain the follow-up video of the training project video and the data file of the follow-up video, and can review its own follow-up training situation by playing back the follow-up video.

[0246] Taking the example of a follow-up training video recorded by a display device including at least one follow-up training action group, the display device can play back the follow-up training video with reference to the process shown in FIG22 , which specifically includes the following steps as shown in FIG22 :

[0247] S2201, in response to a user's playback instruction, generating a first progress bar of the follow-up video according to the group information of each follow-up action group, wherein the first progress bar is used to display the score level of each follow-up action group.

[0248] The playback command is used to instruct the playback of the practice video.

[0249] In some embodiments, the first progress bar includes n first time periods, each of which corresponds to a follow-up exercise group. Each first time period can be labeled with a second follow-up exercise score for the corresponding follow-up exercise group. In this way, the user can determine each follow-up exercise group and the second follow-up exercise score for each follow-up exercise group based on the first progress bar.

[0250] In some embodiments, the first progress bar includes n first time periods, each of which corresponds to a follow-up exercise group. Each first time period is accompanied by a first identifier, which indicates the score level of the follow-up exercise group corresponding to the first time period. The score level indicates the degree of compliance of the follow-up exercise group. Although the second follow-up score of the follow-up exercise group can reflect the standard level of the follow-up exercise group, due to the different difficulty levels of different second exercise groups, the second follow-up score of the follow-up exercise group with a greater difficulty level will generally be lower. Conversely, the second follow-up score of the follow-up exercise group with a less difficult level will generally be higher. It can be seen that the second follow-up score of the follow-up exercise group lacks reference to the difficulty of the second exercise group itself, which can easily lead to a lack of objectivity in the scoring. Compared to the second follow-up score of the follow-up exercise group, the score level of the follow-up exercise group is a scoring result given with reference to the difficulty of the second exercise group itself, which is more objective. In this way, users can more accurately understand their follow-up exercise situation.

[0251] The scoring levels of the follow-up action group can be divided according to the accuracy of the follow-up action group. The accuracy of the follow-up action group is the percentage of the second follow-up score of the follow-up action group to the standard score of the follow-up action group. The scoring levels of the follow-up action group can include m levels, where m is an integer greater than 0. For example, an accuracy rate greater than or equal to 80% can be set as a first scoring level, such as perfect; an accuracy rate greater than or equal to 70% and less than 80% can be set as a second scoring level, such as excellent; an accuracy rate greater than or equal to 50% and less than 70% can be set as a third scoring level, such as very good; and an accuracy rate less than 50% can be set as a fourth scoring level, such as keep it up.

[0252] In some embodiments, the first identifier can be a filling style of each first time period in the first progress bar, such as color, pattern, etc. For example, purple represents perfect, pink represents great, yellow represents very good, and blue represents cheer.

[0253] In some embodiments, the first identifier can be an emoticon displayed in the first progress bar corresponding to the first time period, for example, laughing represents perfect, smiling represents great, frowning represents very good, and crying represents cheering.

[0254] In some embodiments, the first identifier may be text displayed in the first progress bar corresponding to the first time period, for example, "perfect", "very good", "very good", or "keep going".

[0255] In some embodiments, the first progress bar includes n first time periods, each of which corresponds to a follow-up exercise group. Each first time period is marked with a first identifier, and the second follow-up exercise score of the corresponding follow-up exercise group can be annotated in each first time period. In this way, the user can not only understand the second follow-up exercise score of each follow-up exercise group, but also understand the score level of each follow-up exercise group.

[0256] In some embodiments, the display device may generate a first progress bar according to the process shown in FIG23 , which specifically includes the following steps:

[0257] S2301: Calculate the length of a first time period corresponding to the follow-up exercise group on a first progress bar based on the total duration of the follow-up exercise video and the start and end times of the follow-up exercise group. If each first time period is marked with a first identifier, calculate a score level for the follow-up exercise group based on a second follow-up exercise score for the follow-up exercise group and a standard score for the follow-up exercise group.

[0258] The display device obtains the total duration of the follow-up training video (all Length), and the display device determines the total length of the first progress bar (all Length), where the total length of the first progress bar represents the total duration of the follow-up training video.

[0259] The display device traverses the data file, obtains the start time and end time of each follow-up action group, and calculates the length of the first period corresponding to each follow-up action group on the first progress bar. The display device can calculate the length of the first period according to formula (1):

[0260] action_ProgressBar i =((action_end_time-action_start_time) / all Second_)×all Length (1)

[0261] Among them, action_ProgressBar i It represents the length of the first period corresponding to the i-th follow-up action group on the first progress bar, and action_end_time-action_start_time represents the length of the first period corresponding to the i-th follow-up action group.

[0262] The display device traverses the data file, obtains the second follow-up score and the standard score of each follow-up action group, and calculates the accuracy of each follow-up action group. The display device can calculate the accuracy according to the following formula (2):

[0263] action Accuracyi=action_scorei / action_standard_scorei(2)

[0264] Among them, action Accuracy i Represents the accuracy of the i-th training action group, action_score i Represents the second follow-up score of the i-th follow-up action group, action_standard_score i Represents the standard score of the i-th follow-up action group.

[0265] The display device can determine the score level of each follow-up action group based on the preset correspondence between the accuracy rate and the score level.

[0266] S2302: Generate a first progress bar according to the length of the first time period corresponding to each follow-up action group on the first progress bar and the score level of each follow-up action group.

[0267] The display device draws a first progress bar according to the length of the first time period corresponding to each follow-up action group on the first progress bar, and adds a first mark on the first progress bar according to the score level of each follow-up action group.

[0268] If the display device marks the second follow-up score of the follow-up action group on the first progress bar, the first progress bar will be drawn according to the length of the first time period corresponding to each follow-up action group on the first progress bar, and the follow-up score will be marked at the corresponding position of each first time period according to the second follow-up score of each follow-up action group.

[0269] S2202, displaying a playback interface, playing the practice video in the playback interface, and displaying a first progress bar.

[0270] In the playback interface, the display device can play the practice video in the first window and display the first progress bar in the second window, and the second window can be displayed on the upper layer of the first window. The second window can be displayed at the top or bottom of the playback interface.

[0271] In some embodiments, the size of the second window may match the size of the first progress bar.

[0272] In some embodiments, the size of the second window may be larger than the size of the first progress bar, wherein the portion of the second window that exceeds the first progress bar is transparent to avoid excessively blocking the image in the first window.

[0273] In some embodiments, the first progress bar also displays a focus, and the position of the focus on the first progress bar is associated with the playback progress of the follow-up practice video. The position of the focus on the first progress bar indicates the playback progress of the follow-up practice video. By adjusting the position of the focus on the first progress bar, the playback progress of the follow-up practice video can be adjusted in association. The display device can calculate the length of the focus from the starting position of the first progress bar, that is, the position of the focus on the first progress bar, based on the total length of the first progress bar, the total duration of the follow-up practice video, and the playback progress of the follow-up practice video (the duration played).

[0274] Taking the recorded follow-up training video including 5 follow-up training action groups as an example, the 5 follow-up training action groups are warm-up exercises, squats, sit-ups, plank support, and stretching. Among them, the second follow-up training score of each follow-up training action group is 80 points, 65 points, 30 points, 70 points, and 35 points, respectively, and the recording time corresponding to each follow-up training action group is equal.

[0275] Example 1: The display device may display a playback interface as shown in FIG24 in response to the user's playback instruction. In the playback interface, the display device plays the follow-up practice video in the first window 1001. The display device displays a first progress bar in the second window 1002, and the display device may draw the first progress bar according to steps S2301-S2302. The first progress bar includes 5 first time periods, and each first time period corresponds to 5 follow-up practice action groups. The lengths of the 5 first time periods are equal, and the second follow-up practice score of the corresponding follow-up practice action group is marked on each first time period.

[0276] In Example 2, the display device may respond to a user's playback command and display a playback interface as shown in FIG25 . In this playback interface, the display device plays the follow-up exercise video in a first window 1101. The display device displays a first progress bar in a second window 1102. The display device may draw the first progress bar according to steps S2301-S2302. If the standard scores of each follow-up exercise group are 95, 100, 70, 90, and 80, respectively, then the accuracy rates of each follow-up exercise group are 84%, 65%, 43%, 78%, and 44%, respectively. If the preset correspondence between the accuracy rate and the score level is: an accuracy rate greater than or equal to 80% corresponds to perfect; an accuracy rate greater than or equal to 70% and less than 80% corresponds to excellent; an accuracy rate greater than or equal to 50% and less than 70% corresponds to very good; and an accuracy rate less than 50% corresponds to "good," then the warm-up corresponds to perfect, the squat corresponds to very good, the sit-up corresponds to "good," the plank corresponds to excellent, and the stretching corresponds to "good." As shown in Figure 25, the first progress bar includes five first time periods, each corresponding to five follow-up exercise groups. The five first time periods are of equal length, and a first marker corresponding to the follow-up exercise group is added to each first time period. The first marker uses the fill style of the first time period in the first progress bar, where the first marker corresponding to "perfect" is a diagonal stripe, the first marker corresponding to "very good" is a grid, the first marker corresponding to "very good" is a spotted stripe, and the first marker corresponding to "come on" is a vertical stripe.

[0277] Example 3, the display device may display a playback interface as shown in FIG26 in response to the user's playback instruction. In the playback interface, the display device plays the follow-up video in the first window 1201. The display device displays a first progress bar in the second window 1202, and the display device may draw the first progress bar according to steps S2301-S2302. The first progress bar is marked with the second follow-up score of each follow-up action group (see Example 1), and the first identifier corresponding to each follow-up action group is added (see Example 2).

[0278] In some embodiments, the display device may also display a second follow-up score of the played follow-up action group on the playback interface.

[0279] When the display device plays a follow-up exercise video, if it recognizes the start time of a follow-up exercise group, it obtains the second follow-up exercise score for that follow-up exercise group and can display the follow-up exercise score in the area corresponding to the follow-up exercise group displayed on the playback interface. With reference to Figures 24, 25, and 26, when a sit-up is currently playing on the playback interface, the display device can display the follow-up exercise score for the sit-up, i.e., 30 points, above the area displaying the follow-up exercise group.

[0280] If the display device recognizes the end time of the follow-up exercise group, it will no longer display the follow-up score. Since the end time of the previous follow-up exercise group and the start time of the next follow-up exercise group coincide with each other, the display device will recognize the start time of the next follow-up exercise group when it recognizes the end time of the follow-up exercise group. Therefore, the display device switches to displaying the second follow-up score of the next follow-up exercise group.

[0281] The display device can play the follow-up training video according to the process shown in Figure 27. The specific steps are as follows:

[0282] S2701, determining the playback speed of each follow-up action group in the follow-up video according to the preset correspondence between the score level and the playback speed.

[0283] The display device may store a correspondence between score levels and playback speeds, wherein a higher score level corresponds to a faster playback speed, and a lower score level corresponds to a slower playback speed.

[0284] The correspondence may include the correspondence between all score levels and playback speeds.

[0285] The correspondence may include a partial correspondence between score levels and playback speeds. For example, only the correspondence between lower score levels and playback speeds may be included, such as the correspondence between "Come on" and playback speed. Another example may be only the correspondence between higher score levels and playback speeds, such as the correspondence between "Perfect" and playback speed.

[0286] The corresponding relationship may be a default corresponding relationship configured by the display device, or a user-defined corresponding relationship.

[0287] The display device's preset correspondence between score levels and playback speeds determines the playback speed of each follow-up exercise group. For score levels not included in the preset correspondence, the corresponding default playback speed can be determined.

[0288] S2702: Play the first time period of the corresponding follow-up action group in the follow-up video at the playback speed of each follow-up action group.

[0289] To illustrate with reference to Example 2, if the preset correspondence between the score level and the playback speed includes that cheering corresponds to 0.5 times the speed, then the playback speed of sit-ups and stretching can be determined to be 0.5 times the speed. If the default playback speed is 1.0 times the speed, then the playback speed of the first warm-up exercise, squats, and plank support can be determined to be 1.0 times the speed. In this way, when playing the follow-up video, the first period corresponding to the first warm-up exercise and squats is played at 1.0 times the speed, followed by the first period corresponding to sit-ups at 0.5 times the speed, and then the first period corresponding to plank support is played at 1.0 times the speed, and finally the first period corresponding to stretching is played at 0.5 times the speed.

[0290] In this way, the display device can adaptively adjust the playback speed of each follow-up movement group in the follow-up practice video based on the user's follow-up practice status. For example, when the user's completion rate is high, such as when the follow-up movement group corresponds perfectly, the playback speed of the follow-up movement group can be accelerated to avoid excessive playback delays. For another example, when the user's completion rate is poor, such as when the follow-up movement group corresponds to encouragement, the playback speed of the follow-up movement group can be slowed down to allow the user to review their follow-up movements and the corresponding second movement group more carefully.

[0291] The display device can quickly switch and play each follow-up action group according to the process shown in Figure 28. The specific steps are as follows:

[0292] S2801: In response to a jump instruction from a user, the focus is moved to a first progress on a first progress bar, wherein the focus is moved to a starting position of each first time period in the first progress bar.

[0293] The jump instruction is used to instruct to move the focus on the first progress bar. When the focus is on the first progress bar, the user can input the jump instruction using the left key and the right key on the control device 100.

[0294] The display device configures the focus on the first progress bar to move on the starting position of each first time period in the first progress bar, that is, in response to the user's jump instruction, when the focus moves on the first progress bar, it moves from the current position to the starting position of the next first time period.

[0295] Based on how the focus moves on the first progress bar, the first progress corresponds to the start time of the next follow-up action group.

[0296] S2802: Play the follow-up video starting from the first progress. The display device determines the start time of the next follow-up action group and jumps to the start time to start playing.

[0297] To illustrate with Example 2, if the user inputs a jump command using the right arrow key on control device 100, the display device, through the fitness app, directly jumps the focus from the current position on the first progress bar to the start position of the first period corresponding to the plank exercise, i.e., jumps to the first progress bar. The display device then begins playing the exercise video from the first progress bar, i.e., the plank exercise.

[0298] In this way, users can use each follow-up action group as a viewing unit and quickly switch between the follow-up action groups.

[0299] The display device is further configured to provide a function of continuously playing a set of follow-up exercises of a specified score level. The display device can continuously play a set of follow-up exercises of a specified score level according to the process shown in FIG29 . The specific steps are as follows:

[0300] S2901: Display a button on the playback interface, where the button indicates the corresponding score level.

[0301] The score levels indicated by each button include the score levels corresponding to each follow-up action group.

[0302] In some embodiments, the display device displays buttons on the playback interface, and the button indicates a score level that is consistent with the score level corresponding to each follow-up exercise group. For example, if the score levels include perfect, excellent, very good, and go for it, and the score levels corresponding to each follow-up exercise group include perfect, very good, and go for it, then three buttons indicating perfect, very good, and go for it, respectively, will be displayed on the playback interface.

[0303] In some embodiments, the display device displays buttons on the playback interface, and the score levels indicated by the buttons include the score levels corresponding to each follow-up exercise group. For example, if the score levels include Perfect, Great, Very Good, and Cheer Up, four buttons indicating Perfect, Great, Very Good, and Cheer Up are displayed on the playback interface. If the score levels corresponding to each follow-up exercise group include Perfect, Great, and Cheer Up, the three buttons indicating Perfect, Very Good, and Cheer Up are enabled, and the button indicating Great is unavailable, such as grayed out.

[0304] S2902: In response to the user selecting a designated button, obtaining a waiting time period in the follow-up training video, wherein the waiting time period is the first time period of the follow-up training action group corresponding to the score level indicated by the designated button.

[0305] The score level indicated by the designated button is the score level that the user wants to watch continuously. For example, if the user is more concerned about the movement group with poor training results, the score level that the user wants to watch continuously may be "Come on", and accordingly, the score level indicated by the designated button is "Come on".

[0306] The user can control the focus to move to a designated button through the control device 100 and select the designated button based on the confirmation key in the control device 100. In some embodiments, there can be one or more designated buttons, that is, the user can have one or more score levels that he or she wants to watch continuously.

[0307] In some embodiments, the display device can, in response to a user selecting a designated button, determine the score level indicated by the designated button (hereinafter referred to as the designated level), and further determine a follow-up action group corresponding to the designated level (hereinafter referred to as the designated action group). The display device can obtain the first time period corresponding to each designated action group in the follow-up video, i.e., the waiting time period, based on the start time and end time of the designated action group.

[0308] S2903: Continuously play the waiting time period and do not play other time periods.

[0309] The display device continuously plays each waiting time period and does not play other time periods. In this way, the user can more focusedly watch the follow-up action group corresponding to the score level of concern.

[0310] Based on Example 2, as shown in Figure 30, the display device also displays buttons 1601, 1602, 1603, and 1604 on the playback interface. Among them, the score level indicated by button 1601 is perfect, the score level indicated by button 1602 is great, the score level indicated by button 1603 is very good, and the score level indicated by button 1604 is Come on. If the score level that the user wants to play continuously is Come on, the user can select button 1604 through the control device 100. The display device determines the designated action group with a score level of Come on, namely, sit-ups and stretching. The display device obtains the first time period corresponding to sit-ups and stretching in the follow-up video, namely the waiting time period. The display device plays the two waiting time periods continuously to play sit-ups and stretching continuously without playing warm-up exercises, squats, and plank support.

[0311] In some embodiments, if the user selects multiple specified buttons, that is, the user wants to continuously play multiple specified levels of follow-up action groups, the display device can play each waiting time period in sequence according to the recording order of the waiting time period in the follow-up video after determining each waiting time period.

[0312] 30 , if the user selects button 1603 and button 1604 , that is, the user wants to continuously play the waiting time periods corresponding to “very good” and “come on”, the display device will continuously play the waiting time periods corresponding to squats, sit-ups and stretching.

[0313] In some embodiments, if a user selects multiple designated buttons, meaning the user wishes to continuously play multiple practice exercise groups of designated levels, the display device, after determining each pending time period, may sequentially play the pending time periods within each group. The group includes the pending time periods corresponding to the score level indicated by a designated button. Within each group, the display device sequentially plays the pending time periods in the order in which they were recorded in the practice video.

[0314] In some embodiments, the display device may play each packet sequentially in a random order.

[0315] In some embodiments, the display device may play the corresponding groups in sequence according to the order in which the user selects the designated buttons.

[0316] In some embodiments, the display device may play the corresponding groups in sequence according to the order in which the first waiting-to-play time period in each group is recorded in the follow-up video.

[0317] In some embodiments, the display device may play the corresponding groups in order according to the ranking of the scores.

[0318] As illustrated in conjunction with FIG30 , if the user selects button 1602 and button 1604, that is, the user wants to continuously play the follow-up exercise groups corresponding to “Very Good” and “Come On”, the display device can determine two groups, one group including the waiting time period corresponding to “Very Good”, and the other group including the waiting time period corresponding to “Come On”. The display device plays the waiting time periods in the two groups in sequence. For example, the display device can first play the group corresponding to “Very Good”, and then play the group corresponding to “Come On”. Since there are multiple waiting time periods corresponding to “Come On”, when playing the group corresponding to “Come On”, each waiting time period can be played in sequence according to the order in which the waiting time periods were recorded in the follow-up exercise video. In this way, the waiting time periods continuously played by the display device correspond to plank support, sit-ups, and stretching in sequence.

[0319] In some embodiments, if the display device plays the waiting time period in each group in sequence according to the grouping, the action_start_time and action_end_time of each group can be extracted. The action_start_time of the group is the start time of the first waiting time period in the group, and the action_end_time of the group is the end time of the last waiting time period in the group. The display device can jump the follow-up video to the first waiting time period in the corresponding specified level by calling seek(action_start_time), and start playing each waiting time period in the group in sequence. When the follow-up video is played to action_end_time, it calls seek(action_start_time) of the next group and starts playing each waiting time period in the group. In this way, all groups are played.

[0320] In order to facilitate users to keep track of their own training progress at any time during the training process, the display device can score each of the user's training action groups in real time and display the corresponding score level in real time. For example, the display device plays a training project video through a fitness application, and the training project video includes n second action groups, where n is an integer greater than 0. The display device can display the training interface according to the process shown in Figure 31. The specific steps are as follows:

[0321] S3101: In response to a user's follow-up instruction, capture a follow-up movement group of a training program video being followed by the user and display a follow-up training interface. The follow-up training interface displays the follow-up movement group and the second movement group, respectively, and displays a second progress bar for the training program video; the second progress bar includes at least one second time period, each second time period corresponding to a second movement group.

[0322] Step S3101 can refer to step S2101, the main difference is that the display device also displays a second progress bar of the training item video in the follow-up training interface.

[0323] In some embodiments, the second progress bar may be a video progress bar that comes with the training program video.

[0324] In some embodiments, the second progress bar can be a progress bar generated by the display device based on the training project video. The display device can set a total length (Length_M) of the second progress bar, which represents the total duration (all Second_M) of the training project video. The display device can calculate the length of the second period corresponding to each second action group on the second progress bar based on the start time (action_start_time) and end time (action_end_time) of each second action group in the training project video, as well as the total duration of the training project video and the total length of the second progress bar. The display device can calculate the length of the second period according to formula (3):

[0325] action_ProgressBarj=((action_end_time-action_start_time) / all Second_M)Length_M (3)

[0326] Among them, action_ProgressBarj represents the length of the second time period corresponding to the j-th second action group on the second progress bar, and action_end_time-action_start_time represents the length of the second time period corresponding to the j-th second action group.

[0327] In the follow-up training interface, the display device can display the training program video and follow-up training movements in a first window and a second window, and display a second progress bar in a third window. The third window is located above the first window and the second window.

[0328] In one implementation, the size of the third window may match the size of the second progress bar.

[0329] In some embodiments, the size of the third window may be larger than the size of the second progress bar, wherein the portion of the third window that exceeds the second progress bar is transparent to avoid excessively blocking the images in the first window and the second window.

[0330] S3102, after playing a second action group, calculating the second follow-up practice score of the follow-up practice action group corresponding to the second action group, and calculating the score level of the follow-up practice action group based on the second follow-up practice score of the follow-up practice action group and the standard score of the follow-up practice action group.

[0331] The process of the display device calculating the second follow-up score of the follow-up action group and obtaining the corresponding group information can refer to step S2102 and is not repeated here. The process of the display device calculating the score level of the follow-up action group can refer to step S2301 and is not repeated here.

[0332] S3103, adding a second identifier to the target time period in the second progress bar according to the score level of the follow-up action group; the target time period is the second time period corresponding to the second action group corresponding to the follow-up action group, and the second identifier is used to indicate the score level of the follow-up action group.

[0333] In some embodiments, the display device may mark the second follow-up score of the corresponding follow-up action group on the target time period in the second progress bar.

[0334] In some embodiments, the display device may add a second marker to the target time period in the second progress bar.

[0335] The style of the second logo can refer to the first logo and will not be repeated here.

[0336] In some embodiments, if the second progress bar is a video progress bar that comes with the training project video, the display device can draw a second identifier in a separate window. For example, in the follow-up training interface, the display device displays the training project video and the follow-up training movements in a split screen in the first window and the second window, and the display device can draw the second identifier in the third window. The third window is located on the upper layer of the first window and the second window. The position of the second identifier in the follow-up training interface is consistent with the position of the second progress bar in the follow-up training interface, and the coverage size of the second identifier is consistent with the display size of the corresponding target time period in the second progress bar. In this way, the added second identifier can be covered on the corresponding target time period and matched with the size of the target time period, presenting an effect of a self-contained identifier on the second progress bar. The calculation process of the size and position of the second identifier can refer to formula (3).

[0337] In some embodiments, if the second progress bar is a progress bar generated by the display device according to the training program video, the display device may directly draw the second marker on the second progress bar in the third window.

[0338] For example, a training video includes five secondary exercise groups: warm-up, squats, sit-ups, planks, and stretches. Each secondary exercise group has equal playback duration. The video has currently reached the sit-up, and the warm-up and squat secondary exercise scores are 80 and 65, respectively.

[0339] Example 4, the display device may display a follow-up training interface as shown in Figure 32 in response to the user's follow-up training instruction. In the follow-up training interface, the display device plays the training project video in the first window 1801 and plays the follow-up training action group in the second window 1802. The display device also displays a second progress bar of the training project video in the third window 1803. The second progress bar includes 5 second time periods, each of which corresponds to 5 second action groups, and the lengths of the 5 second time periods are equal. The display device marks the corresponding second follow-up training scores on the target time periods corresponding to the warm-up exercise and squats on the second progress bar, which are 80 points and 65 points respectively.

[0340] Example 5, the display device can display the follow-up training interface as shown in Figure 33 in response to the user's follow-up training instruction. In this follow-up training interface, the display device plays the training project video in the first window 1901 and plays the follow-up training action group in the second window 1902. The display device also displays a second progress bar of the training project video in the third window 1903. The second progress bar includes 5 second time periods, each second time period corresponds to 5 second action groups, and the lengths of the 5 second time periods are equal. The display device adds a second identifier to the target time period corresponding to the warm-up exercise, squat action, etc. displayed in the second progress bar. The process of determining the score level of the follow-up training action group corresponding to the warm-up exercise and squat action, etc. can refer to Example 2 and will not be repeated here. The second identifier adopts the filling style of the second time period in the second progress bar. For example, as shown in Figure 33, the second identifier for perfect correspondence can be a diagonal pattern, the second identifier for very good correspondence can be a grid pattern, etc., and is distinguished by filling style. For details, please refer to the filling method of the aforementioned first identifier and will not be repeated here.

[0341] Example 6: In response to the user's follow-up training instruction, the display device may display a follow-up training interface as shown in Figure 34. In this follow-up training interface, the display device plays the training project video in the first window 2001 and plays the follow-up training movement group in the second window 2002. The display device also displays a second progress bar of the training project video in the third window 2003. The display device marks the second follow-up training score of the corresponding follow-up training movement group on the second progress bar (see Example 4), and adds a second identifier corresponding to the corresponding follow-up training movement group (see Example 5).

[0342] In some embodiments, a buffer period may be added between two adjacent second action groups in a training program video. The buffer period may be used for the display device to display the second follow-up score of the most recently completed follow-up action group on the follow-up interface.

[0343] When the display device is playing the training program video, if it recognizes the end time of the second action group, it enters the buffer period (not shown on the second progress bar), and displays the second follow-up score of the follow-up action group that has just ended during the buffer period. In conjunction with Example 5, the display device enters the buffer period after recognizing the end time corresponding to the squat. During the buffer period, referring to Figure 35, the display device displays the result display page on the follow-up interface, which includes the second follow-up score of the follow-up action group that has just ended, that is, 65 points. After the buffer period ends, the display device continues to play the second period of sit-ups.

[0344] In this way, when following the practice, users can not only grasp the practice score of each action group in real time, but also grasp the score level of each action group.

[0345] Furthermore, in real-world applications, users typically control display devices via remote controls or voice commands. For example, users can use remote controls or voice commands to control the display device, such as playing a training video or recording a screen recording. However, if the user cannot find the remote control or the terminal device cannot accurately recognize voice commands due to a strong accent, the user will be unable to control the display device.

[0346] To further enhance the user experience, the display device of the disclosed embodiment can also perform user identification on the current image being captured, obtaining a corresponding user identification result. If the user identification result corresponding to the current image includes body information of at least one user, the current target user can be determined from the at least one user. Based on the body information of the current target user, the current target user can be detected as being within the user area of ​​the display device, obtaining a corresponding detection result. If the current image is the last frame in the current control cycle, the operating state of the display device can be controlled based on the detection result corresponding to at least one frame in the current control cycle.

[0347] In this way, the display device can determine whether the current target user corresponding to the current image is located in the user usage area, that is, determine whether there is someone / no one in the user usage area for the current target user, and then control the working state of the display device based on the someone / no one situation corresponding to at least one frame of image in the current control cycle. Therefore, there is no need for the user to actively input control instructions to the display device through a remote control or voice to control the working state of the display device. The display device can automatically determine whether there is someone / no one in the user usage area and then control the working state based on the someone / no one situation, making the control of the display device more convenient and intelligent.

[0348] The implementation may be as shown in FIG36 , which is another flow chart of a device control method according to some embodiments. As shown in FIG36 , the method includes the following steps:

[0349] S3601. In response to a user identification instruction, receive a current image captured by an image collector;

[0350] In the embodiment of the present disclosure, the user-used area of ​​the display device is the acquisition area of ​​the image collector. The image collector can send the image currently acquired in the acquisition area (i.e., the current image) to the processor. In this way, the processor can receive the current image.

[0351] In some embodiments, the user usage area is a collection of locations where the user may be when using the display device. When the user is in the user usage area, it indicates that the user is using the display device. When the user leaves the usage area, it indicates that the user has paused using the display device.

[0352] In some embodiments, the image type of the current image may include a binary image, a grayscale image, a color image, etc., but is not limited thereto.

[0353] In some embodiments, before S3601 , the process further includes: obtaining the current intelligent control state. Accordingly, if the current intelligent control state is in a valid state, the image collector is controlled to collect the collection area to obtain the current image, and the current image is received.

[0354] FIG37 is a logical diagram illustrating obtaining the current intelligent control state according to some embodiments. Referring to FIG37 , the switch monitoring management can monitor the results of intelligent control switch changes. For example, the switch monitoring management can register and monitor the database contentObserver(callback) to receive notifications of intelligent control switch changes, but is not limited to this. The display device can receive the intelligent control switch change results sent by the switch monitoring management via a processor. If the intelligent control switch change result is on, the display device can control the conflict detection module of the processor to detect whether the intelligent control service conflicts with the currently ongoing service. If no conflict exists, the intelligent control state is determined to be valid; otherwise, the intelligent control state is determined to be invalid.

[0355] It is understandable that when the current intelligent control state is valid, turning on the intelligent control can prevent the intelligent control from affecting other ongoing businesses, thereby ensuring the normal operation of the display device.

[0356] S3602: Perform user recognition on the current image to obtain a corresponding user recognition result.

[0357] In an embodiment of the present disclosure, the display device can identify the user in the current image and obtain a corresponding user identification result. If no user is identified, the user identification result is empty. If at least one user is identified, the user identification result includes the body information of the identified user.

[0358] In some embodiments, the limb information is information describing the user's limb characteristics. For example, the limb information may include, but is not limited to, the position information of multiple key limb points, the confidence information of multiple key limb points, the order and meaning of multiple key limb points, and limb contour information.

[0359] Key limb points are key points on a limb. For example, key limb points may include key points corresponding to facial organs, key points corresponding to bone joints, etc., but are not limited thereto.

[0360] The position information of key limb points can be represented by the coordinates of the key limb points in an image coordinate system, but is not limited to this. There are various ways to establish an image coordinate system. For example, Figure 38 is a schematic diagram of an image coordinate system according to some embodiments. Referring to Figure 38 , the upper left corner of the image is the coordinate origin. From the coordinate origin, the right direction is the positive X direction, and from the coordinate origin, the downward direction is the positive Y direction, but this is not a limitation.

[0361] The confidence information of key limb points is used to characterize the reliability of key limb points.

[0362] The order of key limb points is used to represent the order of key limb points among multiple key limb points, and the meaning of key limb points is used to represent which part of the limb the key limb points correspond to. For example, Figure 39 is a schematic diagram of the order of multiple key limb points and their meanings according to some embodiments. Referring to Figure 39, the key limb points and their order are as follows: 0 nose key point, 1 left eye key point, 2 right eye key point, 3 left ear key point, 4 right ear key point, 5 left shoulder key point, 6 right shoulder key point, 7 left elbow key point, 8 right elbow key point, 9 left wrist key point, 10 right wrist key point, 11 left hip key point, 12 right hip key point, 13 left knee key point, 14 right knee key point, 15 left ankle key point, 16 right ankle key point, 17 left palm key point, 18 right palm key point, but are not limited to these.

[0363] Limb outline information is used to describe the human shape outline of limbs.

[0364] In some embodiments, there are various specific implementations for performing user identification on the current image. Typical examples are described below, but do not constitute a limitation of the present disclosure.

[0365] In some embodiments, when executing the aforementioned step S3602, the current image may be input into a user recognition model, and a user recognition result output by the user recognition model may be obtained, wherein the user recognition model is pre-trained using sample images and their corresponding user recognition results.

[0366] Of course, in other embodiments, semantic segmentation can be used to separate the area where the user is located, and then an edge detection algorithm can be used to perform edge detection on the area where the user is located to obtain limb contour information, and any key limb point extraction algorithm known to those skilled in the art can be used to perform feature recognition on the area where the user is located to obtain the position information of the key limb points.

[0367] S3603: If the user recognition result corresponding to the current image includes body information of at least one user, determine a current target user from the at least one user.

[0368] In some embodiments, if the user recognition result corresponding to the current image is empty, it is determined that there is no user in the user usage area. Further, when there is no user in the user usage area, the electronic device can be controlled to enter a dormant state or shut down to save power consumption.

[0369] The current target user is the target user corresponding to the current image. Whether the target user is located in the user usage area (ie, whether the user usage area is occupied or not for the target user) is a reference factor for controlling the working state of the display device.

[0370] There are many specific implementations for determining the current target user from at least one user. Typical examples are described below, but do not constitute a limitation of the present disclosure.

[0371] In some embodiments, determining the current target user from at least one user may include: randomly selecting one of the at least one user as the current target user. Implementation may include the following steps:

[0372] S361. If the total number of at least one user is 1, then the user is used as the current target user;

[0373] S362: If the total number of at least one user is greater than 1, determine a current target user from the at least one user based on the body information of the at least one user.

[0374] S362 may include: S3621, matching the historical target user with at least one user based on limb information, wherein the historical target user is the target user determined based on the previous frame image; if there is a user matching the historical target user among at least one user, the historical target user is used as the current target user; if there is no user matching the historical target user among at least one user, a user who meets the preset conditions is selected from at least one user as the current target user.

[0375] In some embodiments, any matching algorithm known to those skilled in the art can be used to match the historical target user with at least one user (i.e., the user identified from the current image) to achieve tracking of the historical target user. In some embodiments, the historical target user is matched with at least one user based on limb contour information in the limb information. For example, for each of the at least one user (i.e., the user identified from the current image), the limb size information (i.e., limb width and height) of the user is determined based on the limb contour information of the user, and the intersection over union (IOU) of the user and the historical target user is calculated based on the limb size information. Then, based on the IOU, it is determined whether the user matches the historical target user. For another example, at least one user (i.e., the user identified from the current image) and the users identified in the previous frame image form a user pair; for each user pair, the limb size information (i.e., limb width and height) of the two users in the user pair is determined based on the limb contour information of the two users, and the IOU of the two users is calculated based on the limb size information; based on the IOU of the two users in each user pair, the user identified in the previous frame image is matched with the user identified in the current image; if there is a user among the users identified in the current image who matches the historical target user, it is used as the current target user.

[0376] In some embodiments, the specific content of the preset conditions can be set by those skilled in the art according to actual conditions and is not limited here. For example, the preset conditions may include at least one of the following: minimum distance from the image collector, maximum limb size, and the highest priority group.

[0377] FIG40 is a flow chart of determining the current target user according to some embodiments. Referring to FIG40 , the following steps are included:

[0378] S4001: Obtain the user recognition result corresponding to the current image;

[0379] S4002: Determine the number of detected users;

[0380] S4003: If multiple users are detected, a current target user is selected from the multiple users based on target tracking switching;

[0381] S4004: If a user is detected, the user is used as the current target user;

[0382] S4005: If no user is detected, the process of determining the current target user ends.

[0383] If only one user is detected for the current image, that user is selected as the current target user. If multiple users are detected, the current target user is selected from the multiple users based on target tracking switching (i.e., step S3621 described above). If no user is detected, the current process ends.

[0384] In some embodiments, determining the current target user from at least one user may include: S371, performing gesture recognition on the current image to obtain a corresponding gesture recognition result; S372, if the gesture recognition result corresponding to the current image includes a switch gesture, determining the user who performed the switch gesture from the at least one user as the current target user; S373, if the gesture recognition result does not include a switch gesture, matching a historical target user with at least one user based on body information, where the historical target user is the target user determined based on the previous image frame; if a user in the at least one user matches the historical target user, determining the historical target user as the current target user; if a user in the at least one user does not match the historical target user, selecting a user from the at least one user that meets preset conditions as the current target user. Step S373 has the same meaning as step S3621 and is not further described here. The specific gesture type of the switch gesture can be determined by those skilled in the art based on actual circumstances and is not limited here. For example, the switch gesture may include a fist gesture, a thumbs-up gesture, a five-fingers-together gesture, a scissors hand gesture, etc.

[0385] FIG41 is another flow chart of determining the current target user according to some embodiments. Referring to FIG41 , the process includes the following steps:

[0386] S4101: Obtaining gesture recognition results corresponding to the current image;

[0387] S4102: Determine whether a switching gesture occurs;

[0388] S4103: If yes, switch the target and set the user who made the switch gesture as the current target user;

[0389] S4104: If not, target tracking switching is performed based on the user recognition result corresponding to the current image to obtain the current target user.

[0390] For the gesture recognition result corresponding to the current image, if a switching gesture appears, the target is switched, and the user who makes the switching gesture is used as the current target user. If no switching gesture appears, target tracking switching is performed based on the user recognition result corresponding to the current image (i.e., S373) to obtain the current target user.

[0391] It can be understood that by setting a switching gesture, flexible switching of the current target user can be achieved. When there is no switching gesture, the current target user can be determined by target tracking switching, which can make the method of determining the current target user simple and convenient, and help reduce the difficulty of implementation.

[0392] S3604: Detect whether the current target user is located in the user usage area of ​​the display device based on the body information of the current target user, and obtain a corresponding detection result.

[0393] In some embodiments, the detection result is either that the current target user is located in the user usage area, that is, there is someone in the user usage area for the current target user, or that the current target user is not located in the user usage area, that is, there is no one in the user usage area for the current target user.

[0394] In some embodiments, S3604 may include: inputting the limb information of the current target user into a pre-trained detection model, and obtaining the detection result output by the detection model.

[0395] S3605: If the current image is the last frame image in the current control cycle, control the working state of the display device based on the detection result corresponding to at least one frame image in the current control cycle.

[0396] In some embodiments, the number of images to be collected in each control cycle can be set by those skilled in the art based on actual conditions and is not limited thereto. For example, one frame of image can be collected in each control cycle, or more than one frame of image can be collected, such as two, three, or ten frames.

[0397] In some embodiments, the final detection result can be determined based on the detection result corresponding to at least one frame of image in the current control cycle, that is, the detection result of whether there is someone / no one in the user usage area in the control cycle, and then the working status of the display device can be controlled based on the final detection result.

[0398] In some embodiments, for the detection results corresponding to all images in the current control cycle, if the number of current target users located in the user usage area is large, the final detection result is that the current target user is located in the user usage area; otherwise, the final detection result is that the current target user is not located in the user usage area.

[0399] Therefore, the display device of the present disclosure can determine whether the current target user corresponding to the current image is located in the user usage area based on the current image, that is, determine whether there is someone / no one in the user usage area for the current target user, and then control the working state of the display device based on the someone / no one situation corresponding to at least one frame of image in the current control cycle. It can be seen that in the embodiment of the present disclosure, there is no need for the user to actively input control instructions to the display device through a remote control or voice, etc. to control the working state of the display device. The display device can automatically determine whether there is someone / no one in the user usage area and then control the working state based on the someone / no one situation, making the control of the display device more convenient and intelligent.

[0400] FIG42 is another flow chart of a device control method according to some embodiments. As shown in FIG42 , the method includes the following steps.

[0401] S4201. In response to a user identification instruction, receive the current image acquired by the image collector; this step has the same meaning as the aforementioned step S3601 and will not be repeated here.

[0402] S4202: Perform user identification on the current image to obtain a corresponding user identification result; this step has the same meaning as the aforementioned step S3602 and will not be repeated here.

[0403] S4203: If the user recognition result corresponding to the current image includes body information of at least one user, determine a current target user from the at least one user;

[0404] The limb information includes the position information and confidence information of multiple key limb points, the multiple key limb points including a first target key limb point, and the first target key limb point including left and right shoulder key points and / or left and right span key points. This step has the same meaning as the aforementioned step S3603 and is not further described here. It should be understood that the left and right shoulder key points are the left shoulder key point and the right shoulder key point. The left and right span key points are the left span key point and the right span key point.

[0405] S4204: If the position information of the first target key limb point is not empty and the confidence information is greater than a first preset confidence threshold, determining that the current target user is located in the user usage area;

[0406] The specific value of the first preset confidence threshold can be set by those skilled in the art according to actual conditions and is not limited thereto. For example, the first preset confidence threshold is greater than or equal to 0.5, such as 0.6, 0.7, 0.8, etc.

[0407] If the position information of the first target key limb point is not empty, it indicates that the first target key limb point is within the image acquisition area (i.e., the user usage area), and the confidence information is greater than the first preset confidence threshold, it indicates that the position information of the first target key limb point is relatively reliable; if the position information of the first target key limb point is empty, it indicates that the first target key limb point is not within the image acquisition area (i.e., the user usage area), and the confidence information is less than or equal to the first preset confidence threshold, it indicates that the position information of the first target key limb point is relatively unreliable.

[0408] Of course, in other embodiments, when the first target key limb points include left and right shoulder key points and left and right span key points, the left shoulder key point and the right shoulder key point can also be detected first. If the position information of the left shoulder key point and the right shoulder key point is not empty and the confidence information is greater than the first preset confidence threshold, it is determined that the current target user is located in the user use area. Otherwise, continue to detect the left span key point and the right span key point. If the position information of the left span key point and the right span key point is not empty and the confidence information is greater than the first preset confidence threshold, it is determined that the current target user is located in the user use area.

[0409] Figure 43 is a schematic diagram showing that the left and right shoulder key points and the left and right span key points are located within the user use area according to some embodiments, in which case it is determined that the current target user is located within the user use area. Figure 44 is a schematic diagram showing that the left shoulder key point and the left span key point are located within the user use area according to some embodiments, in which case it may be further detected whether they are located within the user use area. Figure 45 is a schematic diagram showing that the right shoulder key point and the right span key point are located within the user use area according to some embodiments, in which case it may be further detected whether they are located within the user use area. Figure 46 is a schematic diagram showing that the left and right span key points are located within the user use area according to some embodiments, in which case it is determined that the current target user is located within the user use area. Figure 47 is a schematic diagram showing that the left and right shoulder key points are located within the user use area according to some embodiments, in which case it is determined that the current target user is located within the user use area.

[0410] It is understood that the left and right shoulder key points and the left and right span key points are key points on the torso. When the position information of the left and right shoulder key points and / or the left and right span key points is not empty and the confidence information is greater than a first preset confidence threshold, it indicates that the torso of the current target user is located within the user usage area. In this case, the determination that the current target user is located within the user usage area is relatively accurate and has a high degree of reliability. Furthermore, because the left and right shoulder key points and the left and right span key points are key points on the torso, the probability of their position information changing is relatively low, and the robustness is relatively strong. For example, when a user shakes their arms, the position information of the left elbow keypoint, right elbow keypoint, left wrist keypoint, right wrist keypoint, left palm keypoint, and right palm keypoint will change. When a user shakes their legs, the position information of the left knee keypoint, right knee keypoint, left ankle keypoint, and right ankle keypoint will change. When a user shakes their head, the position information of the nose keypoint, left eye keypoint, right eye keypoint, left ear keypoint, and right ear keypoint will change. However, regardless of whether the user shakes their arms, legs, or head, the probability of the position information of the left and right shoulder keypoints and the left and right span keypoints on the torso changing is relatively low. Therefore, detecting whether the current target user is within the user usage area based on the position information of the left and right shoulder keypoints and / or the left and right span keypoints can make the detection result more robust and reduce the risk of sudden changes in the presence or absence of people between two adjacent frames of images.

[0411] In some embodiments, the method further includes: if the position information of at least one first target key limb point is empty, or the confidence information is less than or equal to a first preset confidence threshold, determining that the current target user is not located in the user usage area.

[0412] In some embodiments, if the position information of at least one first target key limb point is empty, or the confidence information is less than or equal to a first preset confidence threshold, the position of the current target user in the current image along the target direction is determined; if the current target user's position is in the middle, it is determined that the current target user is located in the user usage area.

[0413] In some embodiments, the target direction includes a horizontal direction and / or a vertical direction. With respect to the horizontal direction, the current target user's position is either located in the center of the current image or at an edge of the current image (i.e., slightly to the left or right). With respect to the vertical direction, the current target user's position is either located in the center of the current image or at an edge of the current image (i.e., slightly above or below).

[0414] In some embodiments, the position information and confidence information of multiple key limb points of the current target user can be input into a trained position recognition model, and the position recognition model outputs the current target user's position. Next, the width of the current image in the target direction is obtained; based on the position information of the multiple key limb points of the current target user, the position information of the limb center point of the current target user in the target direction is determined; and based on the position information and width of the limb center point, the position of the current target user in the target direction is determined.

[0415] In some embodiments, the position information of the limb center point of the current target user in the target direction can be determined based on the coordinate values ​​of multiple key limb points in the target direction. The maximum and minimum values ​​are selected from the coordinate values ​​of the multiple key limb points in the target direction, and the position information of the limb center point of the current target user in the target direction is determined based on the maximum and minimum values.

[0416] For example, if the target direction includes the horizontal direction, the position information of the limb center point in the horizontal direction can be calculated using the following formula (4):

[0417] body_center x=(maxx-minx) / 2 + minx (4)

[0418] Among them, body_center_x is the position information of the center point of the limb in the horizontal direction, maxx is the maximum value of the coordinate values ​​of multiple key limb points in the horizontal direction, and minx is the minimum value of the coordinate values ​​of multiple key limb points in the vertical direction.

[0419] For another example, if the target direction includes the vertical direction, the position information of the limb center point in the vertical direction can be calculated using the following formula (5):

[0420] body_centery=(maxy-miny) / 2 + miny (5)

[0421] Among them, body_center_y is the position information of the center point of the limb in the vertical direction, maxy is the maximum value of the coordinate values ​​of multiple key limb points in the vertical direction, and miny is the minimum value of the coordinate values ​​of multiple key limb points in the vertical direction.

[0422] In some embodiments, the coordinate range intervals corresponding to the middle and edge in the target direction can be determined based on the width of the current image in the target direction, and then the position of the current target user in the target direction can be determined based on the coordinate range interval in which the position information of the center point of the limb in the target direction falls.

[0423] For example, in the horizontal direction, the coordinate range interval corresponding to the left quarter of the width in the current image is associated with the left side, the coordinate range interval corresponding to the right quarter of the width is associated with the right side, and the coordinate range interval corresponding to the middle half of the width is associated with the center.

[0424] Therefore, if the coordinate system containing the position information of multiple key limb points is as shown in Figure 38, then when body_center x < screen_width / 4, the current target user's horizontal position is determined to be biased to the left; when body_center x > screen_width3 / 4, the current target user's horizontal position is determined to be biased to the right; and when screen_width / 4 < body_center x < screen_width3 / 4, the current target user's horizontal position is determined to be biased to the right. Where screen_width / 4 is the coordinate value corresponding to one-quarter of the width measured from the left, and screen_width3 / 4 is the coordinate value corresponding to three-quarters of the width measured from the left.

[0425] For another example, in the vertical direction, the coordinate range interval corresponding to the upper quarter width of the current image is associated with the upper part, the coordinate range interval corresponding to the lower quarter width is associated with the lower part, and the coordinate range interval corresponding to the middle half width is associated with the middle part.

[0426] Therefore, if the coordinate system containing the position information of multiple key limb points is as shown in Figure 38, then when body_center y < screen_height / 4, the current target user is determined to be standing slightly above the vertical position; when body_center y > screen_height3 / 4, the current target user is determined to be standing slightly below the vertical position; and when screen_height / 4 < body_center y < screen_height 3 / 4, the current target user is determined to be standing slightly below the vertical position. Where screen_height / 4 is the coordinate value corresponding to one-quarter of the width measured from the top, and screen_height 3 / 4 is the coordinate value corresponding to three-quarters of the width measured from the top.

[0427] It can be understood that determining the position of the current target user in the target direction based on the position information and width of the limb center point makes the position determination method simple and convenient, which helps to reduce the difficulty of implementation.

[0428] It can be understood that if the position information of at least one first target key limb point is empty, or the confidence information is less than or equal to the first preset confidence threshold, it indicates that part of the torso of the current target user is not located in the user use area. In this case, by determining the position of the current target user in the target direction to further detect whether the current target user is located in the user use area, the detection result can be more in line with actual needs, more reliable, and reduce the risk of jumps between people and no people in two adjacent frames of images.

[0429] In some embodiments, if the current target user's position is at the edge, it can be determined that the current target user is not in the user usage area. During implementation, if the current target user's position is at the edge, the number of second target key points whose position information is empty and whose confidence information is less than or equal to the second preset confidence threshold is determined, wherein, when the current target user's position is to the left, the second target key points include the left ear key point, the left shoulder key point, the left hip key point, the left knee key point, and / or the left ankle key point; when the current target user's position is to the right, the second target key points include the right ear key point, the right shoulder key point, the right hip key point, the right knee key point. , and / or right ankle key points; when the current target user's standing position is slightly higher, the second target key limb points include the left ear key point, the right ear key point, the left shoulder key point, and / or the right shoulder key point; when the current target user's standing position is slightly lower, the second target key limb points include the left hip key point, the right hip key point, the left knee key point, the right knee key point, the left ankle key point, and / or the right ankle key point; if the number is greater than the preset number threshold, it is determined that the current target user is located in the user use area, otherwise, it is determined that the current target user is not located in the user use area.

[0430] The specific value of the second preset confidence threshold can be set by those skilled in the art according to actual conditions and is not limited thereto. For example, the second preset confidence threshold is greater than or equal to 0.5, such as 0.6, 0.7, 0.8, etc.

[0431] In some embodiments, the specific value of the preset number threshold can be set by those skilled in the art according to actual conditions and is not limited thereto. For example, the preset number threshold is greater than or equal to half of the number of the second target key limb points.

[0432] It can be understood that when the current target user is standing at the edge, further detecting whether the current target user is located in the user usage area based on the position information of the second target key limb point can make the detection result more in line with actual needs, more reliable, and reduce the risk of jumps between two adjacent frames of images.

[0433] S4205. If the current image is the last frame image in the current control cycle, the working state of the display device is controlled based on the detection result corresponding to at least one frame image in the current control cycle; this step has the same meaning as the aforementioned step S3605 and will not be repeated here.

[0434] In the embodiment of the present disclosure, the image captured by the image collector has edge characteristics, that is, the current target user may be located at the edge of the image, which affects the robustness of the detection of whether the current target user is located in the user use area, and may further cause the working state of the display device to change frequently when the current target user is in the edge scene. For example, when controlling the broadcast control of a smart TV, the broadcast control may start and stop and flicker. However, in the embodiment of the present disclosure, the display device can determine that the current target user is located in the user use area when the position information of the first target key limb point is not empty and the confidence information is greater than the first preset confidence threshold, so that the robustness of the detection result is strong and the risk of jumping between people and no people in two adjacent frames of images is reduced. Moreover, when the position information of at least one first target key limb point is empty, or the confidence information is less than or equal to the first preset confidence threshold, the display device can further detect whether the current target user is located in the user use area based on the position information and confidence information of the second target key limb point, which can further enhance the robustness of the detection result.

[0435] Figure 48 is a flowchart of another device control method according to some embodiments. The embodiment of the present disclosure is optimized based on the above embodiment, and the embodiment of the present disclosure can be combined with various optional solutions in one or more of the above embodiments.

[0436] As shown in FIG. 48 , the device control method may include the following steps.

[0437] S4801. In response to the user identification instruction, receive the current image acquired by the image collector; this step has the same meaning as the aforementioned step S3601 and will not be repeated here.

[0438] S4802: Perform user identification on the current image to obtain a corresponding user identification result; this step has the same meaning as the aforementioned step S3602 and will not be repeated here.

[0439] S4803. If the user recognition result corresponding to the current image includes body information of at least one user, determine the current target user from the at least one user; this step has the same meaning as the aforementioned step S3603 and will not be repeated here.

[0440] S4804. Detect whether the current target user is located in the user usage area of ​​the display device based on the body information of the current target user, and obtain a corresponding detection result; this step has the same meaning as the aforementioned step S3604 and will not be repeated here.

[0441] S4805: If the current image is the last image frame in the current control cycle, and the detection results corresponding to the current image and the images N frames before it are the same, then the detection result corresponding to the current image is determined as the final detection result, where N is a positive integer and is less than the total number of images in the current control cycle;

[0442] In some embodiments, the specific value of N can be set by those skilled in the art according to actual conditions and is not limited here. For example, N is 2, 3, etc.

[0443] In some embodiments, if the detection results corresponding to the current image and its previous N frames of images are all that the current target user is located in the user usage area, the final detection result is determined as the current target user is located in the user usage area; if the detection results corresponding to the current image and its previous N frames of images are all that the current target user is not located in the user usage area, the final detection result is determined as the current target user is not located in the user usage area.

[0444] It can be understood that if the current image is the last frame image in the current control cycle, then the acquisition time of the current image and the previous N frames of images are closer to the current moment, and their corresponding detection results can better reflect the actual situation of whether the current target user is located in the user usage area. When their corresponding detection results are the same, their corresponding detection results are used as the final detection results, which can not only quickly determine the final detection results, but also make the final detection results more in line with the current actual situation.

[0445] In some embodiments, if there are at least two images with different detection results between the current image and the N previous image frames, the final detection result is determined based on the detection results corresponding to all images in the current control cycle.

[0446] It can be understood that by setting that if the detection results corresponding to the current image and at least two frames of the previous N frames are different, the final detection result is determined based on the detection results corresponding to all images in the current control cycle, and the final detection result can be determined based on the detection results corresponding to more frames of images, thereby making the final detection result more accurate.

[0447] In some embodiments, the final detection result is determined based on the detection results corresponding to all images in the current control cycle, including: for each frame image in the current control cycle, if the corresponding detection result is that it is located within the user use area, then mark a first value; if the corresponding detection result is that it is not located within the user use area, then mark a second value; weighted sum the values ​​marked for each frame image in the current control cycle to obtain a sum value; if the sum value is greater than a preset sum threshold, then determine that the final detection result is located within the user use area, otherwise determine that the final detection result is not located within the user use area.

[0448] In some embodiments, the first value is greater than the second value, for example, the first value is 1 and the second value is 0.

[0449] In some embodiments, when performing weighted summation, the earlier the image is captured, the smaller the weight of the corresponding value. It is understood that the earlier the image is captured, the further away it is from the current moment, and the less likely the corresponding detection result is to be the same as the current actual situation. Therefore, by setting the earlier the image is captured, the smaller the weight of the corresponding value can be, so that the detection result that better reflects the current actual situation can have a greater impact on the final detection result, which is conducive to making the final detection result closer to the current actual situation.

[0450] It should be noted that the statement “the earlier the image is acquired, the smaller the weight of the corresponding numerical value” includes the following two situations. The first is that the weight values ​​of the numerical values ​​corresponding to the frame images in the same control cycle are different. For example, the same control cycle includes 10 frames of images, the weight value of the numerical value corresponding to the 10th frame image (i.e., the current image) is 1.0, the weight value of the numerical value corresponding to the 9th frame image is 0.9, the weight value of the numerical value corresponding to the 8th frame image is 0.8, the weight value of the numerical value corresponding to the 7th frame image is 0.7, the weight value of the numerical value corresponding to the 6th frame image is 0.6, the weight value of the numerical value corresponding to the 5th frame image is 0.5, the weight value of the numerical value corresponding to the 4th frame image is 0.4, the weight value of the numerical value corresponding to the 3rd frame image is 0.3, the weight value of the numerical value corresponding to the 2nd frame image is 0.2, and the weight value of the numerical value corresponding to the 1st frame image is 0.1. The second type: within the same control cycle, the weight values ​​of the numerical values ​​corresponding to two frame images are the same. For example, within the same control cycle, there are 10 frame images, and the weight values ​​of the numerical values ​​corresponding to the 10th frame image (i.e., the current image), the 9th frame image, and the 8th frame image are 1.0, the weight values ​​of the numerical values ​​corresponding to the 7th frame image, the 6th frame image, the 5th frame image, and the 4th frame image are 0.5, and the weight values ​​of the numerical values ​​corresponding to the 3rd frame image, the 2nd frame image, and the 1st frame image are 0.2.

[0451] The specific value of the preset sum threshold can be set by those skilled in the art according to actual conditions and is not limited here.

[0452] S4806: Control the working state of the display device based on the final detection result. This step has the same meaning as the aforementioned step S3605 and will not be repeated here.

[0453] Figure 49 is a schematic diagram of a system architecture according to some embodiments. Referring to Figure 49, the system architecture may include an application layer, an SDK layer, a Framework layer, and a system layer.

[0454] In some embodiments, the application layer corresponds to a fitness application and may include an image acquisition module, an interface call module, a result receiving module, a scoring module, an interface display module, a screen recording module, a timing module, a storage module, and a playback module.

[0455] In some embodiments, the SDK layer includes a limb recognition module, which may include an image processing submodule, an image recognition submodule, and a result processing submodule.

[0456] In some embodiments, the image acquisition module is used to collect the user's follow-up training images in real time, the interface calling module is used to call the SDK layer interface to pass the follow-up training images to the SDK layer, the SDK layer inputs the follow-up training images to the limb recognition module, the image processing submodule is used to process the follow-up training images (such as denoising, enhancement, etc.), the image recognition submodule is used to identify the limb key points in the follow-up training images to obtain recognition results, which include the position coordinates of each limb key point in the follow-up training action (i.e., the second limb key point position data), and the result processing submodule sends the identified second limb key point position data to the result receiving module of the application layer.

[0457] In some embodiments, the result receiving module sends the second limb key point position data to the scoring module, the scoring module obtains the first limb key point position data from the action template data, and compares the first limb key point position data with the second limb key point position data, calculates the action matching degree, and calculates the training score corresponding to each frame of the follow-up image based on the action matching degree and the benchmark score, and then periodically obtains a frame of follow-up image, accumulates the training scores corresponding to multiple frames of follow-up images, and obtains the action group follow-up score.

[0458] In some embodiments, the interface display module is used to display the follow-up training interface. The follow-up training interface can refer to the example of Figure 13. The second window 132 on the follow-up training interface displays the dynamically updated action group follow-up training score, and the third window 134 uses the action prompt control 134a to prompt the final action group follow-up training score of the standard action in the follow-up training state. In some embodiments, when the training project video ends, the interface display module is also used to display the follow-up training details list page. The follow-up training details list page can refer to the example of Figure 16 or Figure 18.

[0459] In some embodiments, the screen recording module is used to record a complete screen recording file or to record in segments according to the action group segments. The timing module is used to time and obtain the action group recording start time Ti1 and the action group recording end time Ti2. The specific functions of the screen recording module and the timing module can be referred to in the previous related embodiments and will not be repeated here. The screen recording module sends the recorded video file and the related action group recording information to the storage module so that the storage module saves the recorded video file and the action group recording information.

[0460] In some embodiments, the playback module is configured to retrieve the matching recorded video file and action group recording information from the storage module upon receiving a user click on the playback control 181 in the practice details list page, and then control the display module to display the practice segment that the user wishes to review on the practice playback interface. For details, please refer to the previous related embodiments and will not be repeated here. In some embodiments, the Framework layer and the system layer can be configured and operated according to the system architecture of the display device (e.g., the Android system), which is not limited or specifically described in the present embodiment.

[0461] The functional improvements to the system architecture in this disclosure are mainly focused on the application layer and the interaction between the application layer and the SDK layer. The UI drawings provided in this disclosure are merely exemplary. Based on the implementation effect of this solution, the layout and content configuration and changes of the UI such as the follow-up training interface, follow-up training playback interface, and follow-up training details list page can be made. The recognition of key points of the limbs and the calculation of the degree of matching of the movements in this disclosure can be implemented with reference to related technologies, and the embodiments of this disclosure will not be repeated here.

[0462] In some embodiments, the present disclosure further provides a computer storage medium that can store a program. When the computer storage medium is configured in the display device 200, the program, when executed, can include the program steps included in the device control method in the above embodiment. The computer storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

Claims

1. A display device, comprising: An image collector is configured to collect the user's follow-up training images; A display, configured to display a follow-up training interface; A user input interface configured to receive instructions from a user; a memory configured to store computer instructions; At least one processor is connected to the image acquisition device, the display, the user input interface and the memory, and is configured to execute computer instructions to cause the display device to perform: In response to the operation of starting a training project video, an action group list associated with the training project video is obtained; wherein the training project video contains an action group segment corresponding to at least one standard action, the action group segment includes a continuous multi-frame reference action image corresponding to the standard action, and the action group list includes action information and action start and end time of the standard action; Control the display to display the training item video in the first window of the training interface, and to display the training image in the second window of the training interface; When it is detected that the training project video is played to the first action group segment corresponding to the first standard action, based on the alignment mechanism of the playback timestamp of the reference action image and the acquisition timestamp of the follow-up training image, the reference action image included in the first action group segment is compared with the aligned first follow-up training image, and the training score corresponding to the first follow-up training image is calculated; Accumulate the training scores corresponding to the multiple frames of the first follow-up training images to obtain and store the first follow-up training score corresponding to the first standard action; Control the display to display the first follow-up training score on the follow-up training interface.

2. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Controlling the display to display the action prompt controls corresponding to the standard actions included in the training program video in a preset area of ​​the training interface; Controlling the action prompt control to mark the training status of the standard action, wherein the training status includes a trained state, a training state, and a waiting training state; Obtain a target action prompt control corresponding to a target standard action in a practiced state, and control the target action prompt control to display an action group practice score corresponding to the target standard action.

3. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: When the training program video ends, the display is controlled to display a training details list page, which includes training details information of the standard movements, including movement information, exercise duration, energy consumption and movement group training scores.

4. The display device according to claim 3, wherein the training details list page further includes a retraining control corresponding to each of the standard actions, and the processor is further configured to execute: In response to a user clicking on a retraining control corresponding to the first standard action, controlling the display to display the first action group segment in a first window of the follow-up training interface, and displaying a follow-up training image corresponding to the first standard action in a second window of the follow-up training interface; When the first action group segment ends playing, recalculating the first follow-up score; The first follow-up training score is updated in the follow-up training details list page.

5. The display device according to claim 3 or 4, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: When it is detected that the training program video is played to the first action group segment, start recording the screen content; When the first action group clip ends playing, stop recording the screen content; The screen recording file of the first action group segment and the first action group recording information are stored in a preset storage location, wherein the first action group recording information includes the action information of the first standard action, the first follow-up score, the first action group recording start time and the first action group recording end time.

6. The display device according to claim 5, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: When the first window of the training interface starts to display the training item video, starting a timer; When it is detected that the training item video is played to the first action group segment, setting the first time recorded by the timer as the start time of recording the first action group; When the playback of the first action group segment ends, the second time recorded by the timer is set as the end time of recording the first action group.

7. The display device according to claim 3 or 4, wherein the at least one processor is further configured to execute computer instructions to cause the display device to execute: When the training program video is played, start recording the screen content and start the timer; When it is detected that the training item video is played to any action group segment, the first time recorded by the timer is set as the action group recording start time; When the playback of the standard action group ends, setting the second time recorded by the timer as the action group recording end time; When the training program video is finished playing, the screen content is stopped from being recorded, and a complete screen recording file is generated; wherein, The complete screen recording file includes at least one recording segment, and the recording segment records the user's practice process of a standard action; The complete screen recording file is stored, and the action group recording information associated with the recording segment is stored, wherein the action group recording information includes the action information of the standard action, the action group follow-up training score, the action group recording start time and the action group recording end time.

8. The display device according to claim 7, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a user clicking on a playback control corresponding to the first standard action, acquiring the complete screen recording file and first action group recording information corresponding to the first standard action from the preset storage location, wherein the first action group recording information includes a first action group recording start time and a first action group recording end time; According to the recording start time of the first action group and the recording end time of the first action group, searching the first recording segment corresponding to the first standard action from the complete screen recording file; When the complete screen recording file is started, the playback progress of the complete screen recording file is jumped to the start time of recording of the first action group, so that the follow-up playback interface plays the first recording segment.

9. The display device according to claim 7, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: After the complete screen recording file is generated, according to the action group recording start time and the action group recording end time, Splitting the complete screen recording file into a plurality of recording segments; independently storing a video file of the recorded segment and recording information of the action group associated with the recorded segment; In response to the user clicking on the playback control corresponding to the first standard action, the follow-up playback interface is controlled to play the first recorded segment corresponding to the first standard action.

10. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device calculates the training score corresponding to the first training image in the following manner: According to the action information of the first standard action, obtaining a first benchmark score and first limb key point position data of the first standard action from preset action template data; Acquire the key point position data of the second limb of the following training action contained in the first following training image; Calculating the action matching degree between the first limb key point position data and the second limb key point position data; A training score corresponding to the first training image is calculated according to the action matching degree and the first benchmark score.

11. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: According to a preset period, the reference action image is periodically acquired from the first action group segment, and the first follow-up training image is periodically acquired from the video code stream acquired by the image acquirer.

12. The display device according to claim 1, wherein the memory is further configured to: store a follow-up training video and a data file, wherein the follow-up training video includes at least one follow-up training action group; the data file includes group information of each of the follow-up training action groups, and the group information includes at least a second follow-up training score of the follow-up training action group; The at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a playback instruction from a user, generating a first progress bar of the follow-up video according to the group information of each of the follow-up action groups, wherein the first progress bar is used to display a score level of each of the follow-up action groups; Control the display to display the playback interface, where: The follow-up video is played in the playback interface, and the first progress bar is displayed.

13. The display device according to claim 12, wherein the first progress bar includes at least one first time period, each of the first time period corresponds to one of the following exercise groups; each of the first time period has a first identifier, and the first identifier is used to indicate a score level of the following exercise group corresponding to the first time period; The at least one processor is further configured to execute computer instructions to cause the display device to perform: The second follow-up score of the played follow-up action group is displayed on the playback interface.

14. The display device according to claim 13, wherein the group information of the follow-up exercise group further includes a start time and an end time of the follow-up exercise group, and a standard score of the follow-up exercise group; The at least one processor is specifically configured to execute computer instructions so that the display device generates a first progress bar of the follow-up video according to the group information of each follow-up action group in the following manner: The length of the first time period corresponding to the following training action group on the first progress bar is calculated according to the total duration of the following training video, the start time and the end time of the following training action group, wherein: If each of the first time periods carries the first identifier, calculating the score level of the action group according to the second follow-up score of the follow-up action group and the standard score of the follow-up action group; The first progress bar is generated according to the length of the first time period corresponding to each of the follow-up action groups on the first progress bar and the score level of each of the follow-up action groups.

15. The display device according to claim 12, wherein the at least one processor is specifically configured to execute computer instructions so that the display device displays the playback interface in the following manner: Determining the playback speed of each of the following practice action groups in the following practice video according to a preset correspondence between the score level and the playback speed; The first time period corresponding to the corresponding follow-up practice action group in the follow-up practice video is played at the playback speed of each follow-up practice action group.

16. The display device according to claim 12, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: After the playback interface is displayed, in response to a jump instruction from the user, the focus is moved to the first progress on the first progress bar, wherein: The focus moves to the starting position of each of the first time periods in the first progress bar; The follow-up training video is played starting from the first progress.

17. The display device according to claim 12, wherein the at least one processor is specifically configured to execute computer instructions so that the display device displays the playback interface in the following manner: Displaying buttons on the playback interface, the buttons indicating corresponding score levels; In response to an instruction of a user selecting a designated button, obtaining a waiting time period in the training video, wherein the waiting time period is the first time period of the training action group corresponding to the score level indicated by the designated button; The waiting time period is played continuously.

18. The display device according to claim 17, wherein the at least one processor is specifically configured to execute computer instructions so that the display device continuously plays the waiting time period in the following manner: If the user selects multiple designated buttons, the waiting time periods are played in sequence according to the order in which they are recorded in the follow-up training video; Alternatively, the waiting time period in each group is played in sequence according to the group, and the group includes the waiting time period corresponding to the score level indicated by a designated button, wherein: In each of the groups, the time periods to be played are played in sequence according to the order in which they were recorded in the follow-up video.

19. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to the user identification indication, receiving a current image acquired by the image acquirer; Performing user recognition on the current image to obtain a corresponding user recognition result; If the user recognition result corresponding to the current image includes body information of at least one user, determining a current target user from the at least one user; Detecting whether the current target user is located in the user use area of ​​the terminal device based on the body information of the current target user, and obtaining a corresponding detection result; If the current image is the last frame image in the current control cycle, the working state of the terminal device is controlled based on the detection result corresponding to at least one frame image in the current control cycle.

20. The display device according to claim 19, wherein the at least one processor is specifically configured to execute computer instructions so that the display device determines the current target user from the at least one user in the following manner: Performing gesture recognition on the current image to obtain a corresponding gesture recognition result; If the gesture recognition result corresponding to the current image includes a switching gesture, determining the user who makes the switching gesture among the at least one user as the current target user; If the gesture recognition result does not include a switching gesture, the historical target user is identified with at least A user is matched, wherein the historical target user is a target user determined based on the previous frame image; If there is a user matching the historical target user among the at least one user, taking the historical target user as the current target user; If there is no user matching the historical target user among the at least one user, a user meeting a preset condition is selected from the at least one user as the current target user.

21. The display device according to claim 19, wherein the limb information comprises position information and confidence information of a plurality of key limb points, wherein the plurality of key limb points comprises a first target key limb point, wherein the first target key limb point comprises left and right shoulder key points, and / or left and right span key points; The at least one processor is specifically configured to execute computer instructions so that the display device detects whether the current target user is located in the user use area of ​​the terminal device based on the body information of the current target user in the following manner to obtain a corresponding detection result: If the position information of the first target key limb point is not empty and the confidence information is greater than a first preset confidence threshold, it is determined that the current target user is located in the user use area.

22. The display device according to claim 21, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: If the position information of at least one of the first target key limb points is empty, or the confidence information is less than or equal to the first preset confidence threshold, determining the position of the current target user in the current image along the target direction; If the current target user's station is located in the middle, it is determined that the current target user is located in the user use area.

23. The display device according to claim 22, wherein the plurality of key limb points include a second target key limb point; and the at least one processor is further configured to execute computer instructions to cause the display device to execute: If the current target user is located at the edge, determining the number of second target key points whose position information is empty and whose confidence information is less than or equal to a second preset confidence threshold in the second target key limb points; in, When the current target user's standing position is biased to the left, the second target key limb points include a left ear key point, a left shoulder key point, a left hip key point, a left knee key point, and / or a left ankle key point; when the current target user's standing position is biased to the right, the second target key limb points include a right ear key point, a right shoulder key point, a right hip key point, a right knee key point, and / or a right ankle key point; When the current target user's standing position is upward, the second target key limb points include a left ear key point, a right ear key point, a left shoulder key point, and / or a right shoulder key point; When the current target user's standing position is lower, the second target key limb points include a left span key point, a right span key point, a left knee key point, a right knee key point, a left ankle key point, and / or a right ankle key point; If the number is greater than a preset number threshold, it is determined that the current target user is located in the user use area; otherwise, it is determined that the current target user is not located in the user use area.

24. The display device according to claim 22, wherein the at least one processor is specifically configured to execute computer instructions so that the display device determines the position of the current target user in the target direction in the following manner: Obtaining the width of the current image in the target direction; Determine the position information of the limb center point of the current target user in the target direction based on the position information of multiple key limb points of the current target user; The position of the current target user in the target direction is determined based on the position information of the limb center point and the width.

25. The display device according to claim 19, wherein the at least one processor is specifically configured to execute computer instructions so that the display device controls the working state of the terminal device based on the detection result corresponding to at least one frame of image in the current control period in the following manner: If the detection result corresponding to the current image is the same as the detection result corresponding to the previous N frames of images, the detection result corresponding to the current image is determined as the final detection result, wherein, N is a positive integer and is less than the total number of images in the current control cycle; The working state of the terminal device is controlled based on the final detection result.

26. The display device according to claim 25, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: If there are at least two images with different detection results between the current image and the previous N frames of images, the final detection result is determined based on the detection results corresponding to all images in the current control cycle.

27. The display device according to claim 26, wherein the at least one processor is specifically configured to execute computer instructions so that the display device determines the final detection result based on the detection results corresponding to all images in the current control cycle in the following manner: For each frame of image in the current control cycle, if the corresponding detection result is that it is located in the user use area, a first value is marked; if the corresponding detection result is that it is not located in the user use area, a second value is marked; Perform weighted summation on the values ​​of the image marks of each frame in the current control cycle to obtain a sum value; If the sum value is greater than a preset sum threshold, the final detection result is determined to be located within the user usage area; otherwise, the final detection result is determined to be not located within the user usage area.

28. A device control method, comprising: In response to the operation of starting a training project video, an action group list associated with the training project video is obtained; wherein the training project video contains an action group segment corresponding to at least one standard action, the action group segment includes a continuous multi-frame reference action image corresponding to the standard action, and the action group list includes action information and action start and end time of the standard action; Displaying the training project video in a first window of the training interface, and displaying the training image collected by the image collector in a second window of the training interface; When it is detected that the training project video is played to the first action group segment corresponding to the first standard action, based on the alignment mechanism of the playback timestamp of the reference action image and the acquisition timestamp of the follow-up training image, the reference action image included in the first action group segment is compared with the aligned first follow-up training image, and the training score corresponding to the first follow-up training image is calculated; The training scores corresponding to the multiple frames of the first follow-up training images are accumulated to obtain and store a first follow-up training score corresponding to the first standard action; and the first follow-up training score is displayed on the follow-up training interface.

Images