Display device, terminal device and screen connection action score display method

Abstract

The embodiment of the application provides a display device, a terminal device and a screen connection action score display method. The method receives user action data collected by the terminal device in response to a starting instruction, and the user action data comprises a plurality of action frames with frame time stamps. After a delay duration, standard action video frames played are captured according to a collection frame rate to obtain standard action data. The delay duration is obtained according to a first system time of the current display device and a second system time of the terminal device, and the collection frame rate is a sampling frequency when the terminal device collects the user action data. Then, a target frame is found according to the frame time stamp, and an action score is generated and displayed according to the similarity between the action frame and the target frame. According to the display device and the terminal device, the delay duration is calculated to synchronize the time reference of the display device and the terminal device, so that the display device and the terminal device compare the pictures of the same time stamp, and the accuracy of the action score display is improved.

Description

Technical Field

[0001] This application relates to the field of terminal interconnection technology, and in particular to a display device, a terminal device, and a method for displaying screen-to-screen action scoring. Background Technology

[0002] Display devices refer to terminal devices capable of outputting specific display images, such as smart TVs, communication terminals, smart advertising screens, and projectors. Taking smart TVs as an example, smart TVs are television products based on Internet application technologies, possessing open operating systems and chips, and having open application platforms. They enable two-way human-computer interaction and integrate multiple functions such as audio-visual, entertainment, and data to meet diverse and personalized user needs.

[0003] Based on the aforementioned human-computer interaction functions, users can also use display devices to assist in exercise. For example, they can use display devices to correct fitness movements to achieve the goal of improving physical fitness. The display device can acquire videos of the user's fitness movements by establishing a data transmission channel with an image acquisition device, and then compare and score them with the instructional videos played on the display device. The scoring results are used to correct the user's fitness movements. However, there is a certain time consumption during the transmission of fitness movement videos from the image acquisition device to the display device. This causes the display device to compare the user's current movements with the movements performed at a previous time point in the instructional video, resulting in a significant deviation between the user's fitness movements and the instructional movements. This leads to inaccurate scoring, resulting in inaccurate displayed continuous movement scores and a reduced user experience. Summary of the Invention

[0004] This application provides a display device, a terminal device, and a method for displaying motion scores across multiple screens, in order to solve the problem of inaccurate display of motion scores across multiple screens during the transmission of fitness motion videos by an image acquisition device.

[0005] In a first aspect, some embodiments of this application provide a display device, including a display, a communicator, and a controller, wherein the communicator is configured to establish a communication connection with a terminal device, the terminal device being used to collect user action data, and the controller is configured to:

[0006] In response to the start command, the terminal device collects user action data, which includes several action frames with frame timestamps.

[0007] After a delay, standard motion video frames corresponding to the user motion data are captured according to the acquisition frame rate to obtain standard motion data; the delay is calculated based on the first system time of the current display device and the second system time of the terminal device; the acquisition frame rate is the sampling frequency when the terminal device acquires user motion data.

[0008] The target frame is located in the standard motion data based on the frame timestamp.

[0009] The similarity between the action frame and the target frame is calculated, and an action score is generated based on the similarity. The similarity is calculated based on the deviation between the skeletal point coordinates in the action frame and the skeletal point coordinates in the target frame.

[0010] Secondly, some embodiments of this application provide a terminal device, including a camera, a communication port, and a controller, wherein the camera is configured to collect user action data; the communication port is configured to establish a communication connection with the display device described in any of the first aspects; and the controller is configured to:

[0011] In response to the acquisition signal sent by the display device, the camera is controlled to acquire user action data, which includes a number of action frames with frame timestamps.

[0012] The user action data is sent to the display device through the communication port, so that the display device can find the target frame in the standard action data according to the frame timestamp, and generate an action score based on the similarity between the action frame and the target frame. The similarity is calculated based on the deviation between the skeletal point coordinates in the action frame and the skeletal point coordinates in the target frame.

[0013] Thirdly, some embodiments of this application provide a method for displaying screen-based action scoring. The method is applied to a display device, which includes a display, a communicator, and a controller. The communicator is configured to establish a communication connection with a terminal device, which is used to collect user action data. The method includes:

[0014] In response to the start command, the terminal device collects user action data, which includes several action frames with frame timestamps.

[0015] After a delay, standard motion video frames corresponding to the user motion data are captured according to the acquisition frame rate to obtain standard motion data; the delay is calculated based on the first system time of the current display device and the second system time of the terminal device; the acquisition frame rate is the sampling frequency when the terminal device acquires user motion data.

[0016] The target frame is located in the standard motion data based on the frame timestamp.

[0017] The similarity between the action frame and the target frame is calculated, and an action score is generated based on the similarity. The action score is then displayed on the monitor. The similarity is calculated based on the deviation between the skeletal point coordinates in the action frame and the skeletal point coordinates in the target frame.

[0018] As can be seen from the above solutions, this application provides a display device, a terminal device, and a method for displaying action scores across multiple screens. The method responds to a start command by receiving user action data collected by the terminal device. The user action data includes several action frames with frame timestamps. After a delay, standard action video frames corresponding to the user action data are captured based on the collected frame rate to obtain standard action data. The delay duration is calculated based on the first system time of the current display device and the second system time of the terminal device. The collected frame rate is the sampling frequency when the terminal device collects user action data. Then, the target frame is searched based on the frame timestamp, and an action score is generated and displayed based on the similarity between the action frame and the target frame. This application can calculate the delay duration based on the display device and the terminal device to synchronize the time base of the display device and the terminal device through the delay duration, enabling the display device and the terminal device to compare images at the same timestamp, thus improving the accuracy of the action score display. Attached Figure Description

[0019] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This application illustrates the usage scenario of the display device.

[0021] Figure 2 This is a hardware configuration diagram of the display device in the embodiments of this application;

[0022] Figure 3 A diagram illustrating the time delay for uploading user motion data to the image acquisition device;

[0023] Figure 4 A flowchart illustrating a method for displaying screen-based action scoring in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a first embodiment of the display device calculating the delay duration in this application.

[0025] Figure 6 This is a schematic diagram of a second embodiment of the display device calculating the delay duration in this application.

[0026] Figure 7This is a schematic diagram of the joint location identified by the joint feature recognition algorithm in an embodiment of this application.

[0027] Figure 8 This is a schematic diagram illustrating the generation of a segment point array based on skeletal point coordinates in an embodiment of this application.

[0028] Figure 9 This is a schematic diagram showing the coordinates of skeletal points in an embodiment of this application;

[0029] Figure 10 This is a flowchart illustrating the calculation of coordinate deviation based on skeletal point coordinates in an embodiment of this application.

[0030] Figure 11 This is a flowchart illustrating the process of determining redundant data in an embodiment of this application;

[0031] Figure 12 This is a schematic diagram illustrating the process of deleting redundant data in the display device in an embodiment of this application. Detailed Implementation

[0032] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0033] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0034] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0035] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0036] The display device provided in this application can have various implementation forms, such as a television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc.

[0037] Figure 1 This is a schematic diagram illustrating the operational scenario between the display device and the control unit according to the embodiment. Figure 1 As shown, the user can operate the display device 200 through the control device 300 or the control unit 100.

[0038] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device 200 includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0039] In some embodiments, a control device 300 (such as a mobile phone, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the control device 300 may be used to control the display device 200.

[0040] In some embodiments, the display device 200 may receive instructions not by using the control device 300 or control unit 100 described above, but by receiving user control through touch or gestures.

[0041] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the control device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0042] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0043] like Figure 2 As shown, the display device 200 may include at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0044] In some embodiments, controller 250 may include a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to an nth interface for input / output.

[0045] The display 260 may include the following components: a display screen component for displaying images; a driving component for driving image display; a component for receiving image signals output from the controller 250 and displaying video content, image content, and a menu control interface, as well as a user control UI interface, etc.

[0046] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0047] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0048] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote control).

[0049] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0050] The external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0051] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. In some embodiments, the controller 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0052] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0053] In some embodiments, the controller 250 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0054] Users can input commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0055] A user interface (GUI) is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0056] In some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Layer"), and the kernel layer.

[0057] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0058] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0059] In some embodiments, the external device interface 240 may also be connected to an image acquisition device, which may be a camera or a terminal device with video recording capabilities, such as a smartphone, smart TV, or tablet computer. The image acquisition device can be used to capture images of the external environment, record videos, etc.

[0060] In some embodiments, based on Internet application technology, the display device 200 may also have an open operating system and chip, possessing an open application platform, and enabling two-way human-computer interaction. Taking a smart TV as an example, the display device 200 can be a television product integrating multiple functions such as audio-visual, entertainment, and data, to meet the diverse and personalized needs of users.

[0061] Based on the aforementioned human-computer interaction functions, users can also use the display device 200 to assist with exercise. For example, the display device can be used to correct fitness movements, enabling users to perform the movements correctly and achieve the goal of improving physical fitness. The display device 200 can establish a data transmission channel with an image acquisition device through the external device interface 240 to acquire the user's fitness movement video, or it can use its built-in image acquisition interface as an image acquisition device to acquire the user's fitness movement video. After acquiring the fitness movement video, the controller 250 can compare and score the fitness movement video with the fitness instruction video played on the display device 200. Users can then correct their own fitness movements based on the score results.

[0062] To facilitate comparison between user fitness movements and standard fitness movements, the display device 200, after acquiring the user's fitness movement video, divides it into at least two playback areas. To distinguish between the two playback areas, some embodiments of this application define the two playback areas as a first playback area and a second playback area, respectively. The controller 250 can control the display 260 to play the fitness movement video acquired by the image acquisition device in the first playback area, and control the display 260 to play the instructional fitness video in the second playback area, so as to simultaneously demonstrate the fitness movement video and the instructional fitness video, facilitating comparison of fitness movement deviations.

[0063] In some embodiments, the display device 200 can simultaneously compare the fitness movements of two or more users. In this case, the controller 250 can divide the display into N+1 playback areas, where N is the number of users that need to be compared at the same time, so that the display 260 can simultaneously display instructional fitness videos and fitness movement videos of multiple users.

[0064] Before the display device 200 acquires the fitness exercise video, the controller 250 also needs to acquire the instructional fitness video. In some embodiments, the controller 250 can connect to the Internet via a server and download the instructional fitness video from the Internet according to the corresponding URL link. The controller 250 can also traverse the display device 200 to acquire the software program inside the image acquisition device. For example, when the image acquisition device is a terminal device, the terminal device can have a software program with fitness functions installed inside, which includes instructional fitness videos. The controller 250 can establish a communication connection with the smart terminal through the communicator 220 to establish a data transmission channel, and the terminal device can transmit the instructional fitness video to the display device 200 through the data transmission channel.

[0065] In some embodiments, the display device 200 can also compare and analyze the user's fitness movements based on the scoring results, select fitness movements with large deviations, and provide the user with feedback information on these fitness movements and corresponding correction suggestions. The correction suggestions may include the angle and direction of the user's body extension, the movement trend of the skeletal joints from the standard fitness movements, etc.

[0066] However, during the aforementioned process, there is a certain delay in the image acquisition device transmitting the fitness movement video to the display device 200, such as the time consumption for uploading the fitness movement video. This causes the display device 200 to compare the user's current actions with the actions performed at a previous time point in the instructional video. For example, if... Figure 3As shown, when the time delay of the image acquisition device is 0.2 seconds, the display device 200 plays an instructional fitness video while simultaneously controlling the image acquisition device to acquire the user's fitness video. However, due to the time delay, the accuracy of the scoring during the continuous screen display process is affected. For example, when the image acquisition device uploads the first frame of the user's fitness video to the display device 200, the instructional fitness video has already played for 0.2 seconds. The display device 200 will compare the first frame of the user's fitness video with the instructional fitness video 0.2 seconds later, resulting in a large deviation between the user's fitness movements and the instructional movements, inaccurate scoring, and a reduced user experience.

[0067] To address the issue of inaccurate display of motion scores during the transmission of fitness video by an image acquisition device, some embodiments of this application provide a display device 200 capable of receiving user motion data acquired by the image acquisition device. This user motion data can be a video of a user performing a specified motion or an image of a user performing a specified motion. In this embodiment, a terminal device is used as an example image acquisition device. The display device 200 should include at least a display 260, a communicator 220, and a controller 250. The communicator 220 can establish a communication connection with the terminal device to receive the user motion data acquired by the terminal device. The controller 250 is configured to execute a method for displaying motion scores across multiple screens, such as... Figure 4 As shown, it includes the following:

[0068] S100: In response to the start command, it receives user action data collected by the terminal device.

[0069] Users can generate start commands through the control device 100. For example, when a user is using an application with fitness functions, the controller 250 can control the display 260 to show the corresponding user interface. The user interface may include start options such as "Start Exercise". Users can move the focus of the option to "Start Exercise" through the control device 100 and click the "OK" button on the control device 100 to generate a start command.

[0070] In some embodiments, for a touch-enabled display device 200, the display device 200 can also listen for touch events on the display 260. When a user touches the corresponding area of ​​"Start Movement" in the user interface, the display device 200 can generate a start command based on the listened touch event.

[0071] After generating a start command, the controller 250 can respond to the start command by collecting user action data through a terminal device that has established a communication connection with the display device 200. The user action data includes several action frames with frame timestamps, where the frame timestamps represent the generation time of the action frames. In some embodiments, the controller 250 can also generate a collection signal to instruct the terminal device to collect user action data. During this process, to ensure the effectiveness of user action data collection, the user needs to move within the collection range of the terminal device. The user can click the record button on the terminal device in advance and then move within the collection range to allow the terminal device to collect user action data.

[0072] In some embodiments, in order to quickly upload user action data, the controller 250 needs to establish a communication connection with the terminal device in advance through the communicator 220. To facilitate the establishment of the communication connection, the display device 200 also includes a network programming port, and the controller 250 can establish a data channel and an instruction channel according to the network programming port.

[0073] During the above process, the controller 250 can send a communication connection request to the terminal device, and the user can send feedback information regarding the communication connection request through the terminal device. If the feedback information indicates agreement to connect, the controller 250 can establish two Socket data channels with the terminal device through the network programming port to establish a communication connection.

[0074] Of the two socket data channels, one is used to transmit standard action data uploaded by the terminal device (i.e., the data channel), and the other is used to transmit service instructions and user action data collected by the terminal device (i.e., the instruction channel). The service instructions can be commands issued by the terminal device to manipulate user action data, such as play commands or pause commands.

[0075] After the controller 250 establishes a communication connection with the terminal device, it is necessary to unify the timing reference of the display device 200 and the terminal device to ensure that the display device 200 and the terminal device start timing from the same timing reference. Since there is a time delay when the terminal device transmits user action data, in some embodiments, the display device 200 needs to play the standard action data after the delay period, so that the user action data and the standard action data are displayed simultaneously, which facilitates the comparison of the action deviation between the user action and the teaching action.

[0076] Therefore, the controller 250 also needs to calculate the latency required by the display device 200. In some embodiments, such as Figure 5As shown, the controller 250 can send a synchronization command to the terminal device via a Socket data channel. The synchronization command may include the first system time of the current display device 200. After receiving the synchronization command, the terminal device can generate synchronization data, which includes the terminal device's second system time. After generating the synchronization data, the terminal device can send the synchronization data to the display device 200. After receiving the synchronization data, the controller 250 can calculate the time difference between the first and second system times. This time difference represents the time consumed by the display device 200 in sending the acquired signal to the terminal device. The controller 250 can determine the delay duration based on the time difference.

[0077] In some embodiments, considering network fluctuations during data transmission, the display device 200 consumes different amounts of time when sending and receiving data. Therefore, as Figure 6 As shown, the controller 250 can also obtain the third system time when the display device 200 receives synchronization data, and calculate the average delay duration based on the first system time and the third system time. The controller 250 can first perform a difference calculation on the first system time and the third system time to obtain the time difference. Then, it can perform an average calculation on the time difference to obtain the average time consumption for one-way data transmission between the display device 200 and the terminal device. The controller 250 can set the delay duration based on the average time consumption.

[0078] S200: After a delay, capture standard motion frame videos based on the capture frame rate to obtain standard motion data.

[0079] The delay duration is calculated based on the first system time of the current display device and the second system time of the terminal device. In this embodiment, since the terminal device incurs time consumption when uploading user action data to the display device 200, in order to synchronize the acquisition time of user action data and standard action data, the controller 250 needs to control the display 260 to play the instructional action video after the delay duration. The instructional action video demonstrates standard fitness movements, and the controller 250 can capture standard action video frames from the instructional action video according to the acquisition frame rate to obtain standard action data.

[0080] In some embodiments, the controller 250 can set the acquisition frame rate itself. However, when the acquisition frame rate of the display device 200 differs from that of the terminal device when acquiring user action data, the magnitudes of user action data and standard action data differ, thus affecting the query speed for subsequent target frame searches. For example, when the terminal device's acquisition frame rate is 2 frames / s and the display device 200's is 1 frame / s, the terminal device can acquire two action frames per second, while the display device 200 can only acquire one standard action video frame per second. Obviously, in the same time period, the user action data acquired by the terminal device is twice the standard action data acquired by the display device 200. Therefore, half of the action frames in the user action data cannot be precisely matched with the standard action video frames in the standard action data. The controller 250 can only match standard action video frames with similar timestamps based on the frame timestamps of the action frames. Therefore, a certain action deviation will occur during the comparison process.

[0081] To eliminate motion discrepancies caused by the inability to accurately match frame timestamps between motion frames and standard motion video frames, in some embodiments, the controller 250 may also set the frame rate of the display device 200 to the sampling frequency when the terminal device collects user motion data, so that the frequency at which the display device 200 collects standard motion video frames is the same as the sampling frequency of the terminal device, thus synchronizing the timestamps of user motion data and standard motion data.

[0082] In some embodiments, the controller 250 can request to obtain the acquisition frame rate of the terminal device through a synchronization command. After obtaining the synchronization command, the terminal device can add the sampling frequency of the terminal device when acquiring user action data to the synchronization data according to the feedback command input by the user, and send the synchronization data to the display device 200, so that the controller 250 sets the acquisition frame rate of the display device 200 to the sampling frequency of the terminal device when acquiring user action data according to the synchronization data.

[0083] S300: Locate the target frame in the standard motion data based on the frame timestamp.

[0084] After acquiring standard motion data based on the frame rate, the controller 250 can locate the target frame based on the motion frames in the user motion data. In some embodiments, when the controller 250 captures standard motion video frames, it can calculate the capture time of the standard motion video frames according to the timing reference based on the playback time and delay duration of the standard motion data. The timestamp of the standard motion data is the playback time plus the delay duration. For example, if the delay duration is 5 seconds, the controller 250 needs to capture the standard motion data after 5 seconds. At a frame rate of 1 frame / s, the capture time of the first standard motion video frame is 5 seconds. The capture times of subsequent standard motion time frames are the delay duration plus the corresponding playback time, i.e., 1+5s, 2+5s, 3+5s, etc.

[0085] Meanwhile, since there is a time consumption equal to the delay duration when the terminal device uploads user motion data, and because frame timestamps can authenticate the time generated by the motion frames in the user motion data, the controller 250 can search for a target frame in the standard motion data based on the frame timestamp. The target frame is the standard motion video frame whose capture time is closest to the time authenticated by the frame timestamp. The controller 250 can use the standard fitness movements in the target frame as a reference to determine whether the user's fitness movements in the motion frame are standard.

[0086] In some embodiments, considering that users first watch fitness video and then imitate the corresponding movements, a certain reaction time is required for the user to perform the movements. Therefore, the controller 250 can also set a reserved time for the terminal device to collect user movement data. For example, when the display 260 plays the standard movement video frame of movement 1, the controller 250 collects the user movement data after the reserved time through the terminal device, giving the user sufficient reaction time to perform movement 1 and ensuring the integrity of the user movement data.

[0087] S400: Generate an action score based on the similarity between the action frame and the target frame, and control the display to show the action score.

[0088] In this embodiment, after acquiring the action frame and the target frame, the controller 250 can calculate the similarity between the action frame and the target frame using the skeletal point coordinates. The controller 250 can calculate the skeletal point coordinates in the action frame and the skeletal point coordinates in the target frame respectively, and calculate the coordinate deviation between the two and the angular deviation between the lines connecting the skeletal points to obtain the similarity.

[0089] For example, taking an arm swing as an example, the hand can be designated as bone point 1, the elbow as bone point 2, and the shoulder as bone point 3. During the arm swing, the hand and elbow move around the shoulder as the center. At this time, the coordinates of bone point 3 remain unchanged, while the coordinates of bone points 1 and 2 change with the arm swing. Therefore, the controller 250 can calculate the coordinates of the hand and elbow bone points based on the user's actions, generate bone point lines between them, and compare them with the bone point coordinates and bone point lines in the target frame. The controller obtains the deviation displacement based on the bone point coordinates and the deviation angle based on the bone point lines. The deviation displacement reflects whether the amplitude of the user's action data is greater or less than the standard action, and the deviation angle reflects the angular deviation between the direction of the user's action data and the direction of the standard action.

[0090] In some embodiments, the controller 250 may also use the skeleton point coordinates as the basis for determining the target frame. After capturing standard motion video frames, the controller 250 can calculate the skeleton point coordinates of the standard motion video frames based on a joint feature recognition algorithm. For example... Figure 7 As shown, the joint feature recognition algorithm can identify joint features in the human body and establish the coordinates of skeletal points in a coordinate system based on the human body according to the joint features.

[0091] After acquiring and calculating the skeletal coordinates of the standard motion video frame, the controller 250 can mark the current display device's system time as the generation time of the skeletal coordinates. This allows the controller 250 to find the generation time of the skeletal coordinates based on the frame timestamp, thereby determining the target frame. To facilitate the storage of skeletal data, in some embodiments, such as... Figure 8 As shown, after the controller 250 generates the bone point coordinates, the generation time and bone point coordinates can be stored as a segment bone point array to save storage space for bone point data.

[0092] In some embodiments, the controller 250 can find a target segment point array based on the frame timestamp, the target segment point array being the segment point array whose generation time is closest to the frame timestamp, and mark the standard motion video frame corresponding to the target segment point array as the target frame.

[0093] After acquiring the target frame, the controller 250 can calculate the skeletal coordinates of the action frame according to the joint feature recognition algorithm, and then extract the skeletal coordinates of the target frame from the target joint bone point array. Before performing coordinate comparison, the controller 250 can also establish a coordinate system based on the human body according to user action data and standard action data. The coordinate system is established with the human body standing at the center of the screen, with the direction of the human body's arms extending horizontally to both sides as the X-axis, and the vertical direction passing through the center point of the human body as the Y-axis. The joint feature recognition algorithm can identify limb parts in the human body, such as the head, hands, chest, shoulders, waist, and legs. Based on the established coordinate system, the above-mentioned joint parts are all set with initial position coordinates.

[0094] The controller 250 can calibrate skeletal point coordinates based on user motion data and standard motion data, and calculate the coordinate deviation between the skeletal point coordinates of the motion frame and the target frame. For example, in motion 1, the standard motion data shows the left arm movement as raising upwards, while the user motion data shows the left arm movement as swinging downwards. See also... Figure 9 Based on the coordinate system established above, the left hand in the standard motion data should be in the first quadrant, while the left hand in the user motion data should be in the fourth quadrant. Therefore, there will be a deviation at least in the Y-axis direction. Figure 9 In the diagram, the dashed line represents the position of the left arm, and the coordinate points represent the coordinates of the skeletal points of the left hand. During this process, the controller 250 can also compare the coordinates of the left hand in the X-axis direction to calculate whether there is a coordinate deviation on the X-axis.

[0095] However, since there are many limbs in the human body, the controller 250 needs to calculate the coordinates of multiple limbs at the same time in order to calculate the deviation of the skeletal point coordinates, which results in a large amount of calculation for the controller 250 and affects the calculation efficiency.

[0096] In some embodiments, such as Figure 10 As shown, to improve the efficiency of coordinate deviation calculation, the controller 250 can also determine local joint points based on the limb parts corresponding to the skeletal point coordinates of the action frame and the limb parts corresponding to the skeletal point coordinates of the target frame. Local joint points can be limb parts identified based on joint feature recognition algorithms, such as the head, hand, chest, shoulder, waist, and leg.

[0097] After determining the local joint points, the controller 250 can calculate the local coordinate deviation based on these local joint points. For example, it can calculate the local coordinate deviation between the hand bone point coordinates in the user's motion data and the hand bone point coordinates in the standard motion data.

[0098] It should be noted that when calculating local coordinate deviation, it is necessary to ensure that the calculated local joint points are on the same side to avoid calculating local joint points on different sides. For example, performing calculations by comparing the coordinates of the left hand bone points in the user's motion data with the coordinates of the right hand bone points in the standard motion data will result in an error in the coordinate deviation calculation.

[0099] In some embodiments, for the calculation of symmetrical parts, the controller 250 can adjust the simultaneously calculated local coordinate deviations into two groups, that is, the controller 250 can simultaneously calculate the coordinate deviations of the left joint point and the right joint point. The controller 250 can also adjust the number of groups of simultaneously calculated local coordinate deviations according to its own data processing capabilities. For example, if the controller 250 sets the number of simultaneously calculated groups to four, the controller 250 can simultaneously calculate the local skeletal point deviations of the left hand, right hand, left shoulder, and right shoulder, thereby improving the calculation efficiency of coordinate deviations.

[0100] After calculating the local coordinate deviation, the controller 250 can perform an average calculation on the local coordinate deviation based on the number of local joints to obtain the coordinate deviation. After calculating the coordinate deviation, the controller 250 can calculate the similarity between the action frame and the target frame based on the coordinate deviation, and generate and display an action score based on the similarity to assess the standard of the user's action data.

[0101] In some embodiments, the action score can be displayed as a rating level, such as "Level A", "Level B" or "Level C", or as a specific rating score. The rating score can be set with an upper limit according to the user's preferences, and can be set to a percentage system or a ten-point system. The higher the similarity between the user's action data and the standard action data, the higher the level or score of the action score.

[0102] After the user finishes using the application, when the user launches the application again, the previously saved skeletal point coordinates and the acquired user action data will occupy the memory of the display device 200. This will cause the application to be slow to respond when requesting memory from the display device 200 during use, resulting in stuttering or delays, which will affect the user experience.

[0103] To avoid the historical skeletal point coordinates and historical user action data consuming too much memory on the display device 200, in some embodiments, the controller 250 can set a time interval for clearing the data. When the controller 250 executes step S200, it can obtain the first system time of the current display device 200. The controller 250 can determine the failure time based on the first system time and the time interval, for example... Figure 11As shown, the first system time is 16:00, the time interval is set to 20 minutes, the controller 250 takes the first system time as the first time node, and searches forward to find the second time node that is 20 minutes away from the first time node. The second time node is 15:40, which is the failure time.

[0104] After determining the failure time, the controller 250 can mark action frames with frame timestamps preceding the failure time and / or cluster arrays with generation times preceding the failure time as redundant data. For example... Figure 12 As shown, when the remaining memory of the display device 200 is less than the memory threshold, the controller 250 deletes redundant data to release the memory of the display device 200.

[0105] In some embodiments, the controller 250 can also perform a timing task on user action data and a segment array according to a time interval, the duration of which is the set time interval. When the terminal device acquires an action frame and uploads it to the display device 200, the controller 250 starts performing a timing task on the action frame. When the duration of the timing task reaches the set time interval, the controller 250 can directly delete the action frame to release memory in the display device 200.

[0106] Some embodiments of this application also provide a terminal device, which includes a camera, a communication port, and a controller. The camera is used to collect user action data, the communication port is used to establish a communication connection with any of the display devices 200 described above, and the controller is configured to control the camera to collect user action data in response to a collection signal sent by the display device 200. The user action data may include a number of action frames with frame timestamps.

[0107] In some embodiments, the camera can be a rear camera or a front camera. To facilitate user adjustment of the recording effect of user motion data, the terminal device can set the front camera as the default camera.

[0108] After acquiring user motion video, the terminal device can send the user motion data to the display device 200 via a communication port. The display device 200 then searches for a target frame in the standard motion data based on the frame timestamp, generates a motion score based on the similarity between the motion frame and the target frame, and controls the display screen 260 in the display device 200 to display the motion score. The similarity is calculated based on the deviation between the skeletal point coordinates in the motion frame and the skeletal point coordinates in the target frame.

[0109] In some embodiments, after establishing a communication connection with the display device 200, the terminal device can obtain a synchronization command sent by the display device 200. The terminal device 200 can generate synchronization data according to the synchronization command. The synchronization data may include the second system time of the current terminal device, so that the display device 200 can calculate the delay duration based on the second system time. The synchronization data may also include the sampling frequency of the terminal device's camera, so that the display device 200 can capture standard motion video frames according to the sampling frequency, thereby accurately finding the target frame and calculating the similarity between the motion frame and the target frame.

[0110] Some embodiments of this application also provide a method for displaying screen-based action scoring, which can be applied to any of the display devices 200 described above, and the method includes:

[0111] S100: In response to the start command, it receives user action data collected by the terminal device.

[0112] The user action data includes several action frames with frame timestamps.

[0113] S200: After the delay period, capture the played standard action video frames corresponding to the user action data according to the acquisition frame rate to obtain standard action data.

[0114] The delay duration is calculated based on the first system time of the current display device and the second system time of the terminal device; the acquisition frame rate is the sampling frequency when the terminal device acquires user action data.

[0115] S300: Locate the target frame in the standard motion data based on the frame timestamp.

[0116] S400: Generate an action score based on the similarity between the action frame and the target frame, and control the display to show the action score.

[0117] The similarity is calculated based on the deviation between the skeletal point coordinates in the action frame and the skeletal point coordinates in the target frame.

[0118] As can be seen from the above solutions, this application provides a display device, a terminal device, and a method for displaying action scores across multiple screens. The method responds to a start command by receiving user action data collected by the terminal device. The user action data includes several action frames with frame timestamps. After a delay, standard action video frames corresponding to the user action data are captured based on the collected frame rate to obtain standard action data. The delay duration is calculated based on the first system time of the current display device and the second system time of the terminal device. The collected frame rate is the sampling frequency when the terminal device collects user action data. Then, the target frame is searched based on the frame timestamp, and an action score is generated and displayed based on the similarity between the action frame and the target frame. This application can calculate the delay duration based on the display device and the terminal device to synchronize the time base of the display device and the terminal device through the delay duration, enabling the display device and the terminal device to compare images at the same timestamp, thus improving the accuracy of the action score display.

[0119] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0121] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the foregoing exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be made based on the foregoing teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the described embodiments.

Claims

1. A display device, characterized in that, include: monitor; A communicator is configured to establish a communication connection with a terminal device used to collect user action data. The controller is configured as follows: In response to the start command, the terminal device collects user action data, which includes several action frames with frame timestamps. Send a synchronization command to the terminal device, the synchronization command including the first system time of the current display device; The synchronization data fed back by the terminal device in response to the synchronization command is obtained. The synchronization data includes the second system time of the terminal device and the collection frame rate of the user action data. The collection frame rate is the sampling frequency when the terminal device collects the user action data. The delay duration is calculated based on the first system time and the second system time; the delay duration is obtained by calculating the first system time of the current display device and the second system time of the terminal device. After the specified delay, standard action video frames that have been played are captured according to the specified frame rate to obtain standard action data. The target frame is located in the standard motion data based on the frame timestamp. An action score is generated based on the similarity between the action frame and the target frame, and the display is controlled to show the action score. The similarity is calculated based on the deviation between the coordinates of the skeleton points in the action frame and the coordinates of the skeleton points in the target frame, and the angular deviation between the line connecting the skeleton points in the action frame and the line connecting the skeleton points in the target frame. The coordinate deviation is used to indicate whether the amplitude of the action in the user action data is greater or less than that of the standard action, and the angular deviation is used to indicate the angular deviation between the direction of the action in the user action data and the direction of the standard action.

2. The display device according to claim 1, characterized in that, After the controller executes the capture and playback of standard motion video frames according to the captured frame rate, it is further configured to: The skeletal coordinates of the standard motion video frame are calculated based on the joint feature recognition algorithm; Obtain the generation time of the bone point coordinates, and store the generation time and the bone point coordinates as a segment point array.

3. The display device according to claim 2, characterized in that, The controller performs the operation of searching for the target frame in the standard motion data based on the frame timestamp, and is further configured to: The target segment point array is found based on the frame timestamp, and the target segment point array is the segment point array whose generation time is closest to the frame timestamp; The standard motion video frames corresponding to the target joint point array are marked as target frames.

4. The display device according to claim 3, characterized in that, The controller generates an action score based on the similarity between the action frame and the target frame, and is further configured to: The coordinates of the skeletal points in the action frame are calculated based on a joint feature recognition algorithm; Extract the bone point coordinates of the target frame from the target segment point array; Calculate the coordinate deviation between the skeleton point coordinates of the action frame and the skeleton point coordinates of the target frame; The similarity is calculated based on the coordinate deviation.

5. The display device according to claim 4, characterized in that, The controller calculates the coordinate deviation between the skeletal point coordinates of the action frame and the skeletal point coordinates of the target frame, and is further configured to: Local joints are determined based on the limb parts corresponding to the skeletal point coordinates of the action frame and the limb parts corresponding to the skeletal point coordinates of the target frame. Calculate the local coordinate deviation based on the local joint points; The local coordinate deviation is calculated by averaging the local coordinate deviation based on the number of local joints to obtain the coordinate deviation.

6. The display device according to claim 5, characterized in that, The controller is further configured to: Set the time interval for clearing data; Obtain the remaining memory of the display device; The failure time is determined based on the first system time and the time interval; If the remaining memory is less than the memory threshold, redundant data is deleted. The redundant data includes action frames with frame timestamps prior to the failure time and / or the array of node points with generation times prior to the failure time.

7. The display device according to claim 1, characterized in that, It also includes a network programming port, and the controller is further configured to: Establish data and instruction channels through the network programming port; The standard action data is received through the data channel, and the user action data and the start command are received through the command channel.

8. A terminal device, characterized in that, include: A camera, configured to collect user motion data; A communication port is configured to establish a communication connection with the display device according to any one of claims 1-7; The controller is configured as follows: In response to the acquisition signal sent by the display device, the camera is controlled to acquire user action data, which includes several action frames with frame timestamps. In response to a synchronization command sent by the display device, synchronization data is sent to the display device through the communication port, so that the display device calculates the delay duration based on the synchronization data; the synchronization command includes a first system time when the display device sends the synchronization command; the synchronization data includes a second system time of the terminal device and the acquisition frame rate of the user action data; the acquisition frame rate is the sampling frequency when the terminal device acquires the user action data; the delay duration is calculated based on the first system time and the second system time. The user action data is sent to the display device through the communication port, so that the display device can find the target frame in the standard action data according to the frame timestamp after the delay time, generate an action score based on the similarity between the action frame and the target frame, and control the display to display the action score. The similarity is calculated based on the deviation between the coordinates of the skeleton points in the action frame and the coordinates of the skeleton points in the target frame, and the angular deviation between the line connecting the skeleton points in the action frame and the line connecting the skeleton points in the target frame. The coordinate deviation is used to indicate whether the action amplitude in the user action data is greater than or less than the standard action, and the angular deviation is used to indicate the angular deviation between the action direction in the user action data and the action direction of the standard action.

9. A method for displaying action scores across multiple screens, characterized in that, Applied to a display device, the display device including a display, a communicator, and a controller, wherein the communicator is configured to establish a communication connection with a terminal device, the terminal device being used to collect user action data, the method comprising: In response to the start command, the terminal device collects user action data, which includes several action frames with frame timestamps. Send a synchronization command to the terminal device, the synchronization command including the first system time of the current display device; The synchronization data fed back by the terminal device in response to the synchronization command is obtained. The synchronization data includes the second system time of the terminal device and the collection frame rate of the user action data. The collection frame rate is the sampling frequency when the terminal device collects the user action data. The delay duration is calculated based on the first system time and the second system time; the delay duration is obtained by calculating the first system time of the current display device and the second system time of the terminal device. After the specified delay, standard action video frames that have been played are captured according to the specified frame rate to obtain standard action data. The target frame is located in the standard motion data based on the frame timestamp. An action score is generated based on the similarity between the action frame and the target frame, and the display is controlled to show the action score. The similarity is calculated based on the deviation between the coordinates of the skeleton points in the action frame and the coordinates of the skeleton points in the target frame, and the angular deviation between the line connecting the skeleton points in the action frame and the line connecting the skeleton points in the target frame. The coordinate deviation is used to indicate whether the amplitude of the action in the user action data is greater or less than that of the standard action, and the angular deviation is used to indicate the angular deviation between the direction of the action in the user action data and the direction of the standard action.

Images