A motion focus switching method and display device

Abstract

This application discloses a motion focus switching method and display device. The method includes: acquiring a first image captured by an image acquisition device during user follow-up training; performing human body recognition and motion detection on the first image; acquiring a candidate image, which refers to an image of a person currently performing a preset action on a target part of the human body; determining a target image based on the candidate image and the first image currently locked by the motion focus; and switching the motion focus in the follow-up training interface displayed on the monitor to the target image. This application is designed for multi-person follow-up training scenarios, where any person can recall the motion focus back to themselves by performing a preset action, achieving dynamic and accurate matching and switching of the motion focus, improving the intelligence, flexibility, and accuracy of motion focus switching, thereby providing users with a better follow-up training experience.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a motion focus switching method and display device. Background Technology

[0002] The display device can be set up with a smart fitness application. After launching the application, users can select training videos of interest to follow along with. Following along involves the display device's camera capturing images of the user's movements. By comparing the user's movements in these images with the corresponding standard movements in keyframes of the training video, the device guides the user's training. This includes scoring the user's movements, correcting errors, and providing encouraging prompts. This allows users to train from home, improving accuracy and motivation.

[0003] The environment that a display device can capture may contain multiple people, and it typically focuses on one person for follow-along training. In multi-person follow-along scenarios, the person with the focus of motion is called the focus person. The display device only performs posture detection, scoring, and motion standardization correction for the focus person's movements. Because the identities and number of people in the environment, as well as each person's limb movements and position, can dynamically change, the display device cannot dynamically and accurately determine and switch the focus of motion, resulting in a poor user follow-along experience. Summary of the Invention

[0004] This application provides a motion focus switching method and a display device to solve the technical problem that the display device cannot dynamically and accurately determine and switch motion focus.

[0005] In a first aspect, embodiments of this application provide a display device, including:

[0006] Image capture device, used to capture the first image of the user during practice;

[0007] A display screen is used to show a follow-up exercise interface, the follow-up exercise interface including a window for displaying the first image;

[0008] The controller is used to perform:

[0009] Acquire the first image captured by the image acquisition device, and perform human body recognition and motion detection on the first image;

[0010] Acquire candidate images, where a candidate image refers to a human image of a target body part currently performing a preset action;

[0011] The target image is determined based on the candidate images and the first image currently locked by the motion focus;

[0012] The control monitor switches the motion focus in the window to the target human figure.

[0013] In some implementations, the controller performs human image recognition and motion detection on the first image, including: inputting the first image into a first computing model; obtaining human body and motion detection information output by the first computing model; wherein the human body and motion detection information includes: human body position information, limb key point position information, human body target part position information, and human body target part motion type.

[0014] In some implementations, the controller is also used to perform the following before acquiring the candidate image:

[0015] Based on the human detection status of the first images of two adjacent frames, the target tracking and matching algorithm is used to detect existing human bodies and / or newly added human bodies contained in the first image of the current frame; the existing human bodies are those that are common to the previous frame and the current frame, and the newly added human bodies are those that did not exist in the previous frame but appear in the current frame.

[0016] A new human figure is set for the newly added human body in the current frame; wherein, the human figure is used to identify and track the human body, and the human figure mapped to each human body is different;

[0017] The character identifier set for the existing human body in the previous frame will be carried over to the current frame;

[0018] If the existing human body had a motion focus in the previous frame, the focus identifier set for the existing human body in the previous frame will be carried over to the current frame.

[0019] In some implementations, the controller acquires candidate images, including:

[0020] Based on the human body location information, a human body bounding box is created; the human body bounding box is used to define the area occupied by any human body detected in the first image, and the human body bounding box is mapped to the human body identifier of the defined human body;

[0021] Based on the location information and action type of the target body part, the first target body part currently performing the preset action is located, and an action frame is created; the action frame is used to define the area occupied by the first target body part.

[0022] In some implementations, the controller acquires candidate images, including:

[0023] Calculate the intersection-union ratio (IoU) between each action bounding box and the corresponding human bounding box for each human body.

[0024] If a first target bounding box is detected, the first target bounding box is deleted; wherein, the first target bounding box is an action bounding box whose intersection-union ratio with all human bounding boxes is 0.

[0025] In some implementations, the controller acquires candidate images, including:

[0026] If the first target box is not detected, but the second target box is detected, then calculate the first distance between the center of each second target box and the key point of the target part in each third target box; wherein, the second target box is an action box with an intersection-union ratio greater than 0 with at least two human body boxes, the third target box is a human body box with an intersection-union ratio greater than 0 with the second target box, and the key point of the target part is a key point in the target part of the human body.

[0027] A fourth target box is determined that matches the second target box. The fourth target box is the third target box with the smallest first distance between the key point of the target part and the center of the second target box.

[0028] Obtain the first person identifier mapped to the fourth target box, and set the first person identifier for the second target box;

[0029] Calculate the second distance between the center of the second target box and the key point of the target part in the fourth target box;

[0030] If a fifth target box is detected, the fifth target box is deleted; wherein, the fifth target box is a second target box whose second distance from the key point of the target part in the fourth target box is greater than a threshold.

[0031] The second target bounding box that is retained is marked as the target action bounding box, and the candidate image bounded by the target action bounding box is obtained.

[0032] In some implementations, the controller acquires candidate images, including:

[0033] If neither the first target box nor the second target box is detected, the second person identifier mapped by the sixth target box is obtained; wherein, the sixth target box is the only human body box with an intersection-union ratio greater than 0 with the action box;

[0034] Set the second character identifier for the action frame;

[0035] Calculate the third distance between the center of the action frame and the key point of the target part in the sixth target frame;

[0036] If a seventh target box is detected, the seventh target box is deleted; wherein, the seventh target box is an action box whose third distance from the key point of the target part in the sixth target box is greater than the threshold.

[0037] The retained action box is marked as the target action box, and the candidate image defined by the target action box is obtained.

[0038] In some implementations, the controller determines the target portrait based on the candidate portrait and the first portrait currently focused on motion, including:

[0039] If the number of target action boxes is equal to 0, the target portrait is determined to be the first portrait, and the motion focus is not switched;

[0040] If the number of target actions is greater than 0, obtain the target person identifier corresponding to the target action box;

[0041] If the candidate image mapped by the target person identifier includes the first person image, the target person image is determined to be the first person image, and the motion focus is not switched.

[0042] In some implementations, the controller determines the target portrait based on the candidate portrait and the first portrait currently focused on motion, including:

[0043] If the number of target action boxes is equal to 1, and the candidate image mapped by the target person identifier is not the first person image, then the current candidate image is determined to be the target person image;

[0044] If the number of target action frames is greater than 1, and the candidate images mapped by the target person identifier do not include the first person image, the target person image is selected from the multiple candidate images according to a preset rule;

[0045] The preset rule includes: selecting the candidate image that is closest to the center of the screen in the image coordinate system as the target.

[0046] Alternatively, the candidate image with the smallest distance from the screen in the direction perpendicular to the screen can be selected as the target image, or the candidate image with the largest image area in the first image can be selected as the target image.

[0047] Secondly, embodiments of this application also provide a motion focus switching method, including:

[0048] The first image captured by the image acquisition device during the user's follow-up exercise is obtained, and human body recognition and motion detection are performed on the first image;

[0049] Acquire candidate images, where the candidate images refer to the images of people currently performing preset actions on the target body parts; determine the target image based on the candidate images and the first image currently locked by the motion focus;

[0050] Switch the focus of the exercise on the target human figure in the training interface displayed on the monitor.

[0051] Thirdly, embodiments of this application also provide a computer storage medium storing program instructions that, when executed on a computer, cause the computer to perform the methods involved in the above aspects and their various implementations.

[0052] In the five embodiments of this application, the image acquisition device acquires the first image of the user's training scene, and the controller can process the first image.

[0053] Human body recognition and motion detection are performed to detect a human image currently performing a preset action on a target body part, and this image is then used as a candidate image. The controller can automatically match the target image from at least one candidate image and switch the motion focus to the target image. The preset action can be a gesture, lip movement, facial expression, or other similar action.

[0054] In this multi-person training scenario, any individual can bring the focus of movement back to themselves by performing preset actions, achieving dynamic and precise matching and switching of the focus of movement, thus improving the intelligence, flexibility, and accuracy of focus switching.

[0055] This will provide users with a better practice experience. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the embodiments or the prior art will be described below.

[0057] The accompanying drawings used in the description are briefly introduced. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, they can be used to further illustrate the embodiments without creative effort.

[0058] Other accompanying figures are available.

[0059] Figure 1 This application provides an example of an operational scenario between a display device and a control device 100.

[0060] Figure 2 A hardware configuration block diagram of the control device 100 provided in the embodiments of this application;

[0061] Figure 3 A hardware configuration block diagram of a display device 200 provided in an embodiment of this application;

[0062] Figure 4 This is a software configuration diagram of the display device 200 provided in the embodiments of this application;

[0063] Figure 5 This is a schematic diagram of the homepage of a display device provided in an embodiment of this application;

[0064] Figure 6 A schematic diagram of a training project list page provided in an embodiment of this application;

[0065] Figure 7 A schematic diagram of a facial recognition page provided in an embodiment of this application;

[0066] Figure 8 A schematic diagram of the follow-up interface provided in the embodiments of this application. Figure 1 ;

[0067] Figure 9 A schematic diagram of the follow-up interface provided in the embodiments of this application. Figure 2 ;

[0068] Figure 10 A schematic diagram of the follow-up interface provided in the embodiments of this application. Figure 3 ;

[0069] Figure 11 This is a UI diagram illustrating the switching of motion focus provided in an embodiment of this application;

[0070] Figure 12 This is a schematic diagram illustrating the motion focus switching principle provided in an embodiment of this application.

[0071] Figure 13 A flowchart of the multi-target tracking method provided in the embodiments of this application;

[0072] Figure 14 A schematic diagram of the human body frame and action frame provided in the embodiments of this application;

[0073] Figure 15 A flowchart of a method for matching a human body with preset actions provided in an embodiment of this application;

[0074] Figure 16 A flowchart of the motion focus switching method provided in the embodiments of this application. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0076] Based on the exemplary embodiments shown in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete technical solution on its own.

[0077] It should be understood that the terms "first," "second," "third," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate, for example, to allow implementation in orders other than those given in the embodiments illustrated or described in this application.

[0078] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0079] The display device provided in this application embodiment can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific embodiment of the display device of this application.

[0080] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in an embodiment of this application. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control device 100.

[0081] In some implementations, the control device 100 can be a remote control. Communication between the remote control and the display device includes infrared or Bluetooth communication, as well as other short-range communication methods, to control the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, or control panel input.

[0082] In some implementations, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, an application running on the smart device can be used to control the display device 200.

[0083] In some implementations, the display device may not receive instructions using the aforementioned smart devices or control devices, but instead receive user control through touch or gestures.

[0084] In some implementations, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by receiving user voice commands directly through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving user voice commands through a voice control device set outside the display device 200.

[0085] In some implementations, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 can provide various content and interactive features to the display device 200. The server 400 can be a cluster or multiple clusters, and may include one or more types of servers.

[0086] Figure 2 This is a configuration block diagram of the control device 100 provided in an embodiment of this application. (See diagram below.) Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0087] Figure 3 This is a hardware configuration block diagram of a display device 200 provided in an embodiment of this application. For example... Figure 3 As shown, the display device 200 includes at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0088] In some implementations, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to an nth interface for input / output.

[0089] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0090] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0091] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the control device 100 or the server 400 through the communicator 220.

[0092] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote control).

[0093] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0094] The external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0095] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0096] In some implementations, the controller 250 and the tuner 210 can be located in different separate devices, that is, the tuner 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0097] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0098] In some implementations, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to nth interface for input / output, a communication bus, etc.

[0099] Users can input commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0100] A "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0101] Figure 4 This is a software configuration diagram of the display device 200 provided in an embodiment of this application. For example... Figure 4 As shown, in some implementations, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Library Layer"), and the kernel layer.

[0102] In some implementations, at least one application runs in the application layer. These applications can be Windows programs included with the operating system, system settings programs, or clock programs, or they can be applications developed by third-party developers. In practice, the application packages in the application layer are not limited to the examples above.

[0103] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0104] like Figure 4 As shown, the application framework layer in this embodiment includes managers, content providers, etc., wherein the managers include at least one of the following modules: ActivityManager, which interacts with all activities running in the system; LocationManager, which provides access to system location services for system services or applications; PackageManager, which retrieves various information related to application packages currently installed on the device; NotificationManager, which controls the display and clearing of notification messages; and WindowManager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0105] In some implementations, the activity manager is used to manage the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and navigating back. The window manager is used to manage all window programs, such as obtaining the screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling display window changes (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0106] In some implementations, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0107] In some implementations, the kernel layer acts as an intermediary between the hardware and software. For example... Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0108] For display devices with the same or similar hardware and software functions as those mentioned above, fitness applications can be set up to enable users to perform AI-powered fitness training through interaction between the display device and the user. Figure 5 This is a schematic diagram of a display device provided in an embodiment of this application. Figure 5 As shown, the homepage includes a fitness application page, which offers at least one training subject. These subjects include, but are not limited to, aerobics, yoga, parent-child exercises, square dancing, Tai Chi, martial arts, Pamela Reif, etc. This allows users to choose training subjects according to their preferences, achieving diversified training.

[0109] In some implementations, after a user selects any training subject, such as "Aerobics" and then selects HIIT (High-Intensity Interval Training) from the sub-items of Aerobics, the HIIT training program list page will be displayed. Figure 6 This is a schematic diagram of the training project list page provided in an embodiment of this application, as shown below. Figure 6 As shown, the training item list page 60 includes a training item list 61, and the training item list 62 includes at least one training item. Each training item is presented as a combination of a thumbnail of a frame from the training item video and the training item name. The thumbnail may also display information such as the video's playback duration and viewing permissions. Users can switch between selected training item videos using a control device (e.g., a remote control).

[0110] The display device detects that the remote control's focus has moved to the target training item (e.g., Figure 6 (For example, a beginner biceps workout). The control display shows the following in a designated area of ​​the training program list page 60: a description of the target training program 62 and a "Start Training" button 63. The description 62 may include, but is not limited to: the theoretical calorie consumption of the training program, the target muscle group, the difficulty level, the training intensity level, the video duration, the number of chapters, the number of participants, VIP information, and training knowledge. After the user selects the target training program, the display device can activate the image acquisition device in response to the user clicking the "Start Training" button 63.

[0111] In some implementations, the image acquisition device can capture a local image of the environmental scene within the camera-capable area in front of the screen (hereinafter referred to as: the first image). The first image may contain no people or at least one human image. If the display device detects that there is no human image in the first image, it can control the human image recognition page on the display.

[0112] Figure 7 This is a schematic diagram of a facial recognition page provided in an embodiment of this application. Figure 7As shown, the human image recognition page 70 displays a virtual human image 71 and a first prompt message. A focal aperture 72 is displayed at the feet of the virtual human image 71. The first prompt message 72 prompts the user to remain within the focal aperture 72, thus making the focal aperture 72 an effective area for the image acquisition device to recognize the user. The image acquisition device can be equipped with adjustment devices such as a gimbal to adjust the lens's shooting position and angle. The aperture can also move with the lens; for example, the focal aperture 72 can be adjusted to correspond to the center of the screen. When the user moves their body into the aperture area, the display device detects that a single human image has entered the focal aperture 72 and can control the display to show a follow-up training interface, allowing the user to perform exercise training according to the interface's guidance.

[0113] Figure 8 A schematic diagram of a follow-up exercise interface provided in an embodiment of this application. Figure 1 .like Figure 8 As shown, the follow-along interface can include two windows: a first window 81 and a second window 82. The first window 81 displays the target training video, allowing the user to adjust their limbs and perform the follow-along movements by referring to the standard actions provided in the video. The second window 82 displays a first image captured by an image acquisition device, containing a portrait of the user and their follow-along movements. Displaying the two windows in parallel on the follow-along interface facilitates user practice and comparison of movement simulations.

[0114] In some implementations, such as Figure 8 As shown, when the target training video plays to a keyframe, the standard motion information contained in the keyframe can be displayed in the first window 81. The standard motion information includes, but is not limited to: motion name (e.g., "jumping jacks"), number of completed jumps / number of jumping jacks (e.g., 4 / 10), motion elements (e.g., Step 1: standing jump, feet spread out about 1.5 shoulder widths, hands clapping above the head).

[0115] In some implementations, such as Figure 8As shown, the display device can use algorithms such as facial recognition and tracking to track and locate the position of the human figure in the first image in real time, and control the display to synchronously display the focal aperture 72 in the foot area of ​​the human figure in the second window 82. After the user simulates the standard movement shown in the first window 81, adjusts their limbs and performs the corresponding follow-up movement, the controller can use a trained model to perform matching analysis on the standard movement and the follow-up movement, calculate the movement matching degree, and thus provide different levels of encouragement, which are displayed in the second window 82, such as "Good", "Great", "Perfect", etc. The controller can control the display to display movement standardization prompts in the second window 82 based on the deviation between the follow-up movement and the standard movement. For example, if the user's follow-up movement is not as straight as the standard movement, the second window 82 will prompt "Pay attention to keeping your elbows as straight as possible and close to the sides of your head". After seeing the movement standardization prompts, the user can adjust their movement to make their follow-up movement closer to the standard movement. The follow-up interface can also display the user's training time, movement accuracy, energy consumption and other information. The application configuration and UI interaction of smart fitness are not limited to the embodiments of this application.

[0116] The aforementioned implementation requires that only one person's image be visible within the focal aperture of f / 72. Figure 8 The first image contains only one person, A. Figure 9 This application provides an embodiment of a follow-up practice interface. Figure 2 If two new characters, B and C, are added to the first image, meaning the fitness scene includes three people: A, B, and C, with A located inside the focal aperture 72 and B and C located outside the focal aperture 72, then the focus character will be locked to A, and the motion focus of the image acquisition device will not be switched to B and C.

[0117] Figure 10 This application provides an embodiment of a follow-up practice interface. Figure 3 Character C stops training, character A remains inside the focal aperture 72, and character B moves from outside to inside the focal aperture 72. This causes the display device to be unable to determine the motion focus, thus allowing control of the display. Figure 7 In the example of the facial recognition page, Person A and Person B need to negotiate with each other to determine which person stays within the focus aperture 72 and which person leaves the focus aperture 72. This inevitably interferes with the user's training.

[0118] It is evident that because the identities and number of people in a fitness scenario, as well as the limb movements and positions of each person, can change dynamically, display devices cannot dynamically, flexibly, and accurately determine or switch the focus of the movement, thus affecting the user's fitness experience.

[0119] To address the aforementioned technical issues, this application provides a motion focus switching solution for fitness scenarios. When the display device detects a preset action, it identifies the target person performing that preset action and switches the motion focus of the image acquisition device to the target person. The preset action can be a specific limb movement or form of the whole body or a part of the human body, including but not limited to: hand gestures, facial expressions, lip movements, and postures.

[0120] Figure 11 This is a UI illustration of switching motion focus provided in an embodiment of this application. Figure 11 As shown in view (a), the first image captured by the image acquisition device contains people A and B, and at this time, the motion focus 83 is locked on person A. Figure 11 As shown in view (b), person B performs a preset action, such as raising their hand and making an "OK" gesture. After processing the first image, the controller detects the preset action and determines that it is performed by person B's right hand. Then, it transfers the motion focus 83 of the image acquisition unit from person A to person B. In this way, anyone in a multi-person fitness scenario can recall the motion focus through a preset action. The motion focus can dynamically move with the preset action, rather than forcing a single person to stand within a designated aperture area. This supports multiple people exercising simultaneously and enables dynamic, flexible, and precise switching of the motion focus, thus providing users with a better fitness experience.

[0121] In some implementations, a visual motion focus is set for the target person in the second window 82, which is not limited to the following forms: setting a focus aperture in the area under the feet of the target person, setting a focus locator above the head of the target person, or setting a focus frame around the body of the target person, etc.

[0122] To enable switching of motion focus through preset actions in multi-person training scenarios, this application embodiment also provides the underlying implementation logic of the display device. Figure 12 The schematic diagram of motion focus switching provided in the embodiments of this application is as follows: Figure 12 As shown, the overall logic may include, but is not limited to, the following four steps: human body and motion detection, multi-target tracking, human body and preset motion matching, and motion focus switching.

[0123] (i) Human body and motion detection refers to: the controller acquiring the first image captured by the image acquisition device, inputting the first image into the first computational model, processing the image, and outputting human body and motion detection information. Human body and motion detection information may include, but is not limited to: human body position information, limb key point position information, target body part position information, and target body part motion type, etc. The human body and motion detection process can be executed by the controller 250, or by the detection module included in the controller 250.

[0124] The limb key points can be key points on the human body, such as the center of mass of the left hand, right hand, head, and joints. These key points can be selected based on the configured algorithm and model. The target body part is associated with a preset action. For example, if the preset action is a gesture, the target body part can be the hand (including the left and right hands), and the action type of the target body part is the gesture type, such as the "scissors" gesture, the "OK" gesture, or the heart gesture. In some implementations, during image processing, an image coordinate system XOY can be established based on the screen, and the relevant position information can be recorded in (x, y) coordinate form.

[0125] The first computational model involves algorithms related to facial recognition, human detection, limb keypoint detection, and target body motion detection (e.g., gesture detection). It can employ model architectures such as machine learning and deep neural networks. Using the first image as input data, it outputs relevant parameter information (i.e., human body and motion detection information) for matching motion focus in multi-person training scenarios. The first computational model is not limited to the following types of models: YOLO series models (e.g., YOLOv1-v7, YOLOX), R-CNN series models (e.g., Fast R-CNN, Mask R-CNN), lightweight detection models (e.g., MobileNetv1-3, ShuffleNet, SqueezeNet), etc.

[0126] (II) Multi-target tracking refers to the use of target tracking and matching algorithms to track detected human bodies and assign a unique person identifier to each body. This person identifier is equivalent to the identity ID of the tracked target, used to identify the person to whom the body is mapped. Each body has a different person identifier. Human body tracking algorithms are not limited to, for example, the Hungarian matching algorithm or neural network-based tracking algorithms (e.g., Siamese networks).

[0127] In some implementations, embodiments of this application also provide a Hungarian matching algorithm based on IOU (Intersection over Union) for tracking multiple targets. This method can be executed by controller 250, or by a multi-target tracking module included in controller 250, such as... Figure 13 As shown, it includes:

[0128] Step S131: Obtain the first image captured by the image acquisition device.

[0129] Step S132: Determine whether a human body is detected in the first image. If no human body is detected in the first image, return to step S131 and wait for the next frame of the first image. If a human body is detected in the first image, proceed to step S133.

[0130] Step S133: Determine whether a human body was detected in the first image of the previous frame.

[0131] If no human body was detected in the first image of the previous frame, it means that all human bodies detected in the current frame are newly added human bodies, which are human bodies without a human body label. Then, proceed to step S134. If a human body was detected in the first image of the previous frame, the human bodies in the previous frame and the current frame may or may not overlap. The Hungarian matching algorithm based on intersection-union ratio can be used to match the human bodies in the current frame with those in the previous frame, and then proceed to step S135.

[0132] Step S134: Set a new human figure identifier for the human body detected in the first image of the current frame. Then return to step S131 and wait for the next first image.

[0133] Step S135: Determine whether there is at least one first target human body matching between the current frame and the previous frame.

[0134] If no human body matches in the current frame and the previous frame, it means that the current frame and the previous frame have no intersection, and all detected human bodies in the current frame are newly added human bodies, then proceed to step S134. If at least one first target human body matches in the current frame and the previous frame, it means that the current frame and the previous frame have an intersection, which is the same first target human body (i.e., an existing human body), then proceed to step S136.

[0135] Step S136: The character identifier set for the first target human body in the previous frame will be carried over to the current frame, and new character identifiers will be set for other non-first target human bodies.

[0136] For example, if {human body M, human body N} are detected in the current frame and {human body A, human body B, human body C} are detected in the previous frame, then the current frame and the previous frame have no overlap, and the controller or multi-target tracking module sets new human body identifiers for human body M and human body N respectively.

[0137] For example, if {human body A, human body C, human body Q} are detected in the current frame, and {human body A, human body B, human body C} are detected in the previous frame, then the intersection of the current frame and the previous frame is {human body A, human body C}. That is, human body A and human body C are existing human bodies from the previous frame, and human body Q is a newly added human body in the current frame. We can query the character identifier set for human body A in the previous frame, which is 0001, and the character identifier set for human body C, which is 0003. Thus, the current frame uses the character identifier 0001 mapped to human body A, uses the character identifier 0003 mapped to human body C, and sets a new character identifier for human body Q, for example, setting the character identifier mapped to human body Q to 0004.

[0138] In some implementations, the controller or multi-target tracking module can also set a focus identifier for a human body with a motion focus (hereinafter referred to as: focus human body) and record the mapping relationship between the focus identifier and the human body's identifier. When switching motion focus, the controller or multi-target tracking module synchronously modifies the human body identifier mapped to the focus identifier. For example, if the motion focus switches from human body C to human body A, the mapping relationship between human body identifier 0003 and the focus identifier is canceled, and a mapping relationship between human body identifier 0001 and the focus identifier is established, thereby achieving accurate human body tracking and focus management.

[0139] In some implementations, if the current frame matches at least one first target human body in the previous frame, the controller or multi-target tracking module can determine whether the human body identifier of the first target human body is mapped to a focus identifier to find the focus human body among the first target human bodies. If a focus human body can be found among the first target human bodies, the human body identifier and focus identifier of the focus human body set in the previous frame will be carried over to the current frame, the human body identifier of the non-focus first target human body set in the previous frame will be carried over to the current frame, and new human body identifiers will be set for other non-first target human bodies. Here, non-focus first target human bodies refer to human body categories among the matched first target human bodies that do not have a motion focus, and non-first target human bodies refer to human body categories that did not successfully match in the current and previous frames.

[0140] In some implementations, if no focal human body is found among the first target human bodies, the human body identifier of the first target human body set in the previous frame can be carried over to the current frame, and new human body identifiers can be set for other non-first target human bodies. In this implementation, if the controller or detection module does not detect a human body performing a preset action, the motion focus can be determined according to preset rules. These preset rules include, but are not limited to: determining the human body closest to the center of the screen in the image coordinate system as the focal human body, or determining the human body closest to the screen in the Z-axis direction (perpendicular to the screen) as the focal human body, or determining the human body with the largest human image area in the first image as the focal human body, etc.

[0141] For example, if {human body A, human body C, human body Q} are detected in the current frame, and {human body A, human body B, human body C} are detected in the previous frame, then the intersection of the current frame and the previous frame is {human body A, human body C}. That is, human body A and human body C are the first target human bodies, and human body Q is a newly added human body in the current frame. The system queries the previous frame to find the character identifier set for human body A (0001) and human body C (0003), and the character identifier 0003 maps to the focus identifier. Thus, the current frame uses the character identifier 0001 mapped to human body A, the character identifier 0003 mapped to human body C, and the focus identifier, and sets a new character identifier 0004 for human body Q.

[0142] For example, if the character identifier set for human body A in the previous frame was 0001 and the character identifier set for human body C was 0003, but the focus identifier was mapped to human body B, then since human body B is not detected in the current frame, the current frame will continue using the character identifier 0001 mapped to human body A and the character identifier 0003 mapped to human body C, and set a new character identifier 0004 for human body Q. Furthermore, according to the aforementioned preset rules, the focus human body is determined from human body A, human body C, and human body Q. For example, if human body Q is detected as being closest to the center of the screen in the image coordinate system, then human body Q will be selected as the focus human body, and a mapping relationship between character identifier 0004 and the focus identifier will be created.

[0143] (III) Matching the human body with preset actions refers to: after acquiring the human body and action detection information, comparing whether the action type of the target part is a preset action. For example, if the preset action is the "OK" gesture, then it is determined whether each gesture detected in the first image is the "OK" gesture. If it is not determined, it indicates that no one has performed the preset action, and the motion focus is not switched; if it is determined, it indicates that someone has performed the preset action, and this embodiment of the application needs to further determine who performed the preset action. The purpose of doing this is: during multi-person training, when the distance between the human bodies is close, the limbs of multiple people may cross or overlap, which can easily lead to mismatch between the human body and the preset action, affecting the accuracy of the motion focus switching. For example, in a real fitness scenario, human body A performs the preset action, but the controller mistakenly identifies that human body B performed the preset action, causing the motion focus to be switched to human body B instead of human body A. Therefore, after the display device detects the preset action from the first image, it can execute the human body and preset action matching process to improve the accuracy of switching the motion focus.

[0144] In this embodiment, the controller 250, or the human body and preset action matching module included in the controller 250, can match the second target human body performing the preset action based on the human body and action detection information obtained in the first stage and the multi-target tracking processing result of each frame of the first image in the second stage, and set the same human body identifier for the target part performing the preset action and the second target human body. In this way, the human body performing the preset action can be recorded so as to accurately switch the motion focus.

[0145] In some implementations, the human body position information detected in the first step may include the coordinates of at least one pair of diagonal vertices of the human body. This pair of diagonal vertices can be the top-left and bottom-right vertices, or the bottom-left and top-right vertices. Thus, as... Figure 14 As shown, the controller 250 or the human body and preset action matching module can set the human body frame 131 according to the coordinates of any pair of opposite vertices of the human body. The human body frame 131 is used to define the area occupied by any human body in the first image. The human body frame 131 is mapped to the human body icon of the defined human body.

[0146] In some implementations, the target location information detected in the first stage may include the coordinates of at least one pair of diagonal vertices of the target location. For example, if the preset action is a gesture, the target location is the hand. The controller 250 or the human body and preset action matching module finds the first target location that performs the preset action based on the action type of the detected target location. For example, if the preset action is the "OK" gesture, and human body A, human body C, and human body Q are detected in the current frame, where human body A's hands do not make any gesture, human body B's left hand makes a "fist" gesture, and human body Q's right hand makes an "OK" gesture, then human body Q's right hand is determined to be the first target location.

[0147] In some implementations, such as Figure 14 As shown, the controller 250 or the human body and preset action matching module can set the action frame 132 according to the coordinates of any pair of diagonal vertices of the first target part. The action frame 132 is used to define the area occupied by the first target part in the first image that presents the preset action.

[0148] In some implementations, embodiments of this application also provide a method for matching a human body with preset actions. This method can be executed by a controller 250, or by a human body and preset action matching module included in the controller 250, such as... Figure 15 As shown, it includes:

[0149] Step S151: Construct the action frame and the corresponding human body frame for each human body.

[0150] The number of human bounding boxes can be zero or at least one. If no human bounding box is captured, it means no human body was detected in the current frame, and the system waits for the next frame. The number of action bounding boxes can also be zero or at least one. If no action bounding box is captured, it means no one is performing the preset action, and the current motion focus remains unchanged, i.e., the motion focus is not switched. The following process applies to the case where at least one human bounding box and at least one action bounding box are captured.

[0151] Step S152: Calculate the Intersection over Union (IOU) between each action bounding box and the corresponding human bounding box for each human body. ij .

[0152] Assume the current frame contains N human bounding boxes and M action bounding boxes, where 1 ≤ M ≤ N. N represents the total number of human bodies detected in the current frame, and M represents the total number of target body parts for which a preset action is performed. Calculate the Intersection over Union (IoU). ij Where i represents the sequence number of the action box, j represents the sequence number of the human body box, 1≤i≤M, 1≤j≤N, IOU ij This represents the intersection-union ratio (IU / UU) between the i-th action box and the j-th human body box. There are a total of M*N IU / UU values.

[0153] Step S153: Determine whether a first target box exists.

[0154] The first target bounding box is an action bounding box whose intersection-union ratio (IoU) with all N human bounding boxes is zero. If the first target bounding box exists, it indicates that the preset action is not presented on any human body. For example, a local area in the current frame's background is identified as the preset action, i.e., a misidentification of the preset action has occurred, and step S154 is executed. If the first target bounding box does not exist, it indicates that no misidentification of the preset action has occurred, and step S155 is executed.

[0155] Step S154: Delete the first target box.

[0156] To avoid errors in motion focus switching caused by misidentification of preset actions, such as locking the motion focus on a non-human object, making the user's follow-up training ineffective, the first target box can be deleted. In this way, the first target box will no longer participate in the motion focus switching calculation, ensuring the accuracy of motion focus switching.

[0157] Step S155: Determine whether a second target box exists.

[0158] The second target bounding box is an action bounding box whose intersection-union ratio (IU) with all K human bounding boxes is greater than 0, where 1 < K ≤ N. Since the human bodies are relatively close, limbs and body parts may overlap, meaning a certain action bounding box may intersect with multiple human bounding boxes simultaneously (equivalent to a "one-to-many" relationship). Therefore, the second target bounding box is located, and the corresponding preset action is assigned to a specific human body to ensure accurate matching between the preset action and the human body. If a second target bounding box exists, branch logic one can be executed. If no second target bounding box exists, it means that an action bounding box intersects with only one human bounding box (equivalent to a "one-to-one" relationship), and branch logic two can be executed.

[0159] Branch logic one includes the following steps S156 to S1512:

[0160] Step S156: Obtain the third target bounding box and calculate the first distance D between the center of each second target bounding box and the key point of the target part in each third target bounding box. vw .

[0161] Since the second target bounding boxes are action bounding boxes whose intersection-union ratio (IoU) with the K human bounding boxes is greater than 0, these K human bounding boxes are marked as the K acquired third target bounding boxes. Assume the current frame contains S second target bounding boxes and K third target bounding boxes. In the previous first stage, limb keypoint location information has been detected. This limb keypoint location information includes the position coordinates of keypoints in the target body parts. These keypoints are selected based on the type of preset action. If the preset action is a gesture, the keypoints in the target body parts can include keypoints for both hands; for example, the keypoint for the left hand could be the centroid of the left hand, and the keypoint for the right hand could be the centroid of the right hand. If the preset action is a lip-gesture, the keypoints in the target body parts include, but are not limited to, the corners of the mouth and the cupid's bow.

[0162] Calculate the first distance D vw Where v represents the index of the second target box, w represents the index of the third target box, 1≤v≤S, 1≤w≤K. vw This represents the first distance between the center of the v-th second target box and the key point of the target location in the w-th third target box.

[0163] In some implementations, if the preset action is a gesture, D vw It may also include: D vw_l and D vw_r Among them, D vw_l This represents the first distance between the center of the v-th second target box and the left-hand keypoint in the w-th third target box. D vw_r This represents the first distance between the center of the v-th second bounding box and the right-hand keypoint in the w-th third bounding box. Thus, the first distance value calculated for each second bounding box is 2K, and D... vw The total calculated values ​​are S*K*2.

[0164] Step S157: Based on the principle of minimum distance, determine the fourth target box that matches the second target box.

[0165] For the v-th second bounding box, find the minimum distance value D. vmin D vmin =min{D v1 D v2 D v3 D vK}, obtain the minimum distance value D vmin The corresponding human body frame, such as D vmin =D v5 If the fifth human body frame matches the vth second target frame, it means that the human body frame mapped by the vth human body frame has performed the preset action shown in the vth second target frame. Therefore, the human body frame is the fourth target frame that matches the vth second target frame.

[0166] In some implementations, if the preset action is a gesture, then D vmin =min{D v1_l D v1_r D v2_l D v2_r D vK_l D vK_r}

[0167] Step S158: Obtain the first person identifier mapped to the fourth target box, and set the first person identifier for the second target box.

[0168] The fourth target box points to a target human body, which is mapped to the first person identifier. Since this target human body is the subject performing the preset action in the second target box, the fourth target box is mapped to the same person identifier as the second target box, thus associating and matching the person's identity, the human body, and the preset action. In this way, the controller can accurately identify the actual person performing each preset action in the current frame, so as to accurately locate the target and switch the motion focus.

[0169] Step S159: Calculate the second distance d between the center of the second target box and the key point of the target part in the fourth target box. v .

[0170] Matching human bodies to preset actions based on the intersection-union ratio (IUGR) and minimum distance principle may contain errors. For example, it might be determined that the v-th gesture is performed by the w-th human body, but the actual location of the v-th gesture might be in a non-handed area within the w-th human body's frame, such as the leg area. Clearly, the gesture being performed by the leg contradicts common sense, making the v-th gesture a misidentification. To eliminate such misidentifications of preset actions and improve the accuracy of switching motion focus, after the initial matching using the minimum distance principle, a second distance and threshold can be used for secondary verification.

[0171] Step S1510: Determine whether a fifth target box exists.

[0172] In some implementations, a second distance d can be calculated between the center of each second target box and the key point of the target location in the fourth target box. v And compare the second distance d v A threshold is used to determine whether a false recognition of a preset action occurs. Here, the fifth target box refers to the second distance d between the target location key point and the matched fourth target box. v The second target bounding box that is greater than the threshold.

[0173] Taking a gesture as an example, since the gesture is made by the hand, the center of the gesture's bounding box is relatively close to the key points of the hand. Therefore, a threshold can be set empirically to constrain the maximum distance between the center of the gesture's bounding box and the key points of the hand. If a fifth bounding box exists, it indicates that at least one preset action was not made by a matched target part of the human body, resulting in a misidentification of the preset action; in this case, step S1511 is executed. If no fifth bounding box exists, i.e., any second distance d... v If all values ​​are less than or equal to the threshold, then the preset action matches the human body correctly, triggering the process of switching motion focus.

[0174] Step S1511: Delete the fifth target box.

[0175] Deleting invalid fifth target boxes prevents them from participating in motion focus switching control, thus avoiding motion focus switching errors caused by misidentification of preset actions and improving the accuracy of motion focus switching.

[0176] Branch logic two includes the following steps S1512 to S1515:

[0177] Step S1512: Obtain the second character identifier mapped to the sixth target box, and set the second character identifier for the action box that matches the sixth target box.

[0178] If the intersection-union ratio (IoU) of the i-th action bounding box with only one human bounding box is greater than 0, meaning that the human bounding box and the i-th action bounding box are matched one-to-one, then the human bounding box is marked as the acquired sixth target bounding box. Matching action bounding boxes and sixth target bounding boxes are mapped to the same second person identifier, thus associating and matching the person's identity, the human body, and the preset action. In this way, the controller can accurately identify the actual person performing each preset action in the current frame, enabling precise target location and switching of motion focus.

[0179] Step S1513: Calculate the third distance d between the key points of the target area in the sixth target frame and the center of the matched action frame. i .

[0180] Step S1514: Determine if a seventh target box exists.

[0181] In some implementations, to avoid misidentification of preset actions, a third distance d between the center of the action bounding box and the key point of the target part in the sixth target bounding box can be calculated. i And compare the third distance d i And a threshold. Here, the seventh bounding box refers to the third distance d between the target location keypoints and the matching sixth bounding box. iAction bounding boxes exceeding the threshold. If a seventh target bounding box exists, it indicates that at least one preset action was not performed by a matched target part of the human body, resulting in a misidentification of the preset action, then proceed to step S1515. If no seventh target bounding box exists, i.e., any third distance d... i If all values ​​are less than or equal to the threshold, then the preset action matches the human body correctly, triggering the process of switching motion focus.

[0182] Step S1515: Delete the seventh target box.

[0183] Through branch logic one and branch logic two, action boxes that may cause misidentification and matching errors are filtered out, and action boxes that can be correctly matched with the action executor (hereinafter collectively referred to as: target action boxes) are retained. The person identifier mapped to the target action box is hereinafter collectively referred to as the target person identifier.

[0184] (iv) Switching the focus of motion refers to: based on the matching results between the human body and the preset action obtained in the third step above, and combined with the current focus of motion, deciding whether to switch the focus of motion and which target person to lock the focus of motion.

[0185] In some implementations, embodiments of this application also provide a motion focus switching method, which can be executed by a controller 250 or by a motion focus switching module included in the controller 250, such as... Figure 16 As shown, it includes:

[0186] Step S161: Determine whether the number of target action boxes is greater than 0.

[0187] If the number of target action boxes is 0, it means that no one has been detected performing the preset action in the current frame, so proceed to step S167. If the number of target action boxes is greater than 0, proceed to step S162.

[0188] Step S162: Obtain the target person identifier corresponding to the target action box.

[0189] Step S163: Determine whether the candidate image mapped by the target person identifier currently has a motion focus.

[0190] In some implementations, the controller or motion focus switching module can query whether there is a mapping relationship between the target person identifier and the focus identifier. If there is a mapping relationship between the target person identifier and the focus identifier, it indicates that one of the candidate images performing the preset action currently has the motion focus, and step S167 is executed. If there is no mapping relationship between the target person identifier and the focus identifier, it indicates that none of the candidate images performing the preset action has the motion focus, and the motion focus needs to be moved to a target person image among the candidate images, and step S164 is executed.

[0191] Step S164: Determine whether the number of target action boxes is greater than 1.

[0192] If the number of target action boxes is equal to 1, it indicates that there is only one candidate image in the current frame, so proceed to step S165; if the number of target action boxes is greater than 1, it indicates that there are multiple candidate images in the current frame, so proceed to step S166.

[0193] Step S165: Determine the current candidate image as the target image, and control the display to switch the motion focus to the target image.

[0194] Step S166: Select the target portrait from multiple candidate portraits according to preset rules, and control the display to switch the motion focus to the target portrait.

[0195] If multiple characters perform preset actions in the current frame, and none of these characters have a motion focus, a target character can be selected from multiple candidate images according to preset rules. These preset rules include, but are not limited to: selecting the candidate image closest to the center of the screen in the image coordinate system as the target character, or selecting the candidate image closest to the screen in the Z-axis direction (perpendicular to the screen) as the target character, or selecting the candidate image with the largest character area in the current frame as the target character, etc.

[0196] The presentation of the motion focus is not limited. For example, a focus aperture can be set below the feet area of ​​the target portrait, a focus frame can be set around the target portrait, or a focus locator can be set above the head of the target portrait.

[0197] Step S167: Do not switch the motion focus; keep the current motion focus unchanged. That is, determine that the target portrait is still the first portrait with the current motion focus, and there is no need to shift the motion focus.

[0198] In some implementations, the controller can detect only the follow-up information of the target human body currently having the focus of motion. This follow-up information includes, but is not limited to, the positional information of key limb points. The controller uses this follow-up information to perform relevant functions of the smart fitness application, such as: follow-up scoring, evaluating the accuracy of fitness movements, correcting incorrect follow-up movements, and adaptively providing encouraging messages.

[0199] In this embodiment, the display device can dynamically identify the human body following the exercise based on the first image captured by the camera, and sequentially complete four stages: human body and motion detection, multi-target tracking, matching the human body with preset actions, and switching the motion focus. This enables dynamic adaptive matching and switching of the motion focus in multi-person fitness scenarios if at least one user is detected performing a preset action, reducing the false recognition rate of preset actions and improving the intelligence and accuracy of motion focus switching, thereby providing users with a better exercise experience. The UI display and functions involved in the fitness application are not limited to the examples in this embodiment.

[0200] In some implementations, embodiments of this application also provide a computer storage medium that can store a program. When the computer storage medium is configured in the display device 200, the program, when executed, can include program steps of a motion focus switching method configured for the controller 250. The computer storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0202] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the foregoing exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be made based on the foregoing teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better utilize the described embodiments.

Claims

1. A display device, characterized in that, include: Image capture device, used to capture the first image of the user during practice; A display screen is used to show a follow-up exercise interface, the follow-up exercise interface including a window for displaying the first image; The controller is used to perform: Acquire the first image captured by the image acquisition device, and perform human body recognition and motion detection on the first image; Acquire candidate images, where a candidate image refers to a human image of a target body part currently performing a preset action; The target image is determined based on the candidate images and the first image currently locked by the motion focus; Control the display to switch the motion focus in the window to the target human figure; The controller acquires candidate images, including: A human body bounding box is created based on the human body location information; the human body bounding box is used to define the area occupied by any human body detected in the first image, and the human body bounding box is mapped to the human body identifier of the defined human body; Based on the location information and action type of the target body part, locate the first target body part currently performing the preset action and create an action frame; the action frame is used to define the area occupied by the first target body part. Calculate the intersection-union ratio (IoU) between each action bounding box and the corresponding human bounding box for each human body. If a first target bounding box is detected, the first target bounding box is deleted; wherein, the first target bounding box is an action bounding box whose intersection-union ratio with all human bounding boxes is 0; If the first target box is not detected, but the second target box is detected, then calculate the first distance between the center of each second target box and the key point of the target part in each third target box; wherein, the second target box is an action box with an intersection-union ratio greater than 0 with at least two human body boxes, the third target box is a human body box with an intersection-union ratio greater than 0 with the second target box, and the key point of the target part is a key point in the target part of the human body. A fourth target box is determined that matches the second target box. The fourth target box is the third target box with the smallest first distance between the key point of the target part and the center of the second target box. Obtain the first person identifier mapped to the fourth target box, and set the first person identifier for the second target box; Calculate the second distance between the center of the second target box and the key point of the target part in the fourth target box; If a fifth target box is detected, the fifth target box is deleted; wherein, the fifth target box is a second target box whose second distance from the key point of the target part in the fourth target box is greater than a threshold. The second target bounding box that is retained is marked as the target action bounding box, and the candidate image bounded by the target action bounding box is obtained.

2. The display device according to claim 1, characterized in that, The controller performs facial recognition and motion detection on the first image, including: The first image is input into the first computation model; Obtain the human body and motion detection information output by the first computing model; The human body and motion detection information includes: human body location information, limb key point location information, human body target part location information, and human body target part motion type.

3. The display device according to claim 2, characterized in that, Before acquiring candidate images, the controller is also used to perform: Based on the human detection status of the first images of two adjacent frames, the target tracking and matching algorithm is used to detect existing human bodies and / or newly added human bodies contained in the first image of the current frame; the existing human bodies are those that are common to the previous frame and the current frame, and the newly added human bodies are those that did not exist in the previous frame but appear in the current frame. A new human figure is set for the newly added human body in the current frame; wherein, the human figure is used to identify and track the human body, and the human figure mapped to each human body is different; The character identifier set for the existing human body in the previous frame will be carried over to the current frame; If the existing human body had a motion focus in the previous frame, the focus identifier set for the existing human body in the previous frame will be carried over to the current frame.

4. The display device according to claim 1, characterized in that, The controller acquires candidate images, including: If neither the first target box nor the second target box is detected, the second person identifier mapped by the sixth target box is obtained; wherein, the sixth target box is the only human body box with an intersection-union ratio greater than 0 with the action box; Set the second character identifier for the action frame; Calculate the third distance between the center of the action frame and the key point of the target part in the sixth target frame; If a seventh target box is detected, the seventh target box is deleted; wherein, the seventh target box is an action box whose third distance from the key point of the target part in the sixth target box is greater than the threshold. The retained action box is marked as the target action box, and the candidate image defined by the target action box is obtained.

5. The display device according to claim 1 or 4, characterized in that, The controller determines the target image based on the candidate image and the first image currently focused on motion, including: If the number of target action boxes is equal to 0, the target portrait is determined to be the first portrait, and the motion focus is not switched; If the number of target action boxes is greater than 0, obtain the target person identifier corresponding to the target action box; If the candidate image mapped by the target person identifier includes the first person image, the target person image is determined to be the first person image, and the motion focus is not switched.

6. The display device according to claim 5, characterized in that, The controller determines the target image based on the candidate image and the first image currently focused on motion, including: If the number of target action boxes is equal to 1, and the candidate image mapped by the target person identifier is not the first person image, then the current candidate image is determined to be the target person image; If the number of target action frames is greater than 1, and the candidate images mapped by the target person identifier do not include the first person image, the target person image is selected from the multiple candidate images according to a preset rule; The preset rules include: selecting the candidate image closest to the center of the screen in the image coordinate system as the target image; or selecting the candidate image with the smallest distance from the screen in the direction perpendicular to the screen as the target image; or selecting the candidate image with the largest image area in the first image as the target image.

7. A method for switching motion focus, characterized in that, include: The first image captured by the image acquisition device during the user's follow-up exercise is obtained, and human body recognition and motion detection are performed on the first image; Acquire candidate images, where a candidate image refers to a human image of a target body part currently performing a preset action; The target image is determined based on the candidate images and the first image currently locked by the motion focus; Switch the motion focus in the follow-up interface displayed on the monitor to the target human figure; The process of obtaining candidate images includes: A human body bounding box is created based on the human body location information; the human body bounding box is used to define the area occupied by any human body detected in the first image, and the human body bounding box is mapped to the human body identifier of the defined human body; Based on the location information and action type of the target body part, locate the first target body part currently performing the preset action and create an action frame; the action frame is used to define the area occupied by the first target body part. Calculate the intersection-union ratio (IoU) between each action bounding box and the corresponding human bounding box for each human body. If a first target bounding box is detected, the first target bounding box is deleted; wherein, the first target bounding box is an action bounding box whose intersection-union ratio with all human bounding boxes is 0; If the first target box is not detected, but the second target box is detected, then calculate the first distance between the center of each second target box and the key point of the target part in each third target box; wherein, the second target box is an action box with an intersection-union ratio greater than 0 with at least two human body boxes, the third target box is a human body box with an intersection-union ratio greater than 0 with the second target box, and the key point of the target part is a key point in the target part of the human body. A fourth target box is determined that matches the second target box. The fourth target box is the third target box with the smallest first distance between the key point of the target part and the center of the second target box. Obtain the first person identifier mapped to the fourth target box, and set the first person identifier for the second target box; Calculate the second distance between the center of the second target box and the key point of the target part in the fourth target box; If a fifth target box is detected, the fifth target box is deleted; wherein, the fifth target box is a second target box whose second distance from the key point of the target part in the fourth target box is greater than a threshold. The second target bounding box that is retained is marked as the target action bounding box, and the candidate image bounded by the target action bounding box is obtained.

Images