Display device and virtual character display method

Abstract

Some embodiments of the present application show a display device and a virtual character display method, the method comprising: in response to a user input playing a video instruction, playing the video, and starting a camera; determining first limb point information of the user based on first image data; obtaining second image data of the video, and determining second limb point information of a video character in the video based on the second image data; calculating a scaling ratio according to the first limb point information and the second limb point information; drawing a virtual character based on the first limb point information, the positioning limb point information and the scaling ratio; and displaying the virtual character on a floating layer of a playing interface of the video. Embodiments of the present application draw a virtual character of the same scale as the video character by using the video character limb point information and the user limb point information collected by the camera, and display the virtual character at a specified position, so that the user can more intuitively feel his / her own motion state, and it is convenient to determine the standardization of the action and make timely adjustments to the action.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a method for displaying virtual characters. Background Technology

[0002] Fitness videos in fitness apps mainly fall into two categories: one is regular fitness, which simply plays the video and the user follows along with the movements; the other is AI (Artificial Intelligence) fitness, where the device's camera simultaneously captures the user's movements and displays them on the screen while the video is playing. The current user interface can be divided into two parts: one plays the fitness video, and the other displays the user's image captured by the camera. This allows users to see their own movements while watching the demonstration in the video, thus correcting any improper form. However, users can only roughly assess the correctness of their movements using this method. When higher precision is required, it's impossible to determine the correctness of the movements or make timely adjustments. Summary of the Invention

[0003] Some embodiments of this application provide a display device and a method for displaying virtual characters. By using the limb point information of a video character and the limb point information of a user collected by a camera, a virtual character of the same scale as the video character is drawn and displayed at a designated location. This allows the user to more intuitively feel their own movement state, making it easier to determine the standardization of the movement and make timely adjustments to the movement.

[0004] In a first aspect, some embodiments of this application provide a display device, including:

[0005] monitor;

[0006] Camera;

[0007] The controller is configured as follows:

[0008] In response to a user's instruction to play a video, the video is played, and the camera is turned on so that the camera can capture first image data;

[0009] Based on the first image data, determine the user's first limb point information;

[0010] The second image data of the video is acquired, and the second limb point information of the video character in the video is determined based on the second image data. The second limb point information includes positioning limb point information, which is used to determine the position of the virtual character.

[0011] Calculate the scaling ratio based on the first limb point information and the second limb point information;

[0012] A virtual character is drawn based on the first limb point information, the location limb point information, and the scaling ratio;

[0013] The virtual character is displayed in a floating layer on top of the video playback interface.

[0014] In some embodiments, the controller performs the function of determining second limb point information of a person in the video based on the second image data, which is further configured to:

[0015] Obtain the limb point information of the person in the video from the second image data;

[0016] If the number of limb points in the limb point information to be determined is greater than a preset number and the limb point information to be determined includes positioning limb point information, then the limb point information to be determined is determined as the second limb point information.

[0017] In some embodiments, the controller performs a scaling ratio calculation based on the first limb point information and the second limb point information, and is further configured to:

[0018] The first length of the target bone is calculated based on the first target limb point information and the second target limb point information in the first limb point information. The positions of the first target limb point and the second target limb point are adjacent, and the target bone is the line connecting the first target limb point and the second target limb point.

[0019] The second length of the target bone is calculated based on the first target limb point information and the second target limb point information in the second limb point information;

[0020] The scaling ratio is determined to be the ratio of the second length to the first length.

[0021] In some embodiments, the controller performs a scaling ratio calculation based on the first limb point information and the second limb point information, and is further configured to:

[0022] Based on the first limb point information, the lengths of at least two different first target bones are determined, where the first target bone is the line connecting two adjacent target limb points in the first limb point information.

[0023] Calculate the sum of the lengths of the at least two different first target bones;

[0024] The lengths of at least two different second target bones are determined based on the second limb point information. The second target bone is the line connecting two target limb points in the second limb point information. The first target bone is associated with the second target bone.

[0025] Calculate the sum of the lengths of the at least two different second target bones;

[0026] The scaling ratio is determined to be the ratio of the sum of the lengths of the second target bones to the sum of the lengths of the first target bones.

[0027] In some embodiments, the controller performs the drawing of a virtual character based on the first limb point information, the positioned limb point information, and the scaling ratio, and is further configured to:

[0028] The third limb point information of the scaled virtual character is determined based on the first limb point information and the scaling ratio.

[0029] Determine the horizontal and vertical coordinate offsets;

[0030] The fourth limb point information of the virtual character is determined based on the horizontal coordinate offset, the vertical coordinate offset, and the third limb point information;

[0031] The virtual character is drawn based on the information of the fourth limb point.

[0032] In some embodiments, the controller performs the determination of the horizontal and vertical coordinate offsets, and is further configured to:

[0033] The horizontal coordinate offset is determined based on the information of the first limb point and the information of the third limb point.

[0034] The vertical coordinate offset is determined based on the location limb point information and the third limb point information.

[0035] In some embodiments, the controller performs the determination of the horizontal and vertical coordinate offsets, and is further configured to:

[0036] The horizontal and vertical coordinate offsets are determined based on the location limb point information and the third limb point information.

[0037] In some embodiments, the controller is configured to:

[0038] Collect the pixel colors of the second image data;

[0039] The color of the virtual character is determined based on the pixel color.

[0040] In some embodiments, the pixel color includes a first red value, a first green value, and a first blue value, and the controller performs the function of determining the color of the virtual character based on the pixel color, which is further configured to:

[0041] If the first red value, the first green value, or the first blue value is greater than a preset value, then the second red value, the second green value, or the second blue value is the difference between the first red value, the first green value, or the first blue value and the preset value.

[0042] If the first red value, the first green value, or the first blue value is not greater than a preset value, then the second red value, the second green value, or the second blue value is the sum of the first red value, the first green value, or the first blue value and the preset value;

[0043] The colors of the virtual character are determined to be a second red value, a second green value, and a second blue value.

[0044] Secondly, some embodiments of this application provide a method for displaying a virtual character, including:

[0045] In response to a user's instruction to play a video, the video is played, and the camera is turned on so that the camera can capture first image data;

[0046] Based on the first image data, determine the user's first limb point information;

[0047] The second image data of the video is acquired, and the second limb point information of the target video character in the video is determined based on the second image data. The second limb point information includes positioning limb point information, which is used to determine the position of the virtual character.

[0048] Calculate the scaling ratio based on the first limb point information and the second limb point information;

[0049] A virtual character is drawn based on the first limb point information, the location limb point information, and the scaling ratio;

[0050] The virtual character is displayed in a floating layer on top of the video playback interface.

[0051] Some embodiments of this application provide a display device and a method for displaying a virtual character. Upon receiving a user's input command to play a video, the video is played and a camera is activated to capture first image data of the user. Based on the first image data, first limb point information of the user can be determined. Second image data of the video is then acquired, and second limb point information of the video character is determined based on the second image data. A scaling ratio is determined based on the first and second limb point information. A virtual character is drawn based on the first limb point information, the positioned limb point information, and the scaling ratio, and the virtual character is displayed on the upper layer of the video playback interface. This application embodiment uses the limb point information of the video character and the user's limb point information captured by the camera to draw a virtual character proportional to the video character and display it at a designated location, allowing the user to more intuitively perceive their own movement state, facilitating the determination of the standardization of actions and timely adjustments to the actions. Attached Figure Description

[0052] Figure 1An operational scenario between a display device and a control device according to some embodiments is illustrated;

[0053] Figure 2 A hardware configuration block diagram of a control device according to some embodiments is shown;

[0054] Figure 3 A hardware configuration block diagram of a display device according to some embodiments is shown;

[0055] Figure 4 A software configuration diagram in a display device according to some embodiments is shown;

[0056] Figure 5 A flowchart of a virtual character display method according to some embodiments is shown;

[0057] Figure 6 A flowchart of a camera activation method according to some embodiments is shown;

[0058] Figure 7 A flowchart of a method for determining first limb point information according to some embodiments is shown;

[0059] Figure 8 A schematic diagram of a prompting interface provided according to some embodiments is shown;

[0060] Figure 9 A schematic diagram of a human limb point provided according to some embodiments is shown;

[0061] Figure 10 A schematic diagram of another human limb point provided according to some embodiments is shown;

[0062] Figure 11 A schematic diagram of another recommended interface provided according to some embodiments is shown;

[0063] Figure 12 A schematic diagram of a virtual character display according to some embodiments is shown;

[0064] Figure 13 This diagram illustrates a relationship between a video layer and a floating layer according to some embodiments;

[0065] Figure 14 A system framework diagram is shown according to some embodiments. Detailed Implementation

[0066] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0067] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0068] The terms "first" and "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0069] The term “comprising” and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0070] The display device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific embodiment of the display device of this application.

[0071] Figure 1 This is a schematic diagram illustrating the operational scenario between the display device and the control unit according to the embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control device 100.

[0072] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0073] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0074] In some embodiments, the display device may receive instructions not through the aforementioned smart devices or control devices, but through touch or gestures.

[0075] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0076] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0077] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0078] like Figure 3 The display device 200 includes at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0079] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to an nth interface for input / output.

[0080] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0081] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0082] The display 260 also includes a touch screen, which is used to receive input control commands from the user's fingers by sliding or clicking on the touch screen.

[0083] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0084] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote control).

[0085] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0086] The external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0087] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0088] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0089] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0090] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0091] Users can input commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0092] A "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0093] See Figure 4 In some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Layer"), and the kernel layer.

[0094] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0095] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0096] like Figure 4 As shown, the application framework layer in this embodiment includes managers, content providers, etc., wherein the managers include at least one of the following modules: ActivityManager, which interacts with all activities running in the system; LocationManager, which provides access to system location services for system services or applications; PackageManager, which retrieves various information related to application packages currently installed on the device; NotificationManager, which controls the display and clearing of notification messages; and WindowManager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0097] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling display window changes (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0098] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0099] In some embodiments, the kernel layer is a layer between hardware and software. For example... Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0100] Fitness videos in fitness apps mainly fall into two categories: one is regular fitness, which simply plays the video and the user follows along with the movements; the other is AI (Artificial Intelligence) fitness, where the device's camera simultaneously captures the user's movements and displays them on the screen while the video is playing. The current user interface can be divided into two parts: one plays the fitness video, and the other displays the user's image captured by the camera. This allows users to see their own movements while watching the demonstration in the video, thus correcting any improper form. However, users can only roughly assess the correctness of their movements using this method. When higher precision is required, it's impossible to determine the correctness of the movements or make timely adjustments.

[0101] To address the above technical problems, this application provides a display device 200. The structure and functions of each part of the display device 200 can be found in the above embodiments. Furthermore, based on the display device 200 shown in the above embodiments, this embodiment further improves some functions of the display device 200. For example… Figure 5 As shown, controller 250 performs the following steps:

[0102] Step S501: Receive user input to play video;

[0103] The videos include fitness videos, martial arts videos, and dance videos.

[0104] In some embodiments, receiving a user input instruction to play a video includes:

[0105] In response to a user input instruction to display a user interface, the display is controlled to show the user interface, which includes at least one video control;

[0106] Receive user input indicating that the video control has been selected.

[0107] For example, upon receiving a user's input instruction to open the application, the system controls the display to show the application's homepage, which includes at least one video control; and receives a user's input instruction to select the video control.

[0108] Step S502: In response to the user's input instruction to play the video, play the video and turn on the camera so that the camera can capture the first image data;

[0109] In response to a user's instruction to play a video, a video data request is sent to the server, the video data request including the identifier corresponding to the video;

[0110] Receive video stream data and video information sent by the server according to the corresponding identifier of the video;

[0111] After receiving the video stream data, play the video corresponding to the video stream data;

[0112] In some embodiments, the video information includes a video type field. After receiving the video information, it is determined whether the data in the video type field is an AI fitness video.

[0113] If the data in the video type field is an AI fitness video, then proceed with the step of turning on the camera;

[0114] If the data in the video type field is not an AI fitness video, the step of turning on the camera will not be executed.

[0115] In some embodiments, the video information includes a video type field and a fitness type field, and it is determined whether the data in the video type field is a fitness video;

[0116] If the data in the video type field is not a fitness video, the step of turning on the camera will not be performed.

[0117] If the data in the video type field is a fitness video, determine whether the data in the fitness type field is an AI fitness video;

[0118] If the data in the Fitness Type field is an AI fitness video, then proceed with the step of turning on the camera.

[0119] If the data in the Fitness Type field is not an AI fitness video, the step of turning on the camera will not be executed.

[0120] In some embodiments, after the video media asset loads normally and video information is received, a step is performed to determine whether the data in the video type field is an AI fitness video (fitness video). Normal video media asset loading includes receiving video stream data and starting video playback. If video media asset loading fails, i.e., video stream data is not received or video playback fails to start, playback is stopped and the camera is not turned on.

[0121] In some embodiments, such as Figure 6 As shown, the step of turning on the camera to allow it to acquire first image data includes:

[0122] Step S601: Detect whether the camera exists;

[0123] Among these methods, it can be determined whether the display device is equipped with a camera by obtaining the basic configuration information of the display device, and it can also be determined whether the display device has an external camera by calling the external device interface.

[0124] If the camera is not present, the display will show a prompt to inform the user that no camera has been detected.

[0125] If the camera exists, proceed to step S602: activate the camera;

[0126] Step S603: Determine whether a message indicating successful camera startup has been received;

[0127] If a message indicating successful camera startup is not received or a message indicating camera startup failure is received, the display will show a prompt to inform the user that the camera has failed to start successfully, allowing the user to check whether the camera switch is on or if other applications are using the camera.

[0128] If a message indicating successful camera startup is received, proceed to step S604: Start Preview;

[0129] Step S605: Determine whether a message indicating successful preview creation has been received;

[0130] If no message indicating successful preview creation is received, or a message indicating preview creation failure is received, the display will show a notification to inform the user that the current preview creation has failed.

[0131] If a message indicating successful preview creation is received, proceed to step S606: obtain preview frame data, i.e., obtain the first image data captured by the camera.

[0132] Step S503: Determine the user's first limb point information based on the first image data;

[0133] In some embodiments, after acquiring the first image data captured by the camera, such as Figure 7 As shown, the step of determining the user's first limb point information based on the first image data includes:

[0134] Step S701: Preprocess the first image data. The purpose of preprocessing is to adjust the size of the first image data to a size that can be processed by the limb detection model.

[0135] Step S702: Determine whether the limb detection model has been successfully loaded;

[0136] If the limb detection model is successfully loaded, then step S703 is executed: the preprocessed first image data is input into the limb detection model, and human skeletal point information is obtained through limb point regression processing.

[0137] Step S704: Determine whether the human target exists by using human skeletal point information;

[0138] In some embodiments, the step of determining the presence of a human target using human skeletal point information includes:

[0139] Determine whether the number of the skeletal points is greater than a preset number;

[0140] If the number of skeletal points is greater than a preset number, then the existence of a human target is determined. For example, the preset number is 8.

[0141] If the number of skeletal points is greater than the preset number, then the human target is determined to be non-existent.

[0142] If a human target exists, proceed to step S705: regress the target person through skeletal points, input the target person tracking model, and convert the target person's skeletal points into first limb point information.

[0143] If the human target is not present, the control monitor will display a prompt interface to alert the user that the entire body has entered the center of the screen.

[0144] When a human target is determined to be absent, a virtual character can be drawn based on the detected skeletal point information and displayed in the prompt interface. This allows the user to clearly understand how to adjust the screen angle and position to center the entire body in the frame. For example, the prompt interface is as follows: Figure 8 As shown.

[0145] Step S504: Obtain the second image data of the video, and determine the second limb point information of the video character based on the second image data. The second limb point information includes positioning limb point information, which is used to determine the position of the virtual character.

[0146] After the video starts playing, the second image data of the video is acquired.

[0147] In some embodiments, second image data of the video is acquired starting from the first frame of video data of the video;

[0148] In some embodiments, the first few seconds of a fitness video may be an introduction to the fitness movements. Therefore, in order to reduce the computational load of limb detection on the image data, the second image data of the video can be obtained starting from the video frame data corresponding to the preset playback time of the video.

[0149] In some embodiments, after acquiring the second image data of the video, the step of determining the second limb point information of a person in the video based on the second image data includes:

[0150] To obtain the limb point information of the person in the video in the second image data, that is, to use a limb detection algorithm to detect the second image data and obtain the limb point information of the person in the video in the video. The limb point information includes the coordinate information of multiple limb points.

[0151] In some embodiments, such as Figure 9 As shown, the limb points include the nose, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist, right hip, left hip, right knee, left knee, right ankle, and left ankle.

[0152] In some embodiments, such as Figure 10 As shown, the limb points include the nose, right eye, left eye, right ear, left ear, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist, right hip, left hip, right knee, left knee, right ankle, and left ankle.

[0153] Determine the number of limb points in the limb point information to be determined;

[0154] Determine whether the number of the limb points is greater than a preset number;

[0155] In some embodiments, if the number of limb points is greater than a preset number, it indicates that the whole body or most of the body of the person in the video has appeared in the video image, and the limb point information to be determined can be identified as second limb point information.

[0156] If the number of limb points is not greater than a preset number, it indicates that the whole body or most of the body of the person in the video is not in the video image. Then, the next second image data in the video is acquired until the second limb point information can be determined.

[0157] After determining that the limb point information to be determined is the second limb point information, any limb point in the second limb point information can be selected as the positioning limb point. The positioning limb point information is used to determine the position of the virtual character.

[0158] In some embodiments, if the number of limb points is greater than a preset number, it is determined whether the limb point information to be determined includes the location limb point information.

[0159] In some embodiments, the limb point can be any limb point.

[0160] In some embodiments, the limb point is the nose, right ankle, or left ankle.

[0161] If the limb point information to be determined includes location limb point information, then the limb point information to be determined is determined to be the second limb point information.

[0162] If the limb point information to be determined does not include the location limb point information, it indicates that the location limb point information is not in the limb point information to be determined. Then, the next second image data of the video is acquired until the second limb point information can be determined.

[0163] Step S505: Calculate the scaling ratio based on the first limb point information and the second limb point information;

[0164] In some embodiments, the step of calculating the scaling ratio based on the first limb point information and the second limb point information includes:

[0165] Obtain the position information of the nose and the left (right) ankle from the first limb point information;

[0166] The first height is determined based on the position information of the nose and the position information of the left (right) ankle;

[0167] For example, the location information is coordinate information, the coordinate information of the nose is (x1, y1), the coordinate information of the left ankle (right ankle) is (x2, y2), and the first height h = y2 - y1.

[0168] Obtain the position information of the nose and the left (right) ankle from the second limb point information;

[0169] The second height is determined based on the position information of the nose and the position information of the left (right) ankle;

[0170] For example, the location information is coordinate information, the coordinate information of the nose is (X1, Y1), the coordinate information of the left ankle (right ankle) is (X2, Y2), and the second height H = Y2 - Y1.

[0171] The scaling ratio is determined to be the ratio of the second height to the first height.

[0172] For example, the scaling factor s = H / h.

[0173] In this embodiment, the display height of the user or video character is determined by the distance between the nose and the ankle. Then, a virtual character proportional to the video character is drawn by the ratio of the two distances and the user's actual limb point information, so that the user can more intuitively feel his own movement state and enhance interactivity.

[0174] In some embodiments, the step of calculating the scaling ratio based on the first limb point information and the second limb point information includes:

[0175] The first length of the target bone is calculated based on the first target limb point information and the second target limb point information in the first limb point information. The positions of the first target limb point and the second target limb point are adjacent, and the target bone is the line connecting the first target limb point and the second target limb point.

[0176] In this embodiment of the application, the adjacent positions of the first target limb point and the second target limb point mean that the two target limb points conform to the rules for connecting limb points of virtual characters and there are no other limb points in between the lines connecting the two target limb points.

[0177] For example, such as Figure 9 As shown, the rules for connecting limb points are as follows:

[0178] Head using limb point: 0;

[0179] The left shoulder is connected using limb points: 1–>3–>5;

[0180] The line connecting the right shoulder uses the following limb points: 2–>4–>6;

[0181] The shoulder line is drawn using limb points: 1–>2;

[0182] The waistline is connected using limb points: 7–>8;

[0183] The left leg is connected using limb points: 8–>10–>12;

[0184] The line connecting the right leg uses the following limb points: 7–>9–>11;

[0185] The left side of the body is connected using limb points: 2–>8;

[0186] The line connecting the right side of the body uses limb points: 1–>7.

[0187] Taking the right shoulder (1) as an example, its adjacent limb points are the right elbow (3), right hip (7), and left shoulder (2). The right shoulder (1) and head (0) do not conform to the limb point connection rules; therefore, they are not considered adjacent limb points. The right elbow (3) is contained between the right shoulder (1) and right wrist (5); therefore, they are also not considered adjacent limb points.

[0188] For example, the limb point information is coordinate information. The coordinate information of the right shoulder limb point is (x1, y1), the coordinate information of the right elbow limb point is (x2, y2), and the first length of the target bone is l = (x2 - x1). 2 +(y2-y1) 2 .

[0189] The second length of the target bone is calculated based on the first target limb point information and the second target limb point information in the second limb point information;

[0190] For example, the limb point information is coordinate information. The coordinate information of the right shoulder limb point is (X1, Y1), the coordinate information of the right elbow limb point is (X2, Y2), and the second length L of the target bone is (X2-X1). 2 +(Y2-Y1) 2 .

[0191] The scaling ratio is determined to be the ratio of the second length to the first length.

[0192] For example, the scaling factor s = L / l.

[0193] In the embodiments of this application, the first target limb point and the second target limb point can be any two limb points that are adjacent to each other.

[0194] In this embodiment, the display ratio between the video character and the user is determined by the connection length of two adjacent target limb points. There are no special requirements for the user's posture. That is, the accurate scaling ratio can still be determined when the user is standing, sitting or lying down. Then, a virtual character with the same scale as the video character is drawn by using the scaling ratio and the user's actual limb point information, so that the user can more intuitively feel his own movement state.

[0195] In some embodiments, the step of calculating the scaling ratio based on the first limb point information and the second limb point information includes:

[0196] Based on the first limb point information, the lengths of at least two different first target bones are determined, where the first target bone is the line connecting two adjacent target limb points in the first limb point information.

[0197] Selecting multiple different bones as the first target bone, the more bones selected, the more accurate the calculated scaling ratio will be.

[0198] For example, the bones of the arm and leg are selected as the first target bones. The bones of the arm include the upper arm (right shoulder to right elbow) and the forearm (right elbow to right wrist), and the bones of the leg include the femur (right hip to right knee) and the tibia (right knee to right ankle). The lengths of the upper arm (l1), forearm (l2), femur (l3), and tibia (l4) are calculated using the coordinates of two limb points.

[0199] Calculate the sum of the lengths of the at least two different first target bones;

[0200] For example, the sum of the lengths of the first target bones is l = l1 + l2 + l3 + l4.

[0201] The lengths of at least two different second target bones are determined based on the second limb point information. The second target bone is the line connecting two target limb points in the second limb point information. The first target bone is associated with the second target bone.

[0202] The association between the first target bone and the second target bone means that the bones at the same location are selected.

[0203] For example, the bones of the arm and leg are selected as the second target bones. The bones of the arm include the upper arm (right shoulder to right elbow) and the forearm (right elbow to right wrist), and the bones of the leg include the femur (right hip to right knee) and the tibia (right knee to right ankle). The lengths of the upper arm (L1), forearm (L2), femur (L3), and tibia (L4) are calculated using the coordinates of the two limb points.

[0204] Calculate the sum of the lengths of the at least two different second target bones;

[0205] For example, the sum of the lengths of the first target skeleton is L = L1 + L2 + L3 + L4.

[0206] The scaling ratio is determined to be the ratio of the sum of the lengths of the second target bones to the sum of the lengths of the first target bones.

[0207] For example, the scaling factor s = L / l.

[0208] Since different people may have different body proportions—for example, some people have relatively long calves while others have relatively short calves—calculating the scaling ratio based on the calf bones (tibia) may lead to inaccurate scaling. In this embodiment, the display ratio between the video character and the user is determined by summing the lengths of multiple target bones. This eliminates specific requirements for the user's posture and minimizes errors caused by differences in individual body proportions, resulting in a more accurate scaling ratio. Furthermore, by using this scaling ratio and the user's actual limb point information, a virtual character proportional to the video character is drawn, allowing the user to more intuitively perceive their own movement.

[0209] Step S506: Draw a virtual character based on the first limb point information, the positioning limb point information, and the scaling ratio;

[0210] In some embodiments, the step of drawing a virtual character based on the first limb point information, the positioned limb point information, and the scaling ratio includes:

[0211] The third limb point information of the scaled virtual character is determined based on the first limb point information and the scaling ratio.

[0212] The first limb point information includes the coordinate information of 13 limb points, and the coordinates of the 13 limb points are coord_src = [(x0, y0), (x1, y1), (x2, y2), ..., (x12, y12)].

[0213] The scaling information of the third limb point of the virtual character, that is, the coordinates of the 13 limb points after scaling, coord_temp = s * coord_src, where s is the scaling ratio.

[0214] Determine the horizontal and vertical coordinate offsets;

[0215] The fourth limb point information of the virtual character is determined based on the horizontal coordinate offset, the vertical coordinate offset, and the third limb point information;

[0216] The fourth limb point information of the virtual character includes the horizontal and vertical coordinates of 13 limb points of the virtual character. Specifically, the horizontal coordinate of the 13 limb points is coord_des(x) = coord_temp(x) + offset_x; where offset_x is the horizontal coordinate offset.

[0217] The ordinates of the 13 limbs of the virtual character are coord_des(y) = coord_temp(y) + offset_y; where offset_y is the offset of the ordinate.

[0218] The virtual character is drawn based on the information of the fourth limb point.

[0219] Based on the coordinate information of the fourth limb point, limb points are drawn at their corresponding positions. A circle is drawn with the nose coordinates as the center and a fixed value as the radius to draw the head of the virtual character. The limb points are then connected with simple lines according to the connection rules, and the lines are thickened to generate a virtual character image like a stick figure. The character image can also be filled based on the above limb point connection rules to generate a cartoon-style virtual character image.

[0220] In some embodiments, the step of determining the horizontal axis offset and the vertical axis offset includes:

[0221] The horizontal coordinate offset is determined based on the information of the first limb point and the information of the third limb point.

[0222] The step of determining the horizontal coordinate offset based on the first limb point information and the third limb point information includes:

[0223] The x-coordinates of the first target limb point and the second target limb point are filtered out from the first limb point information, and the positions of the first target limb point and the second target limb point are adjacent.

[0224] The x-coordinates of the first target limb and the x-coordinates of the second target limb are selected from the third limb point information;

[0225] Based on the x-coordinates of the first and second target limbs in the first limb point information, and the x-coordinates of the first and second target limbs in the second limb point information, the x-coordinate offset is determined.

[0226] For example, in the first limb point information, the coordinates of the right shoulder of the limb point are (x1, y1), and the coordinates of the right elbow of the limb point are (x3, y3). In the third limb point information, the coordinates of the right shoulder of the limb point are (X1, Y1), and the coordinates of the right elbow of the limb point are (X3, Y3). The horizontal coordinate offset_x = (X1 + X3) / 2 - (x1 + x3) / 2.

[0227] In this embodiment of the application, when calculating the product of the coordinates of the 13 limb points in the first limb point information and the scaling ratio, the center of the virtual character may shift to one side. For example, when establishing a Cartesian coordinate system with the upper left corner of the display device as the origin, the virtual character will shift to the lower right corner. Therefore, the center of the virtual character can be restored to the user's actual center by calculating the above horizontal coordinate offset.

[0228] In some embodiments, the horizontal coordinate offset may not be calculated, that is, the horizontal coordinate of the third limb point may not be corrected.

[0229] The vertical coordinate offset is determined based on the location limb point information and the third limb point information.

[0230] The step of determining the vertical coordinate offset based on the positioning limb point information and the third limb point information includes:

[0231] Obtain the ordinate of the limb point in the limb point information and the ordinate of the limb point corresponding to the limb point in the third limb point information; wherein, the limb point corresponding to the limb point refers to the limb point at the same position as the limb point.

[0232] Calculate the difference between the ordinate of the limb point corresponding to the location limb point and the ordinate of the location limb point in the third limb point information.

[0233] For example, the limb point is the right ankle, and the coordinates of the right ankle are (x11, y11). In the third limb point information, the coordinates of the right ankle are (x11, y11). The vertical coordinate offset is offset_y = y11 - y11.

[0234] The virtual characters drawn in the embodiments of this application are as follows: Figure 11As shown. In this embodiment, only the vertical coordinate of the virtual character is calibrated, so that the virtual character and the video character are at the same horizontal level. The user can adjust the positional relationship with the video character by moving left or right. The user can choose to be parallel to or overlap with the video character.

[0235] In some embodiments, the step of determining the horizontal axis offset and the vertical axis offset includes:

[0236] The horizontal and vertical coordinate offsets are determined based on the location limb point information and the third limb point information.

[0237] The location limb point information includes a first horizontal coordinate and a first vertical coordinate. The step of determining the horizontal coordinate offset and the vertical coordinate offset based on the location limb point information and the third limb point information includes:

[0238] Obtain the coordinate information of the limb point corresponding to the positioning limb point from the third limb point information, wherein the coordinate information includes a second horizontal coordinate and a second horizontal coordinate.

[0239] The horizontal coordinate offset is determined to be the difference between the second horizontal coordinate and the first horizontal coordinate;

[0240] The vertical coordinate offset is determined to be the difference between the second vertical coordinate and the first vertical coordinate.

[0241] For example, the limb point is located as the nose, and the coordinates of the nose are (x0, y0). In the third limb point information, the coordinates of the nose are (X0, Y0). The horizontal coordinate offset offset_x = X0 - x0, and the vertical coordinate offset offset_y = Y0 - y0.

[0242] The virtual characters drawn in the embodiments of this application are as follows: Figure 12 As shown in the illustration, the horizontal and vertical coordinates of the virtual character are calibrated in this embodiment to ensure that the virtual character coincides with the video character. Regardless of the user's movement, the virtual character will be displayed proportionally and at the same horizontal level as the video character.

[0243] It should be added that the position of the characters in the video or the characters themselves may change during video playback. Therefore, the step of acquiring the second image data of the video at fixed intervals and determining the second limb point information of the characters in the video based on the second image data is performed. For example, the second image data of the video is acquired every 30 seconds, and the second limb point information of the characters in the video is determined based on the second image data. The scaling ratio and the positioning of the limb point information are then re-determined to redraw the virtual character.

[0244] In some embodiments, after acquiring the second image data of the video, the method further includes:

[0245] Collect the pixel colors of the second image data;

[0246] Specifically, the Palette interface is used to collect pixel colors from the second image data. Palette is a new feature in the Android L SDK (Software Development Kit), which allows extracting prominent pixel colors (dominant hues) from the second image data and obtaining the RGB (red, green, blue) proportions in the second image data. The values ​​for red, green, and blue range from 0 to 255.

[0247] The color of the virtual character is determined based on the pixel color, so that the color of the virtual character has a certain color value difference with the main color tone in the second image data, thereby making the virtual character stand out.

[0248] In some embodiments, the pixel color includes a first red value, a first green value, and a first blue value. The step of determining the color of the virtual character based on the pixel color includes:

[0249] Determine if the first red value is greater than the preset value;

[0250] If the first red value is greater than a preset value, then the second red value is determined to be the difference between the first red value and the preset value;

[0251] If the first red value is not greater than a preset value, then the second red value is determined to be the sum of the first red value and the preset value.

[0252] Determine if the first green value is greater than the preset value;

[0253] If the first green value is greater than a preset value, then the second green value is determined to be the difference between the first green value and the preset value;

[0254] If the first green value is not greater than a preset value, then the second green value is determined to be the sum of the first green value and the preset value.

[0255] Determine if the first blue value is greater than a preset value;

[0256] If the first blue value is greater than a preset value, then the second blue value is determined to be the difference between the first blue value and the preset value;

[0257] If the first blue value is not greater than a preset value, then the second blue value is determined to be the sum of the first blue value and the preset value.

[0258] The colors of the virtual character are determined to be a second red value, a second green value, and a second blue value.

[0259] For example, the primary color of the second image data is obtained, with a first red value of 210, a first green value of 20, a first blue value of 150, and a preset value of 128. Then, the second red value is 210-128=82, the second green value is 20+128=148, and the second blue value is 150-128=22. Therefore, the color of the virtual character is the second red value of 82, the second green value of 148, and the second blue value of 22.

[0260] In some embodiments, different colors can be applied to different parts of the virtual character. For example, the head, torso, and limbs of the virtual character can be set to different colors. Taking red as an example, if a first red value is greater than a preset value, then the red value of the virtual character's head is determined to be the difference between the first red value and a first fixed value; the red value of the virtual character's torso is determined to be the difference between the first red value and a second fixed value; and the red value of the virtual character's limbs is determined to be the difference between the first red value and a preset value. The first fixed value is less than the second fixed value, and the second fixed value is less than the preset value. If the first red value is not greater than the preset value, then the red value of the virtual character's head is determined to be the sum of the first red value and the first fixed value; the red value of the virtual character's torso is determined to be the sum of the first red value and the second fixed value; and the red value of the virtual character's limbs is determined to be the sum of the first red value and the preset value.

[0261] It should be noted that the background of the image may change during video playback. Therefore, the step of collecting the pixel colors of the second image data is performed at fixed intervals. For example, the pixel colors of the second image data are collected every 30 seconds, and the colors of the virtual character are re-determined and redrawn based on the pixel colors.

[0262] In some embodiments, after acquiring the second image data of the video, the method further includes:

[0263] The target area of ​​the second image data is extracted to obtain the third image data, wherein the target area is the area where the person in the video is located;

[0264] Among them, after determining the second limb point information of the person in the video based on the second image data, the maximum horizontal coordinate value xmax, the minimum horizontal coordinate value xmin, the maximum vertical coordinate value ymax, and the minimum vertical coordinate value ymin in the second limb point information are determined;

[0265] A closed region can be determined in the second image based on x = xmax, x = xmin, y = ymax, and y = ymin, and the closed region is the target region.

[0266] Collect the pixel colors of the third image data;

[0267] The color of the virtual character is determined based on the pixel colors of the third image data.

[0268] This application's embodiments are applicable to situations where a virtual character and a video character overlap. Since the video character cannot occupy a large area in the second image data, by removing the influence of the background color of the non-video character, the main color tone of the video character can be obtained as much as possible. The color of the virtual character determined based on this main color tone can form a more obvious color value difference, making the virtual character stand out.

[0269] Step S507: Display the virtual character in a floating layer on the upper layer of the video playback interface.

[0270] like Figure 13 As shown, the video playback screen and the virtual character are displayed in layers. The video plays in the video layer, which acts as the view container for displaying the video. The virtual character is displayed in a floating layer above the video layer and moves with the user. The virtual character is a custom view drawn using OpenGL (Open Graphics Library).

[0271] In some embodiments, such as Figure 14 As shown, the application layer includes a camera control module, a video control module, and an interface display control module. The framework layer includes a platform layer and an algorithm layer. The platform layer includes a camera service module, a media asset playback module, and a virtual character rendering module. The algorithm layer includes a limb detection module and a video background acquisition module. The camera control module sends the acquired first image data to the camera service module, which in turn sends the first image data to the limb detection module. The limb detection module processes the first image data to obtain the user's first limb point information and sends this information to the virtual character rendering module. The video control module controls video playback. After receiving the video stream data, the media asset playback module plays the video and sends the video stream data to the limb detection module and the video background acquisition module. The limb detection module processes the video stream data to obtain the second limb point information of the character in the video and sends it to the virtual character rendering module. The video background acquisition module processes the video stream data to obtain the main color tone of the image and sends it to the virtual character rendering module. The virtual character rendering module renders the virtual character based on the first limb point information, the second limb point information, and the main color tone of the image, and then sends the completed virtual character to the interface display control module. The interface display control module controls the display to show the virtual character.

[0272] Some embodiments of this application provide a method for displaying a virtual character. The method is applicable to a display device, which includes a monitor and a controller. The controller is configured to: respond to a user input command to play a video, play the video, and activate a camera to capture first image data; determine first limb point information of the user based on the first image data; acquire second image data of the video, and determine second limb point information of a character in the video based on the second image data, the second limb point information including positioning limb point information used to determine the position of the virtual character; calculate a scaling ratio based on the first limb point information and the second limb point information; draw the virtual character based on the first limb point information, the positioning limb point information, and the scaling ratio; and display the virtual character as a floating layer on the video playback interface. This application embodiment uses the limb point information of the video character and the user's limb point information captured by the camera to draw a virtual character proportional to the video character and display it at a designated location, allowing the user to more intuitively perceive their own movement state, facilitating the determination of the standardization of actions and timely adjustments to the actions.

[0273] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0274] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, include: monitor; Camera; The controller is configured as follows: In response to a user's instruction to play a video, the video is played, and the camera is turned on so that the camera can capture first image data; Based on the first image data, determine the user's first limb point information; The second image data of the video is acquired, and the second limb point information of the video character in the video is determined based on the second image data. The second limb point information includes positioning limb point information, which is used to determine the position of the virtual character. Calculate the scaling ratio based on the first limb point information and the second limb point information; The third limb point information of the scaled virtual character is determined based on the first limb point information and the scaling ratio. Determine the horizontal coordinate offset and the vertical coordinate offset, wherein the vertical coordinate offset is determined based on the positioning limb point information and the third limb point information; The fourth limb point information of the virtual character is determined based on the horizontal coordinate offset, the vertical coordinate offset, and the third limb point information; A virtual character is drawn based on the information of the fourth limb point; the virtual character is displayed in a floating layer on the upper layer of the video playback interface.

2. The display device according to claim 1, characterized in that, The controller, which determines the second limb point information of the person in the video based on the second image data, is further configured to: Obtain the limb point information of the person in the video from the second image data; If the number of limb points in the limb point information to be determined is greater than a preset number and the limb point information to be determined includes positioning limb point information, then the limb point information to be determined is determined as the second limb point information.

3. The display device according to claim 1, characterized in that, The controller performs the calculation of the scaling ratio based on the first limb point information and the second limb point information, and is further configured to: The first length of the target bone is calculated based on the first target limb point information and the second target limb point information in the first limb point information. The positions of the first target limb point and the second target limb point are adjacent. The target bone is the line connecting the first target limb point and the second target limb point. The second length of the target bone is calculated based on the first target limb point information and the second target limb point information in the second limb point information; The scaling ratio is determined to be the ratio of the second length to the first length.

4. The display device according to claim 1, characterized in that, The controller performs the calculation of the scaling ratio based on the first limb point information and the second limb point information, and is further configured to: Based on the first limb point information, the lengths of at least two different first target bones are determined, where the first target bone is the line connecting two adjacent target limb points in the first limb point information. Calculate the sum of the lengths of the at least two different first target bones; The lengths of at least two different second target bones are determined based on the second limb point information. The second target bone is the line connecting two target limb points in the second limb point information. The first target bone is associated with the second target bone. Calculate the sum of the lengths of the at least two different second target bones; The scaling ratio is determined to be the ratio of the sum of the lengths of the second target bones to the sum of the lengths of the first target bones.

5. The display device according to claim 1, characterized in that, The controller performs the task of determining the horizontal coordinate offset and is further configured to: The horizontal coordinate offset is determined based on the information of the first limb point and the information of the third limb point.

6. The display device according to claim 5, characterized in that, The controller, which determines the horizontal and vertical coordinate offsets, is further configured to: The horizontal and vertical coordinate offsets are determined based on the location limb point information and the third limb point information.

7. The display device according to claim 1, characterized in that, The controller is configured as follows: Collect the pixel colors of the second image data; The color of the virtual character is determined based on the pixel color.

8. The display device according to claim 7, characterized in that, The pixel color includes a first red value, a first green value, and a first blue value. The controller performs the function of determining the color of the virtual character based on the pixel color, and is further configured to: If the first red value, the first green value, or the first blue value is greater than a preset value, then the second red value, the second green value, or the second blue value is the difference between the first red value, the first green value, or the first blue value and the preset value. If the first red value, the first green value, or the first blue value is not greater than a preset value, then the second red value, the second green value, or the second blue value is the sum of the first red value, the first green value, or the first blue value and the preset value; The colors of the virtual character are determined to be a second red value, a second green value, and a second blue value.

9. A method for displaying virtual characters, characterized in that, include: In response to a user's instruction to play a video, the video is played, and the camera is turned on so that the camera can capture first image data; Based on the first image data, determine the user's first limb point information; The second image data of the video is acquired, and the second limb point information of the target video character in the video is determined based on the second image data. The second limb point information includes positioning limb point information, which is used to determine the position of the virtual character. Calculate the scaling ratio based on the first limb point information and the second limb point information; The third limb point information of the scaled virtual character is determined based on the first limb point information and the scaling ratio. Determine the horizontal coordinate offset and the vertical coordinate offset, wherein the vertical coordinate offset is determined based on the positioning limb point information and the third limb point information; The fourth limb point information of the virtual character is determined based on the horizontal coordinate offset, the vertical coordinate offset, and the third limb point information; A virtual character is drawn based on the information of the fourth limb point; The virtual character is displayed in a floating layer on top of the video playback interface.

Images