Image processing methods and apparatus

By identifying and scaling key human body points in a user's image on a display device to create a virtual animated character, the problem of uninteresting interactive methods in existing technologies is solved, and the vividness and frequency of user experience are enhanced.

CN115564875BActive Publication Date: 2026-11-17HISENSE VISUAL TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202211185970.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-27
Publication Date
2026-11-17
Estimated Expiration
2042-09-27

AI Technical Summary

Technical Problem

Existing interactive methods for realizing action-based activities through display devices lack fun, and the user image display methods are monotonous, failing to vividly and realistically demonstrate the difference between the user and the video actions.

Method used

By using target recognition algorithms to identify key human body points in user images, scaling them up proportionally, and then drawing virtual animated characters, the vividness and interactivity of user images are enhanced.

Benefits of technology

Transforming user images into virtual animated avatars increases the fun of the user's practice process and increases the frequency of interaction with the display device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115564875B_ABST
    Figure CN115564875B_ABST
Patent Text Reader

Abstract

This disclosure relates to an image processing method and apparatus, applied in the field of image processing technology. The method includes: acquiring an i-th frame of a user image, i = 0, 1, 2, ...; determining the initial coordinates of each human body key point in the i-th frame of the user image using a target recognition algorithm; scaling the initial coordinates of each human body key point according to a target ratio to obtain scaled coordinates of each human body key point, wherein the scaled coordinates of each human body key point are all located within the display area; and drawing a virtual animated image corresponding to the i-th frame of the user image based on the scaled coordinates of each human body key point. This allows the user image to be transformed into a cartoon image for display while the user is practicing with a demonstration video, making it more vivid and interesting, thereby increasing the frequency of interaction between the user and the display device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to image processing technology. More specifically, it relates to an image processing method and apparatus. Background Technology

[0002] As user demands increase, display devices are becoming more and more comprehensive in their functions. Among them, the interactive motion-based activities (such as dance practice, fitness, etc.) provided on display devices are loved by a wide range of users. Users can intuitively see the difference between their own movements and the movements in the video through the display device.

[0003] Existing interactive methods for action-based activities using display devices involve showing a demonstration video of the action while simultaneously displaying the captured user's posture. This is achieved by capturing and playing images of the user in real time using a camera. However, this method is rather simplistic, resembling a user practicing the actions in a demonstration video in front of a mirror. It lacks engagement and is not particularly engaging or vivid. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this application provides an image processing method and apparatus that can transform the user's image into a cartoon character display when the user follows along with a demonstration video, making it more vivid and interesting, thereby increasing the frequency of interaction between the user and the display device.

[0005] In a first aspect, embodiments of this application provide an image processing method, which includes: acquiring a user image of frame i, where i = 0, 1, 2, ...; determining the initial coordinates of each human key point in the user image of frame i using a target recognition algorithm; scaling the initial coordinates of each human key point according to a target ratio to obtain the scaled coordinates of each human key point, wherein the scaled coordinates of each human key point are all located within the display area; and drawing a virtual animated image corresponding to the user image of frame i based on the scaled coordinates of each human key point.

[0006] In some embodiments of this application, before scaling the initial coordinates of each human body key point according to the target ratio, the method further includes: determining a width ratio and a height ratio, wherein the width ratio is the ratio of the width of the area to be displayed to the width of the user image of the i-th frame, and the height ratio is the ratio of the height of the area to be displayed to the height of the user image of the i-th frame; and using the smaller value of the width ratio and the height ratio as the target ratio.

[0007] In some embodiments of this application, drawing a virtual animated image corresponding to the user image of the i-th frame based on the scaling coordinates of each human body key point includes: determining the valid key points and invalid key points among each human body key point based on the confidence level of the scaling coordinates of each human body key point, wherein the confidence level of the scaling coordinates of the valid key points is greater than or equal to a preset threshold, and the confidence level of the scaling coordinates of the invalid key points is less than the preset threshold; and drawing a virtual animated image corresponding to the user image of the i-th frame based on the scaling coordinates of the valid key points among each human body key point.

[0008] In some embodiments of this application, after determining the valid and invalid keypoints among the various human keypoints based on the confidence level of the scaling coordinates of each human keypoint, the method further includes: storing the keypoint information corresponding to the i-th frame user image in a target cache, wherein the keypoint information is used to indicate the valid and invalid keypoints among the various human keypoints, and the scaling coordinates of each valid keypoint; when i is greater than n, performing the following steps for each human keypoint of the i-th frame user image: determining whether the target keypoint is ultimately a valid keypoint based on the keypoint information of the target keypoint and the keypoint information of the keypoints corresponding to the target keypoint in the n frames of user images preceding the i-th frame user image, and determining the final coordinates of the target keypoint if the target keypoint is ultimately a valid keypoint; wherein n is an integer greater than 1, and the target keypoint is any one of the various human keypoints.

[0009] In some embodiments of this application, based on the key point information of the target key point and the key point information of the key points corresponding to the target key point in the n user images preceding the i-th frame, it is determined whether the target key point is ultimately a valid key point, and if the target key point is determined to be a valid key point, the final coordinates of the target key point are determined, including: if the target key point is a valid key point and the number of targets is greater than or equal to a number threshold, the target key point is determined to be a valid key point, and the scaled coordinates of the target key point are determined as the final coordinates of the target key point, where the number of targets is the number of valid key points among the key points corresponding to the target key point in the n user images; if the target key point is a valid key point and the number of targets is less than a number threshold, the final coordinates of the target key point are determined. In the case of invalid keypoints, the target keypoint is determined to be an invalid keypoint; in the case of invalid keypoints and the number of targets being greater than or equal to a threshold, the target keypoint is determined to be a valid keypoint, and the scaling coordinates of the first keypoint are determined as the final coordinates of the target keypoint. The first keypoint is the keypoint in the first user image that corresponds to the target keypoint. The first user image is the user image in m frames that is closest to the i-th frame user image. The m frames user images are the user images in n frames that correspond to the valid keypoints among the keypoints that correspond to the target keypoint. m is a positive integer less than or equal to n. In the case of invalid keypoints and the number of targets being less than a threshold, the target keypoint is determined to be an invalid keypoint.

[0010] In some embodiments of this application, before drawing the virtual animated image corresponding to the user image of the i-th frame based on the scaling coordinates of the effective key points among the various human body key points, the method further includes: setting the target knee key point as an invalid key point if the effective key points among the various human body key points also include a target knee key point, and the distance between the target knee key point and the lower boundary of the area to be displayed is less than or equal to a distance threshold; wherein, the target knee key point is a left knee key point or a right knee key point.

[0011] In some embodiments of this application, the virtual animated image includes a head and a torso. Valid key points among the various human body key points include the head key point, left shoulder key point, right shoulder key point, left hip key point, and right hip key point. Based on the scaling coordinates of the valid key points among the various human body key points, the virtual animated image corresponding to the i-th frame of the user image is drawn, including: drawing the virtual animated image corresponding to the i-th frame of the user image based on the scaling coordinates of the head key point, the left shoulder key point, the right shoulder key point, the left hip key point, and the right hip key point.

[0012] In some embodiments of this application, the head key points include: a nose key point; based on the scaling coordinates of the head key points, the scaling coordinates of the left shoulder key points, the scaling coordinates of the right shoulder key points, the scaling coordinates of the left hip key points, and the scaling coordinates of the right hip key points, drawing the virtual animation image corresponding to the i-th frame user image includes: drawing a first circle with the scaling coordinates of the nose key point as the center and the radius as a first radius to obtain the head of the virtual animation image, the head of the virtual animation image being the area corresponding to the first circle; determining four second circles with the scaling coordinates of the left shoulder key points, the scaling coordinates of the right shoulder key points, the scaling coordinates of the left hip key points, and the scaling coordinates of the right hip key points as the centers and the radius as a second radius, respectively, and drawing the external common tangent of any adjacent second circles to obtain the torso of the virtual animation image, the torso of the virtual animation image being the largest area in the area formed by each of the four external common tangents.

[0013] In some embodiments of this application, the effective key points among the various human body key points also include key points corresponding to the target arm, which include: shoulder key point, elbow key point, and wrist key point; based on the scaling coordinates of the effective key points among the various human body key points, drawing a virtual animation image corresponding to the user image of the i-th frame further includes: drawing three third circles with the scaling coordinates of the key points corresponding to the target arm as the center and the radius as the third radius, and drawing the external common tangent of any adjacent third circles to obtain the target arm of the virtual animation image, the target arm of the virtual animation image being a region composed of four external common tangents; and / or, the effective key points among the various human body key points also include key points corresponding to the target leg, which include: lap key point, knee key point, and ankle key point; based on the scaling coordinates of the effective key points among the various human body key points, drawing a virtual animation image corresponding to the user image of the i-th frame further includes: drawing three fourth circles with the scaling coordinates of the key points corresponding to the target leg as the center and the radius as the fourth radius, and drawing the external common tangent of any adjacent fourth circles to obtain the target leg of the virtual animation image, the target leg of the virtual animation image being a region composed of four external common tangents.

[0014] Secondly, this application provides an image processing apparatus, comprising: an acquisition module, a determination module, a scaling module, and a drawing module; the acquisition module is used to acquire an i-th frame of user image, i = 0, 1, 2, ...; the determination module is used to determine the initial coordinates of each human key point in the i-th frame of user image through a target recognition algorithm; the scaling module is used to scale the initial coordinates of each human key point according to a target ratio to obtain the scaled coordinates of each human key point, wherein the scaled coordinates of each human key point are all located within the display area; and the drawing module is used to draw a virtual animated image corresponding to the i-th frame of user image based on the scaled coordinates of each human key point.

[0015] Thirdly, this application provides a display device including a display, a memory, and a computer program stored in the memory and executable on a processor, wherein the computer program, when executed by the processor, implements the steps of the image processing method as described in the first aspect.

[0016] Fourthly, this application provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the image processing method as described in the first aspect.

[0017] Fifthly, this application provides a computer program product, comprising: when the computer program product is run on a computer, causing the computer to implement the image processing method as shown in the first aspect.

[0018] Compared with the prior art, the technical solution provided in this application has the following advantages: In this application embodiment, the i-th frame of user image is acquired, i = 0, 1, 2...; the initial coordinates of each human key point in the i-th frame of user image are determined by a target recognition algorithm; the initial coordinates of each human key point are scaled according to the target ratio to obtain the scaled coordinates of each human key point, and the scaled coordinates of each human key point are all located within the display area; based on the scaled coordinates of each human key point, a virtual animated image corresponding to the i-th frame of user image is drawn. In this way, the acquired user image is transformed into a virtual animated image for display, which is more vivid and engaging, enhancing the user's enjoyment during practice and increasing the frequency of interaction between the user and the display device. Attached Figure Description

[0019] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 A schematic diagram illustrating an operational scenario between a display device and a control device according to some embodiments is shown;

[0021] Figure 2 A hardware configuration block diagram of a control device 100 according to some embodiments is shown;

[0022] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0023] Figure 4 One of the schematic flowcharts of an image processing method according to some embodiments is shown;

[0024] Figure 5 A schematic diagram of a frame of user image captured according to some embodiments is shown;

[0025] Figure 6 A schematic diagram of key points corresponding to a frame of a user image is shown according to some embodiments;

[0026] Figure 7 A schematic diagram showing the area to be displayed and the display area of ​​the scaled user image according to some embodiments is shown;

[0027] Figure 8a A schematic diagram showing a cartoon image corresponding to a landscape display demonstration video and a user graphic, according to some embodiments, is provided.

[0028] Figure 8b A schematic diagram showing a cartoon image corresponding to a vertical display demonstration video and a user graphic, according to some embodiments, is provided.

[0029] Figure 8c A schematic diagram is shown showing a cartoon image corresponding to a full-screen display of a user graphic according to some embodiments;

[0030] Figure 9 A second schematic flowchart of an image processing method according to some embodiments is shown;

[0031] Figure 10 A third schematic flowchart of an image processing method according to some embodiments is shown;

[0032] Figure 11 A fourth schematic flowchart of an image processing method according to some embodiments is shown;

[0033] Figure 12 Fifth of a series of schematic flowcharts illustrating an image processing method according to some embodiments;

[0034] Figure 13 A schematic flowchart of an image processing method according to some embodiments is shown as sixth;

[0035] Figure 14 A flowchart of an image processing method according to some embodiments is shown as diagram seven;

[0036] Figure 15 Eighth schematic flowchart of an image processing method according to some embodiments is shown;

[0037] Figure 16 A schematic outline of a virtual animated character according to some embodiments is shown;

[0038] Figure 17 A flowchart of an image processing method according to some embodiments is shown as diagram number nine;

[0039] Figure 18 This diagram illustrates a virtual animated character filled with color according to some embodiments;

[0040] Figure 19 A comparative diagram of user images and virtual animated avatars, collected according to some embodiments, is shown;

[0041] Figure 20 A schematic diagram of the structure of an image processing apparatus according to some embodiments is shown. Detailed Implementation

[0042] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0043] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0044] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0045] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0046] The display device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, a mobile phone, a tablet computer, a laptop computer, a handheld computer, an in-vehicle electronic device, etc.

[0047] The image processing method provided in this application can be used in display devices. Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device according to an embodiment, wherein the control device includes a smart device or a control apparatus. Figure 1As shown, the user can operate the display device 200 through the smart device 300 or the control device 100.

[0048] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0049] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0050] In some embodiments, the display device may receive instructions not through the aforementioned smart devices or control devices, but through touch or gestures.

[0051] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0052] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0053] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, an external memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0054] like Figure 3The display device 200 includes at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a user interface 280, an external memory, and a power supply.

[0055] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to an nth interface for input / output.

[0056] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0057] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0058] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0059] User interface 280 can be used to receive control signals from control device 100 (such as an infrared remote control). It can also be used to directly receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to; in this case, it can be called a user input interface.

[0060] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0061] The external device interface 240 may include, but is not limited to, one or more of the following: High-definition multimedia interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0062] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0063] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0064] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory (internal or external memory). The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can perform operations related to the object selected by the user command.

[0065] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), and random access memory (RAM), read-only memory (ROM), a first to an nth interface for input / output, a communication bus, etc.

[0066] RAM, also known as main memory, is an internal memory that directly exchanges data with the controller. It can be read and written at any time (except during refresh) and is very fast, typically serving as temporary data storage for the operating system or other running programs. Its biggest difference from ROM is data volatility; data stored in RAM is lost when power is off. RAM is used in computers and digital systems to temporarily store programs, data, and intermediate results. ROM operates in a non-destructive read-only manner; information can only be read, not written. Once information is written, it is fixed and will not be lost even if power is cut off; therefore, it is also called fixed-function memory.

[0067] Users can input commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0068] A "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of a display device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0069] This application provides an image processing method, which is described below in conjunction with... Figure 4 The image processing method provided in this application embodiment will be described exemplarily by taking the application of this method to a display device as an example. The method may include the following steps 401 to 404.

[0070] 401. Acquire the user image in the i-th frame, where i = 0, 1, 2, ...

[0071] It is understood that the i-th frame of user image collected is the image corresponding to the user's current posture captured by the camera of the display device. The frequency of collection is determined according to actual needs, and this application embodiment does not limit it.

[0072] 402. Determine the initial coordinates of each key point of the human body in the i-th frame of the user image using a target recognition algorithm.

[0073] It is understood that the target recognition algorithm can be a human key point recognition algorithm (i.e., a human joint point recognition algorithm), or a human key point recognition algorithm and a face recognition algorithm (identifying key points of the torso and limbs through the human key point recognition algorithm and identifying key points of the face through the face recognition algorithm), or other algorithms that can identify human key points. This application does not limit the specific algorithms.

[0074] It is understandable that key points on the human body include: key points of the nose, left eye, right eye, mouth, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle, etc.

[0075] For example, such as Figure 5 As shown, this is a captured frame of user image. Inputting this user image into the target recognition algorithm yields the following result: Figure 6The key points shown in the current user image, along with their 2D coordinates: Label 0 indicates the nose key point (x0, y0); Label 1 indicates the left eye key point (x1, y1); Label 2 indicates the right eye key point (x2, y2); Label 3 indicates the left ear key point (x3, y3); Label 4 indicates the right ear key point (x4, y4); Label 5 indicates the left shoulder key point (x5, y5); Label 6 indicates the right shoulder key point (x6, y6); Label 7 indicates the left elbow key point (x7, y7); Label 8 indicates the right elbow key point (x7, y7). Key points, with coordinates (x8, y8), are as follows: 9 indicates the left wrist key point, with coordinates (x9, y9); 10 indicates the right wrist key point, with coordinates (x10, y10); 11 indicates the left hip key point, with coordinates (x11, y11); 12 indicates the right hip key point, with coordinates (x12, y12); 13 indicates the left knee key point, with coordinates (x13, y13); 14 indicates the right knee key point, with coordinates (x14, y14); 15 indicates the left ankle key point, with coordinates (x15, y5); and 16 indicates the right ankle key point, with coordinates (x16, y16).

[0076] It is understandable that target recognition algorithms can obtain the coordinates of key points on the human body and the confidence level of the identified key points. The confidence level is generally greater than 0 and less than 1. The larger the value, the more reliable the identification result.

[0077] 403. Scale the initial coordinates of each human body key point according to the target ratio to obtain the scaled coordinates of each human body key point.

[0078] The scaling coordinates of each key point on the human body are all located within the area to be displayed.

[0079] It is understandable that the initial coordinates of each human body key point are scaled according to the target ratio to obtain the scaled coordinates of each human body key point. In other words, the acquired user image is scaled so that the acquired user image can be displayed in the area to be displayed.

[0080] For example, such as Figure 7 As shown, the area indicated by label 700 is the display screen area of ​​the display device, the area indicated by label 701 is the area to be displayed on the display device, and the area indicated by label 703 is the image area obtained by connecting the scaled coordinates of each human body key point.

[0081] 404. Based on the scaling coordinates of each key point of the human body, draw the virtual animated image corresponding to the user image in the i-th frame.

[0082] It is understood that the drawn virtual animated character can be based on a graphics library, filling facial features with graphics from that library, or it can be drawn according to certain rules; this application's embodiments do not impose such limitations. For example... Figure 8a As shown, the image displayed on the right is a virtual animated character created by filling ellipses of different sizes with the coordinates of the key points of the mouth and nose. The graphics library stores various human body parts in cartoon shapes and various basic graphics, allowing for the creation of different cartoon characters as needed.

[0083] It's understandable that animated characters can be filled with various colors according to actual needs.

[0084] It is understandable that the area is displayed in a virtual animation, and the specific location can be determined based on the actual situation.

[0085] For example, such as Figure 8a As shown, the demonstration video and the cartoon image corresponding to the user's graphic are displayed simultaneously in landscape mode, such as... Figure 8b As shown, the demonstration video and the cartoon image corresponding to the user's graphic are displayed in portrait mode, such as... Figure 8c As shown, the cartoon image corresponding to the user's graphic is displayed in full screen.

[0086] In this embodiment, the i-th frame of the user image is acquired, where i = 0, 1, 2, ...; the initial coordinates of each key human body point in the i-th frame of the user image are determined using a target recognition algorithm; the initial coordinates of each key human body point are scaled according to a target ratio to obtain scaled coordinates of each key human body point, and the scaled coordinates of each key human body point are all located within the display area; based on the scaled coordinates of each key human body point, a virtual animated image corresponding to the i-th frame of the user image is drawn. In this way, the acquired user image is transformed into a virtual animated image for display, making it more vivid and engaging, increasing the user's enjoyment during practice, and thus increasing the frequency of user interaction with the display device.

[0087] In some embodiments of this application, combined with Figure 4 ,like Figure 9 As shown, prior to step 403 above, the method provided in this application embodiment further includes steps 405 and 406 as follows.

[0088] 405. Determine the width ratio and height ratio.

[0089] The width ratio is the ratio of the width of the area to be displayed to the width of the user image in the i-th frame, and the height ratio is the ratio of the height of the area to be displayed to the height of the user image in the i-th frame.

[0090] 406. Use the smaller of the width ratio and the height ratio as the target ratio.

[0091] Understandable, such as Figure 6 and Figure 7 As shown, the width of the area to be displayed is rw, the width of the area to be displayed is rh, the width of the user image in the i-th frame is dw, the width of the user image in the i-th frame is dh, and the target scale is: r = Math.min(rw / dw, rh / dh). Then, the width w of the user image in the i-th frame after scaling is r × dw, and the height h of the user image in the i-th frame after scaling is r × dh.

[0092] Optionally, the target ratio can be the smallest of the width ratio, height ratio, and preset ratio. The preset ratio can be a fixed value that yields a better display effect after scaling, determined based on historically collected user images and the area to be displayed. The specific target ratio is not limited in this embodiment.

[0093] In this embodiment, a width ratio and a height ratio are determined. The width ratio is the ratio of the width of the area to be displayed to the width of the user image in the i-th frame, and the height ratio is the ratio of the height of the area to be displayed to the height of the user image in the i-th frame. The smaller of the width ratio and the height ratio is used as the target ratio. The target ratio calculated in this way allows the captured user image to be displayed completely in the area to be displayed, thus improving the user's viewing experience.

[0094] In some embodiments of this application, combined with Figure 4 ,like Figure 10 As shown, step 404 can be specifically implemented through steps 404a and 404b.

[0095] 404a. Based on the confidence level of the scaling coordinates of each human body keypoint, determine the valid and invalid keypoints among each human body keypoint.

[0096] Among them, the confidence level of the scaling coordinates of valid key points is greater than or equal to a preset threshold, while the confidence level of the scaling coordinates of invalid key points is less than a preset threshold.

[0097] It is understandable that the preset threshold is determined according to actual needs, and it can generally be set to 0.2.

[0098] It is understandable that if keypoints are identified as invalid keypoints, then there is no need to draw invalid keypoints during the drawing process.

[0099] 404b. Based on the scaling coordinates of the effective key points in each human body key point, draw the virtual animated image corresponding to the user image in the i-th frame.

[0100] In some embodiments of this application, based on the confidence level of the scaling coordinates of each human body keypoint, valid and invalid keypoints are determined. The confidence level of the scaling coordinates of valid keypoints is greater than or equal to a preset threshold, while the confidence level of the scaling coordinates of invalid keypoints is less than the preset threshold. Based on the scaling coordinates of the valid keypoints, a virtual animated image corresponding to the user image in the i-th frame is drawn. Thus, when the confidence level of the coordinates of a keypoint determined by the target recognition algorithm is low, it indicates that the accuracy of the determined coordinates of that keypoint is insufficient. Therefore, removing invalid keypoints during drawing can improve the accuracy of the drawing and ensure that the drawn animated image is closer to the user's real-time posture.

[0101] In some embodiments of this application, combined with Figure 10 ,like Figure 11 As shown, after step 404a above, the method provided in this application embodiment further includes the following steps 404c, 404d, 404d1, and 404d2.

[0102] 404c. Store the key point information corresponding to the user image of the i-th frame into the target buffer.

[0103] Among them, key point information is used to indicate the valid and invalid key points in each human body key point, as well as the scaling coordinates of each valid key point.

[0104] It is understandable that the target buffer is used to store the key point information corresponding to each frame of user image captured. The key point information in each frame includes the same key points. For example, if the human body includes 17 key points, the captured image may include all 17 key points, or it may only include some of the key points. When storing, the key points that are not included are stored as invalid key points.

[0105] It is understandable that during storage, invalid and valid keypoints can be distinguished based on their coordinates or different markers. For example, the scaled coordinates of valid keypoints are the actual values, while the coordinates of invalid keypoints are (-1, -1). In other words, the validity of a keypoint can be determined by its coordinates.

[0106] 404d. Determine if i is greater than n.

[0107] If yes, then perform steps 404d1 and 404d2 for each human body key point in the i-th frame of the user image; otherwise, acquire the next frame of the user image.

[0108] It is understandable that, starting from the acquisition of user images, the sequence number i of each frame is recorded sequentially, and n is a preset value greater than 1.

[0109] 404d1. Based on the key point information of the target key point and the key point information of the key points corresponding to the target key point in the user images before the i-th frame, determine whether the target key point is ultimately a valid key point.

[0110] It can be understood that the key point corresponding to the target key point is the same as the human joint point indicated by the target key point. For example, if the target key point in the i-th frame of the user image is the nose key point, then the key points corresponding to the target key point are: the nose key point in the (i-1)-th frame of the user image, the nose key point in the (i-2)-th frame of the user image, etc.

[0111] It can be understood that the target keypoint is any keypoint among all keypoints in the i-th frame of the user image. For any keypoint, whether it is a valid keypoint is determined by combining the n frames preceding the i-th frame of the user image.

[0112] 404d2. If the target key points are determined to be valid key points, determine the final coordinates of the target key points.

[0113] Where n is an integer greater than 1, and the target key point is any one of the key points of the human body.

[0114] It is understandable that the coordinates of each human body key in the i-th frame of the user image can be used to determine whether each key point is a valid key point. Then, by combining the validity of each human body key in the n frames of the user image preceding the i-th frame, the validity of each human body key in the final i-th frame of the user image can be further determined.

[0115] In some embodiments of this application, the key point information corresponding to the i-th frame of the user image is stored in the target cache. The key point information is used to indicate the valid and invalid key points among the various human body key points, as well as the scaling coordinates of each valid key point. When i is greater than n, the following steps are performed for each human body key point of the i-th frame of the user image: based on the key point information of the target key point and the key point information of the key points corresponding to the target key point in the n frames of user images preceding the i-th frame of the user image, it is determined whether the target key point is ultimately a valid key point, and if the target key point is determined to be a valid key point, the final coordinates of the target key point are determined; where n is an integer greater than 1, and the target key point is any one of the various human body key points. In this way, by using the n frames of user images preceding the i-th frame of the user image to determine whether each human body key point in the i-th frame of the user image is ultimately valid, the stability of the drawn cartoon character can be improved, and the flickering phenomenon of the points can be avoided.

[0116] In some embodiments of this application, combined with Figure 11 ,like Figure 12As shown, steps 404d1 and 404d2 can be implemented through steps 404d3 to 404d9.

[0117] 404d3. Determine whether the target key points are valid key points.

[0118] If yes, proceed to step 404d4; otherwise, proceed to step 404d7.

[0119] 404d4. Determine whether the target quantity is less than the quantity threshold.

[0120] If yes, proceed to step 404d5; otherwise, proceed to step 404d6.

[0121] 404d5. The target key point was ultimately determined to be an invalid key point.

[0122] 404d6. Determine the target key points as valid key points and set the scaling coordinates of the target key points as the final coordinates of the target key points.

[0123] The target number is the number of valid keypoints among the keypoints corresponding to the target keypoints in the n frames of user images.

[0124] 404d7. Determine whether the target quantity is less than the quantity threshold.

[0125] If yes, proceed to step 404d8; otherwise, proceed to step 404d9.

[0126] 404d8. The target key point was determined to be an invalid key point.

[0127] 404d9. Determine the target key points as the final valid key points, and determine the scaling coordinates of the first key point as the final coordinates of the target key points.

[0128] Wherein, the first key point is the key point in the first user image that corresponds to the target key point, the first user image is the user image in the m-frame user images that is closest to the i-th frame user image, and the m-frame user images are the user images in the n-frame user images that correspond to the valid key points among the key points that correspond to the target key point; m is a positive integer less than or equal to n.

[0129] It is understood that the quantity threshold can be any value less than n. Specifically, the quantity threshold can be a pre-set fixed value or it can be determined based on n, depending on actual needs. This application does not limit this. For example, if n is 10, the quantity threshold can be a fixed value of 7; if the quantity threshold is determined based on n and is half of n, then the quantity threshold is 5.

[0130] For example, if n is 10, the quantity threshold is 3, and i is 11, then among the first 10 user images, there are 6 user images (frames 1, 2, 3, 4, 7, and 9) whose key points are valid key points corresponding to the target key points. 6 is the target quantity. If the target quantity is greater than 3, then the first user image is the 9th user image, which is the user image closest to the 11th image among the 6 frames (i.e., m).

[0131] For example, the coordinates of invalid keypoints are represented by (-1, -1); i is 6, n is 5, the number threshold is 3, the target keypoint is the nose keypoint, and the target cache stores:

[0132] Keyframe 1 information (17 keyframes): [(x=708.0, y=138.0), (x=756.0, y=108.0)……]; Keyframe 2 information (17 keyframes): [(x=756.0, y=108.0), (x=642.0, y=86.0)……]; Keyframe 3 information (17 keyframes): [(x=806.0, y=122.0), (x=642.0, y=86.0)……] 0)……];Frame 4 key point information (17 key points): [(x=-1,y=-1), (x=642.0,y=86.0)……];Frame 5 key point information (17 key points): [(x=-1,y=-1), (x=642.0,y=86.0)……];The confirmed scaling coordinates of the key points in the 6th frame are: [(x=-1,y=-1), (x=642.0,y=86.0)……];Then the nose key point in the 6th frame The coordinates are (x = -1, y = -1), indicating that the nose keypoint is an invalid keypoint. Combining the user images before frame 6, the scaled coordinates of the valid keypoints among the nose keypoints are: in frame 1 (x = 708.0, y = 138.0), in frame 1 (x = 756.0, y = 108.0), and in frame 1 (x = 806.0, y = 122.0), meaning the target number is 3. In 3 out of 5 frames, the nose keypoint is a valid keypoint, thus meeting the threshold. A value of 3 indicates that the target key point is confirmed as a valid key point. The closest valid nose key point to the 6th frame is the 3rd frame. Therefore, the scaling coordinates of the nose key point in the key point information of the 3rd frame are determined as the scaling coordinates of the target key point. That is, the key point information corresponding to the 6th frame user image stored in the target buffer is: 6th frame key point information (17 key points): [(x=806.0,y=122.0), (x=642.0,y=86.0)……].

[0133] In some embodiments of this application, when the target keypoint is a valid keypoint and the target number is greater than or equal to a number threshold, the target keypoint is determined to be a valid keypoint, and the scaling coordinates of the target keypoint are determined as the final coordinates of the target keypoint. The target number is the number of valid keypoints among the keypoints corresponding to the target keypoint in n frames of user images. When the target keypoint is a valid keypoint and the target number is less than a number threshold, the target keypoint is determined to be an invalid keypoint. When the target keypoint is an invalid keypoint and the target number is greater than or equal to a number threshold, the target keypoint is determined to be a valid keypoint, and the scaling coordinates of the first keypoint are determined as the final coordinates of the target keypoint. The first keypoint is the keypoint in the first user image corresponding to the target keypoint. The first user image is the user image closest to the i-th user image in m frames of user images, and the m frames of user images are the user images in n frames of user images corresponding to the valid keypoints among the keypoints corresponding to the target keypoint. m is a positive integer less than or equal to n. When the target keypoint is an invalid keypoint and the target number is less than a number threshold, the target keypoint is determined to be an invalid keypoint. Thus, when a target key point in a captured frame of user image is invalid, the scaling coordinates of the target key point can be determined by combining the corresponding key points in the previous frame. Alternatively, valid target key points can be set to invalid, which can make the drawn animation image more stable and reduce the occurrence of flickering.

[0134] In some embodiments of this application, combined with Figure 10 ,like Figure 13 As shown, prior to step 404b above, the method of this application embodiment further includes the following step 404e.

[0135] 404e. Among the effective key points of each human body key point, the target knee key point is also included. If the distance between the target knee key point and the lower boundary of the area to be displayed is less than or equal to the distance threshold, the target knee key point is set as an invalid key point.

[0136] The target knee key point is either the left knee key point or the right knee key point.

[0137] It is understood that, among the effective key points of each human body key point, the left knee key point is also included. If the distance between the left knee key point and the lower boundary of the area to be displayed is less than or equal to a distance threshold, the left knee key point is set as an invalid key point. Alternatively, if the effective key points of each human body key point include the right knee key point, and the distance between the right knee key point and the lower boundary of the area to be displayed is less than or equal to a distance threshold, the right knee key point is set as an invalid key point. Alternatively, if the effective key points of each human body key point include both the right and left knee key points, and the distance between the right knee key point and the lower boundary of the area to be displayed is less than or equal to a distance threshold, both the right and left knee key points are set as invalid key points. Alternatively, if the effective key points of each human body key point include both the right and left knee key points, and the distance between the left knee key point and the lower boundary of the area to be displayed is less than or equal to a distance threshold, both the right and left knee key points are set as invalid key points. The specific settings are determined according to actual needs, and this application embodiment does not limit them.

[0138] It is understood that the distance threshold is determined based on the actual size of the area to be displayed, and this application embodiment does not limit it.

[0139] In some embodiments of this application, before drawing the virtual animated image corresponding to the user image in the i-th frame based on the scaled coordinates of the effective key points among the various human body key points, the method further includes: if the effective key points among the various human body key points also include a target knee key point, and the distance between the target knee key point and the lower boundary of the area to be displayed is less than or equal to a distance threshold, then the target knee key point is set as an invalid key point; wherein, the target knee key point is either the left knee key point or the right knee key point. Thus, when the user's position is relatively low, and the user's legs are not fully captured, or the legs are already out of the frame, there is no need to draw the legs, ensuring a better display effect for the animated image.

[0140] In some embodiments of this application, combined with Figure 10 ,like Figure 14 As shown, the virtual animated character includes a head and a torso. The effective key points among the various human body key points include the head key point, the left shoulder key point, the right shoulder key point, the left hip key point, and the right hip key point. The above step 404b can be specifically implemented through the following step 404b1.

[0141] 404b1. Based on the scaling coordinates of the head keypoint, the left shoulder keypoint, the right shoulder keypoint, the left hip keypoint, and the right hip keypoint, draw the virtual animation image corresponding to the user image in the i-th frame.

[0142] It is understood that the key points of the head include any of the following: the key point of the nose, the key point of the left eye, the key point of the right eye, and the key point of the mouth.

[0143] It is understandable that drawing the virtual animated character corresponding to the user image in the i-th frame can be based on a graphics library. This involves determining the head and torso of the target animated character, drawing the head centered on the nose, and drawing the torso in a target area. The target area is the region enclosed by the lines connecting the scaling coordinates of the left shoulder keypoint, the right shoulder keypoint, the left hip keypoint, and the right hip keypoint. Alternatively, the virtual animated character can be drawn based on preset rules; this embodiment does not limit the method.

[0144] In some embodiments of this application, the virtual animated avatar includes a head and a torso. Valid key points among the various human body key points include the head key point, left shoulder key point, right shoulder key point, left hip key point, and right hip key point. Based on the scaling coordinates of the valid key points among the various human body key points, the virtual animated avatar corresponding to the user image in the i-th frame is drawn, including: drawing the virtual animated avatar corresponding to the user image in the i-th frame based on the scaling coordinates of the head key point, left shoulder key point, right shoulder key point, left hip key point, and right hip key point. In this way, the animated avatar can be drawn based on the valid key points among the various human body key points, making the displayed posture more vivid and interesting.

[0145] In some embodiments of this application, key points of the head include: key points of the nose; combined with Figure 14 ,like Figure 15 As shown, step 404b1 can be implemented through steps 404b2 and 404b3.

[0146] 404b2. Draw a first circle with the scaling coordinates of the nose key points as the center and the first radius as the first radius to obtain the head of the virtual animation character.

[0147] The head of the virtual animated character corresponds to the area of ​​the first circle.

[0148] 404b3. Using the scaling coordinates of the left shoulder keypoint, the right shoulder keypoint, the left hip keypoint, and the right hip keypoint as centers and the second radius as the radius, determine four second circles. Draw the external common tangent of any adjacent second circles to obtain the torso of the virtual animation character.

[0149] Among them, the torso of the virtual animated character is the largest area in the region composed of every four external tangents.

[0150] Understandable, such as Figure 16As shown, with the center of the circle labeled 0, draw the first circle with the first radius. With the centers of the circles labeled 5, 6, 10, and 11, draw four second circles with the second radius. Connect the common tangents of the adjacent second circles to obtain the torso of the virtual animation character.

[0151] In some embodiments of this application, the head key points include: a nose key point; based on the scaling coordinates of the head key points, the scaling coordinates of the left shoulder key points, the scaling coordinates of the right shoulder key points, the scaling coordinates of the left hip key points, and the scaling coordinates of the right hip key points, the virtual animated image corresponding to the i-th frame user image is drawn, including: drawing a first circle with the scaling coordinates of the nose key point as the center and a first radius as the radius to obtain the head of the virtual animated image, the head of the virtual animated image being the area corresponding to the first circle; determining four second circles with the scaling coordinates of the left shoulder key points, the scaling coordinates of the right shoulder key points, the scaling coordinates of the left hip key points, and the scaling coordinates of the right hip key points as the centers and a second radius as the radius, respectively, and drawing the external common tangent of any adjacent second circles to obtain the torso of the virtual animated image, the torso of the virtual animated image being the largest area in the area formed by each of the four external common tangents. Thus, by drawing circles of different radii and connecting the external common tangents of adjacent circles, the animated image can be obtained, and the drawing process is simple.

[0152] In some embodiments of this application, the effective key points among the various human body key points also include key points corresponding to the target arm, and the key points corresponding to the target arm include: shoulder key point, elbow key point, wrist key point; and / or, the effective key points among the various human body key points also include key points corresponding to the target leg, and the key points corresponding to the target leg include: hip key point, knee key point, ankle key point;

[0153] After step 404b1 above, combined with Figure 14 ,like Figure 17 As shown, the method in this application embodiment further includes the following steps 404b4 and / or 404b5.

[0154] 404b4. Draw three third circles with the scaling coordinates of the key points corresponding to the target arm as the center and the radius as the third radius. Draw the external common tangent of any adjacent third circles to obtain the target arm of the virtual animation image.

[0155] The target arm of the virtual animated character is the area composed of four external tangents.

[0156] 404b5. Draw three fourth circles with the scaling coordinates of the key points corresponding to the target leg as the center and the radius as the fourth radius. Draw the external common tangent of any adjacent fourth circles to obtain the target leg of the virtual animation image.

[0157] The target leg of the virtual animated character is the area composed of four external tangents.

[0158] It can be understood that, using the scaling coordinates of the left shoulder, left elbow, and left wrist keypoints corresponding to the left arm as centers and the third radius as a radius, three third circles are drawn. The common external tangent of any adjacent third circles is then drawn to obtain the left standard arm of the virtual animation character. Similarly, using the scaling coordinates of the left shoulder, right elbow, and right wrist keypoints corresponding to the right arm as centers and the third radius as a radius, three third circles are drawn. The common external tangent of any adjacent third circles is then drawn to obtain the right standard arm of the virtual animation character. Likewise, using the scaling coordinates of the left hip, left knee, and left ankle keypoints corresponding to the left leg as centers and the fourth radius as a radius, three fourth circles are drawn. The common external tangent of any adjacent fourth circles is then drawn to obtain the left leg of the virtual animation character. Finally, using the scaling coordinates of the left hip, right knee, and right ankle keypoints corresponding to the right leg as centers and the fourth radius as a radius, three fourth circles are drawn. The common external tangent of any adjacent fourth circles is then drawn to obtain the right leg of the virtual animation character. For example... Figure 16 The image shown is an animated character including its limbs.

[0159] It's understandable that the left arm, right arm, left leg, and right leg of the animated character are drawn based on the coordinates of their corresponding key points. For example, if a captured image frame includes key points corresponding to the left arm and right leg, then the drawn animated character will include the left arm and right leg; if a captured image frame only includes key points corresponding to the left arm, then the drawn animated character will include the left arm.

[0160] It's understandable that animated characters can be filled with color, such as... Figure 18 As shown. Figure 19 The image shown is a comparison between a captured user frame and its corresponding animated character. The left side shows the captured user frame, and the right side shows the corresponding animated character.

[0161] In some embodiments of this application, the effective key points among the various human body key points also include key points corresponding to the target arm, which include: shoulder key point, elbow key point, and wrist key point; based on the scaling coordinates of the effective key points among the various human body key points, drawing a virtual animation image corresponding to the user image of the i-th frame further includes: drawing three third circles with the scaling coordinates of the key points corresponding to the target arm as the center and the radius as the third radius, and drawing the external common tangent of any adjacent third circles to obtain the target arm of the virtual animation image, the target arm of the virtual animation image being a region composed of four external common tangents; and / or, the effective key points among the various human body key points also include key points corresponding to the target leg, which include: lap key point, knee key point, and ankle key point; based on the scaling coordinates of the effective key points among the various human body key points, drawing a virtual animation image corresponding to the user image of the i-th frame further includes: drawing three fourth circles with the scaling coordinates of the key points corresponding to the target leg as the center and the radius as the fourth radius, and drawing the external common tangent of any adjacent fourth circles to obtain the target leg of the virtual animation image, the target leg of the virtual animation image being a region composed of four external common tangents.

[0162] In the embodiments of this application, such as Figure 20 As shown, an image processing device is provided, comprising: an acquisition module 2001, a determination module 2002, a scaling module 2003, and a drawing module 2004; the acquisition module 2001 is used to acquire the i-th frame of user image, i = 0, 1, 2, ...; the determination module 2002 is used to determine the initial coordinates of each human body key point in the i-th frame of user image through a target recognition algorithm; the scaling module 2003 is used to scale the initial coordinates of each human body key point according to a target ratio to obtain the scaled coordinates of each human body key point, wherein the scaled coordinates of each human body key point are all located within the display area; and the drawing module 2004 is used to draw a virtual animated image corresponding to the i-th frame of user image based on the scaled coordinates of each human body key point.

[0163] In some embodiments of this application, the determining module 2002 is further configured to determine a width ratio and a height ratio, wherein the width ratio is the ratio of the width of the area to be displayed to the width of the user image in the i-th frame, and the height ratio is the ratio of the height of the area to be displayed to the height of the user image in the i-th frame; and the smaller value of the width ratio and the height ratio is used as the target ratio.

[0164] In some embodiments of this application, the determining module 2002 is specifically used to determine the valid key points and invalid key points among the various human body key points based on the confidence level of the scaling coordinates of each human body key point, wherein the confidence level of the scaling coordinates of the valid key points is greater than or equal to a preset threshold, and the confidence level of the scaling coordinates of the invalid key points is less than the preset threshold; the drawing module 2004 draws a virtual animated image corresponding to the user image of the i-th frame based on the scaling coordinates of the valid key points among the various human body key points.

[0165] In some embodiments of this application, the device further includes: a storage module and an execution module; the storage module is used to determine the valid keypoints and invalid keypoints among the various human keypoints based on the confidence level of the scaling coordinates of each human keypoint, and then store the keypoint information corresponding to the i-th frame user image in a target cache, wherein the keypoint information is used to indicate the valid keypoints and invalid keypoints among the various human keypoints, and the scaling coordinates of each valid keypoint; the execution module is used to perform the following steps for each human keypoint of the i-th frame user image when i is greater than n: determining whether the target keypoint is ultimately a valid keypoint based on the keypoint information of the target keypoint and the keypoint information of the keypoints corresponding to the target keypoint in the n frames of user images preceding the i-th frame user image, and determining the final coordinates of the target keypoint if the target keypoint is ultimately a valid keypoint; wherein n is an integer greater than 1, and the target keypoint is any one of the various human keypoints.

[0166] In some embodiments of this application, the determining module 2002 is specifically used to: determine that a target keypoint is ultimately a valid keypoint when the target keypoint is a valid keypoint and the number of targets is greater than or equal to a quantity threshold; and determine the scaled coordinates of the target keypoint as the final coordinates of the target keypoint, where the target number is the number of valid keypoints among the keypoints corresponding to the target keypoint in n frames of user images; determine that a target keypoint is ultimately an invalid keypoint when the target keypoint is a valid keypoint and the number of targets is less than a quantity threshold; and determine that a target keypoint is ultimately an invalid keypoint when the target keypoint is an invalid keypoint and the number of targets is greater than or equal to a quantity threshold. In the case of a value, the target keypoint is determined to be a valid keypoint, and the scaling coordinates of the first keypoint are determined as the final coordinates of the target keypoint. The first keypoint is the keypoint in the first user image that corresponds to the target keypoint. The first user image is the user image in m frames that is closest to the i-th user image. The m frames are the user images in n frames that correspond to the valid keypoints among the keypoints that correspond to the target keypoint. m is a positive integer less than or equal to n. If the target keypoint is an invalid keypoint, or if the number of targets is less than the number threshold, the target keypoint is determined to be an invalid keypoint.

[0167] In some embodiments of this application, the device further includes: a setting module; the setting module is used to, before drawing the virtual animation image corresponding to the user image of the i-th frame based on the scaling coordinates of the effective key points among the various human body key points, set the target knee key point as an invalid key point if the effective key points among the various human body key points also include a target knee key point, and the distance between the target knee key point and the lower boundary of the area to be displayed is less than or equal to a distance threshold; wherein, the target knee key point is a left knee key point or a right knee key point.

[0168] In some embodiments of this application, the virtual animated image includes a head and a torso, and the effective key points among the various human body key points include the head key point, the left shoulder key point, the right shoulder key point, the left hip key point, and the right hip key point; the drawing module 2004 is specifically used to draw the virtual animated image corresponding to the i-th frame user image based on the scaling coordinates of the head key point, the left shoulder key point, the right shoulder key point, the left hip key point, and the right hip key point.

[0169] In some embodiments of this application, the head key points include: the nose key point; the drawing module 2004 is specifically used to draw a first circle with the scaling coordinates of the nose key point as the center and the radius as a first radius to obtain the head of the virtual animation image, the head of the virtual animation image being the area corresponding to the first circle; four second circles are determined with the scaling coordinates of the left shoulder key point, the scaling coordinates of the right shoulder key point, the scaling coordinates of the left hip key point, and the scaling coordinates of the right hip key point as the center and the radius as a second radius, respectively, and the external common tangent of any adjacent second circles is drawn to obtain the torso of the virtual animation image, the torso of the virtual animation image being the largest area in the area formed by each of the four external common tangents.

[0170] In some embodiments of this application, the effective key points among the various human body key points also include key points corresponding to the target arm, which include: shoulder key point, elbow key point, and wrist key point; the drawing module 2004 is specifically used to draw three third circles with the scaling coordinates of the key points corresponding to the target arm as the center and the radius as the third radius, and to draw the external common tangent of any adjacent third circles to obtain the target arm of the virtual animation image, the target arm of the virtual animation image being a region composed of four external common tangents; and / or, the effective key points among the various human body key points also include key points corresponding to the target leg, which include: lap key point, knee key point, and ankle key point; based on the scaling coordinates of the effective key points among the various human body key points, drawing the virtual animation image corresponding to the user image of the i-th frame also includes: drawing three fourth circles with the scaling coordinates of the key points corresponding to the target leg as the center and the radius as the fourth radius, and to draw the external common tangent of any adjacent fourth circles to obtain the target leg of the virtual animation image, the target leg of the virtual animation image being a region composed of four external common tangents.

[0171] In this embodiment, each module can implement the image processing method provided in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0172] This application embodiment also provides a display device, which may include: a display, a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they can implement the various processes of the image processing method provided in the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0173] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described image processing method and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0174] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0175] The present invention provides a computer program product, comprising: when the computer program product is run on a computer, causing the computer to implement the above-described image processing method.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0177] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. An image processing method, characterized in that, The method includes: Acquire the user image in the i-th frame, where i = 0, 1, 2, ...; The initial coordinates of each key human body point in the i-th frame of the user image are determined by a target recognition algorithm. The initial coordinates of each human body key point are scaled according to the target ratio to obtain the scaled coordinates of each human body key point. The scaled coordinates of each human body key point are all located within the display area. For the target key point in the i-th frame of the user image, determine whether the target key point is a valid key point in the current frame, and count the number of valid key points corresponding to the target key point in the n frames of the user image before the i-th frame; the target key point includes any one of the various human body key points; Based on whether the target keypoint is a valid keypoint in the current frame and whether the number of targets is greater than or equal to a quantity threshold, the final determination of whether the target keypoint is ultimately a valid keypoint and the final coordinates of the target keypoint are made, specifically including: If the number of targets is greater than or equal to the number threshold: if the target key point in the current frame is a valid key point, then the scaled coordinates of the target key point in the current frame are used as the final coordinates; if the target key point in the current frame is an invalid key point, then the target key point is determined to be a valid key point, and the scaled coordinates of the valid key point closest to the current frame in the n-frame user image are used as the final coordinates. If the target keypoint in the current frame is a valid keypoint or an invalid keypoint, and the number of targets is less than the number threshold, the target keypoint is ultimately determined to be an invalid keypoint. The final coordinates of the valid key points among the target key points are used as the scaling coordinates of each human body key point. Based on the scaling coordinates of each human body key point, a virtual animated image corresponding to the user image of the i-th frame is drawn.

2. The method according to claim 1, characterized in that, Before scaling the initial coordinates of each key human body point according to the target ratio, the method further includes: Determine the width ratio and height ratio, wherein the width ratio is the ratio of the width of the area to be displayed to the width of the user image in the i-th frame, and the height ratio is the ratio of the height of the area to be displayed to the height of the user image in the i-th frame; The smaller of the width ratio and the height ratio is taken as the target ratio.

3. The method according to claim 1, characterized in that, The process of drawing a virtual animated image corresponding to the i-th frame of the user image based on the scaling coordinates of each human body key point includes: Based on the confidence level of the scaled coordinates of each human body key point, the valid key points and invalid key points among the various human body key points are determined. The confidence level of the scaled coordinates of the valid key points is greater than or equal to a preset threshold, and the confidence level of the scaled coordinates of the invalid key points is less than the preset threshold. Based on the scaling coordinates of the effective key points among the various human body key points, a virtual animated image corresponding to the user image of the i-th frame is drawn.

4. The method according to claim 3, characterized in that, After determining the valid and invalid keypoints among the various human body keypoints based on the confidence level of their scaled coordinates, the method further includes: The key point information corresponding to the i-th frame user image is stored in the target cache area. The key point information is used to indicate the valid key points and invalid key points in each human body key point, as well as the scaling coordinates of each valid key point. When i is greater than n, the following steps are performed for each human key point in the i-th frame of the user image: Based on the key point information of the target key point and the key point information corresponding to the target key point in the n frames of user images preceding the i-th frame of user image, determine whether the target key point is ultimately a valid key point, and if the target key point is ultimately a valid key point, determine the final coordinates of the target key point. Where n is an integer greater than 1, and the target key point is any one of the various human body key points.

5. The method according to claim 4, characterized in that, The step of determining whether the target keypoint is ultimately a valid keypoint based on the keypoint information of the target keypoint and the keypoint information corresponding to the target keypoint in the n user images preceding the i-th frame of the user image, and determining the final coordinates of the target keypoint if it is determined to be a valid keypoint, includes: If the target key point is a valid key point and the number of targets is greater than or equal to the number threshold, the target key point is determined to be a valid key point in the end, and the scaling coordinates of the target key point are determined as the final coordinates of the target key point. The number of targets is the number of valid key points among the key points corresponding to the target key point in the n frames of user images. If the target key point is a valid key point and the number of targets is less than the number threshold, the target key point is ultimately determined to be an invalid key point. If the target keypoint is an invalid keypoint and the number of targets is greater than or equal to a threshold, the target keypoint is determined to be a valid keypoint, and the scaling coordinates of the first keypoint are determined as the final coordinates of the target keypoint. The first keypoint is the keypoint in the first user image that corresponds to the target keypoint. The first user image is the user image in m frames that is closest to the i-th frame user image. The m frames user images are the user images in the n frames that correspond to the valid keypoints among the keypoints corresponding to the target keypoint. m is a positive integer less than or equal to n. If the target key point is an invalid key point and the number of targets is less than the number threshold, the target key point is ultimately determined to be an invalid key point.

6. The method according to claim 3, characterized in that, Before drawing the virtual animated image corresponding to the i-th frame user image based on the scaling coordinates of the effective key points among the various human body key points, the method further includes: Among the effective key points of each human body key point, there is also the target knee key point. If the distance between the target knee key point and the lower boundary of the area to be displayed is less than or equal to the distance threshold, the target knee key point is set as an invalid key point. The target knee key point is either the left knee key point or the right knee key point.

7. The method according to any one of claims 3 to 6, characterized in that, The virtual animated character includes a head and a torso, and the effective key points among the various human body key points include the head key point, the left shoulder key point, the right shoulder key point, the left hip key point, and the right hip key point. The step of drawing a virtual animated image corresponding to the i-th frame of the user image based on the scaling coordinates of the effective key points among the various human body key points includes: Based on the scaling coordinates of the head key point, the scaling coordinates of the left shoulder key point, the scaling coordinates of the right shoulder key point, the scaling coordinates of the left hip key point, and the scaling coordinates of the right hip key point, the virtual animated image corresponding to the i-th frame user image is drawn.

8. The method according to claim 7, characterized in that, The head key points include: nose key points; the process of drawing the virtual animated image corresponding to the i-th frame user image based on the scaling coordinates of the head key points, the scaling coordinates of the left shoulder key points, the scaling coordinates of the right shoulder key points, the scaling coordinates of the left hip key points, and the scaling coordinates of the right hip key points includes: A first circle is drawn with the scaling coordinates of the key points of the nose as the center and the radius as the first radius, so as to obtain the head of the virtual animation image. The head of the virtual animation image is the area corresponding to the first circle. Using the scaling coordinates of the left shoulder keypoint, the right shoulder keypoint, the left hip keypoint, and the right hip keypoint as centers and the second radius as radii, four second circles are determined. The external common tangent of any adjacent second circles is drawn to obtain the torso of the virtual animation image. The torso of the virtual animation image is the largest area in the region formed by each of the four external common tangents.

9. The method according to claim 7, characterized in that, The effective key points among the various human body key points also include the key points corresponding to the target arm, which include: shoulder key points, elbow key points, and wrist key points. The step of drawing a virtual animated image corresponding to the i-th frame user image based on the scaling coordinates of the effective key points among the various human body key points further includes: Three third circles are drawn with the scaling coordinates of the key points corresponding to the target arm as the center and the radius as the third radius. The external common tangent of any adjacent third circles is drawn to obtain the target arm of the virtual animation image. The target arm of the virtual animation image is a region composed of four external common tangents. And / or, The effective key points among the various human body key points also include the key points corresponding to the target leg, which include: the hip key point, the knee key point, and the ankle key point; The step of drawing a virtual animated image corresponding to the i-th frame user image based on the scaling coordinates of the effective key points among the various human body key points further includes: Three fourth circles are drawn with the scaling coordinates of the key point corresponding to the target leg as the center and the radius as the fourth radius. The external common tangent of any adjacent fourth circles is drawn to obtain the target leg of the virtual animation image. The target leg of the virtual animation image is a region composed of four external common tangents.

10. An image processing apparatus, characterized in that, The device includes: a data acquisition module, a determination module, a scaling module, and a drawing module; The acquisition module is used to acquire the i-th frame of user image, i=0,1,2...; The determining module is used to determine the initial coordinates of each key human body point in the i-th frame user image through a target recognition algorithm; The scaling module is used to scale the initial coordinates of each human body key point according to the target ratio to obtain the scaled coordinates of each human body key point, and the scaled coordinates of each human body key point are all located within the display area. The drawing module is used to determine whether the target key point in the i-th frame user image is a valid key point in the current frame, and to count the number of valid key points corresponding to the target key point in the n frames of user images before the i-th frame; the target key point includes any one of the various human body key points; Based on whether the target keypoint is a valid keypoint in the current frame and whether the number of targets is greater than or equal to a quantity threshold, the final status of the target keypoint and its final coordinates are determined, specifically for: If the number of targets is greater than or equal to the number threshold: if the target key point in the current frame is a valid key point, then the scaled coordinates of the target key point in the current frame are used as the final coordinates; if the target key point in the current frame is an invalid key point, then the target key point is determined to be a valid key point, and the scaled coordinates of the valid key point closest to the current frame in the n-frame user image are used as the final coordinates. If the target keypoint in the current frame is a valid keypoint or an invalid keypoint, and the number of targets is less than the number threshold, the target keypoint is ultimately determined to be an invalid keypoint. The final coordinates of the valid key points among the target key points are used as the scaling coordinates of each human body key point. Based on the scaling coordinates of each human body key point, a virtual animated image corresponding to the user image of the i-th frame is drawn.

Citation Information

Patent Citations

  • Target object key point detection method and device

    CN111274852A

  • Animation generation method, readable storage medium and electronic device

    CN112270734A