Video processing method, device, electronic device and storage medium

Abstract

The present application provides a video processing method, device, electronic device and storage medium, the method comprising: receiving a first user image sent by a television and acquiring a second user current image acquired by an image acquisition device; scaling the second current image according to the height of the first user to obtain a scaled image; determining the original position of each skeletal point of the second user in the above image; adjusting the original position of each skeletal point of the second user according to the body proportion of the first user; generating information to be displayed according to the target position of each skeletal point of the second user in the first current image and the first current image and sending it to the television, so that the television displays a target screen according to the information to be displayed, including the first current image and each skeletal point of the second user attached to the first current image and the lines between each skeletal point. By attaching the user skeletal point to another user image and sending it to the television through a mobile terminal, the user's perception can be improved and the action similarity and accurate scoring can be obtained.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, device, electronic device and storage medium. Background Art

[0002] As people's quality of life improves, they usually watch teaching videos such as gymnastics and yoga on TV and imitate the movements in the teaching videos to keep fit and lose weight. At this time, people hope to intuitively see the comparison between their movements and those in the teaching videos and the scores of their movements, so as to make timely adjustments.

[0003] At present, the comparison and scoring of the above-mentioned user actions with the actions in the teaching video are mostly completed by a TV with a built-in camera or an external camera. Specifically, the TV uses its own central processing unit or image processor to identify the user image captured by the camera and calculate the motion skeleton points, complete the action similarity comparison, and score the user action based on the comparison results.

[0004] However, most TVs do not have cameras, and the computing power of the TV's central processing unit and image processor is relatively poor. The ability to recognize the actions of the above-mentioned users and teaching videos is poor, and the accuracy of capturing the skeleton points of the actions is low. Since the position of the TV camera is fixed, the user's activity area is also limited during the recording process. At the same time, since the TV screen is relatively large, the user needs to observe the left and right sides of the TV screen at the same time and then compare them during exercise, which also causes inconvenience in observation when the user uses it. Summary of the invention

[0005] The purpose of this application is to provide a video processing method, device, electronic device and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problems in the prior art that the television has no camera function, the video processing capability is poor and the user has difficulty in observing.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, the present application provides a video processing method, which is applied to a mobile terminal, wherein an image acquisition device is provided on the mobile terminal, and the mobile terminal is communicatively connected to a television, and the method comprises:

[0008] receiving a first current image of a first user sent by the television, and acquiring a second current image of a second user acquired by the image acquisition device;

[0009] scaling the second current image according to the height of the first user in the first current image to obtain a scaled image, wherein the height of the second user in the scaled image is the same as the height of the first user in the first current image;

[0010] Determining the original position of each skeleton point of the second user in the zoomed image;

[0011] According to the body proportions of the first user, adjusting the original positions of the skeleton points of the second user to obtain target positions of the skeleton points of the second user in the first current image;

[0012] Information to be displayed is generated based on the target positions of the skeleton points of the second user in the first current image and the first current image, and the information to be displayed is sent to the TV so that the TV displays a target picture based on the information to be displayed. The target picture includes the first current image and the skeleton points of the second user attached to the first current image and the lines between the skeleton points.

[0013] Optionally, adjusting the original position of each skeleton point of the second user according to the body proportion of the first user to obtain the target position of each skeleton point of the second user in the first current image includes:

[0014] Acquire proportion information of each body part of the first user and original positions of multiple associated skeleton points of the second user corresponding to each body part;

[0015] According to the proportion information of the body part and the preset association relationship between the associated bone points, the original position of at least one associated bone point is adjusted to obtain the target position of each associated bone point corresponding to the body part in the first current image.

[0016] Optionally, generating information to be displayed according to target positions of each skeleton point of the second user in the first current image and the first current image, and sending the information to be displayed to the television so that the television displays a target picture according to the information to be displayed includes:

[0017] Superimposing a skeleton point mark on each of the target positions of the first current image, and adding a connection mark between adjacent skeleton point marks, to obtain an attached image;

[0018] The attached image is sent to the television as the information to be displayed, so that the television displays the information to be displayed as the target picture.

[0019] Optionally, generating information to be displayed according to target positions of each skeleton point of the second user in the first current image and the first current image, and sending the information to be displayed to the television so that the television displays a target picture according to the information to be displayed includes:

[0020] The target positions of the skeleton points of the second user in the first current image are sent to the TV as the information to be displayed, so that the TV adjusts the information to be displayed according to the screen ratio of the TV and the mobile terminal, and displays the information to be displayed in the first current image as the target picture.

[0021] Optionally, the method further includes:

[0022] determining three-dimensional coordinates of each skeletal point of the first user in the first current image according to the body proportion of the first user and the position of each skeletal point of the first user in the first current image;

[0023] determining three-dimensional coordinates of each skeletal point of the second user in the second current image according to the body proportion of the second user and the position of each skeletal point of the second user in the scaled image;

[0024] determining an action score of the second user according to the three-dimensional coordinates of each skeleton point of the first user and the three-dimensional coordinates of each skeleton point of the second user;

[0025] The action score of the second user is sent to the television for display.

[0026] Optionally, determining the three-dimensional coordinates of each skeletal point of the first user in the first current image according to the body proportion of the first user and the position of each skeletal point of the first user in the first current image includes:

[0027] Acquire proportion information of each body part of the first user and positions of multiple associated skeleton points of the first user corresponding to each body part;

[0028] Determining the bending angle of each associated bone point according to the proportion information of the body part and the position of each associated bone point;

[0029] The three-dimensional coordinates of each associated bone point are determined according to the original position and bending angle of each associated bone point.

[0030] Optionally, before adjusting the original position of each bone point of the second user according to the body proportion of the first user, the method further includes:

[0031] Acquire a first upright image, wherein the first upright image includes the first user in an upright state;

[0032] Determine the position of each skeletal point of the first user in the first upright image;

[0033] The body proportions of the first user are determined according to positions of the skeleton points of the first user in the first upright image.

[0034] In a second aspect, the present application provides a video processing device, which is applied to a mobile terminal, wherein an image acquisition device is provided on the mobile terminal, and the mobile terminal is communicatively connected to a television, and the device comprises:

[0035] A receiving module, configured to receive a first current image of a first user sent by the television, and acquire a second current image of a second user acquired by the image acquisition device;

[0036] a scaling module, configured to scale the second current image according to the height of the first user in the first current image to obtain a scaled image, wherein the height of the second user in the scaled image is the same as the height of the first user in the first current image;

[0037] a determination module, configured to determine the original position of each skeleton point of the second user in the zoomed image;

[0038] an adjustment module, configured to adjust the original positions of the skeleton points of the second user according to the body proportions of the first user, so as to obtain target positions of the skeleton points of the second user in the first current image;

[0039] A sending module is used to generate information to be displayed based on the target positions of the skeleton points of the second user in the first current image and the first current image, and send the information to be displayed to the television, so that the television displays a target picture based on the information to be displayed, and the target picture includes the first current image and the skeleton points of the second user attached to the first current image and the lines between the skeleton points.

[0040] Optionally, the adjustment module is specifically used for:

[0041] Acquire proportion information of each body part of the first user and original positions of multiple associated skeleton points of the second user corresponding to each body part;

[0042] According to the proportion information of the body part and the preset association relationship between the associated bone points, the original position of at least one associated bone point is adjusted to obtain the target position of each associated bone point corresponding to the body part in the first current image.

[0043] Optionally, the sending module is specifically used for:

[0044] Superimposing a skeleton point mark on each of the target positions of the first current image, and adding a connection mark between adjacent skeleton point marks, to obtain an attached image;

[0045] The attached image is sent to the television as the information to be displayed, so that the television displays the information to be displayed as the target picture.

[0046] Optionally, the sending module is specifically used for:

[0047] The generating information to be displayed according to the target position of each skeleton point of the second user in the first current image and the first current image, and sending the information to be displayed to the television so that the television displays a target picture according to the information to be displayed, comprises:

[0048] The target positions of the skeleton points of the second user in the first current image are sent to the TV as the information to be displayed, so that the TV adjusts the information to be displayed according to the screen ratio of the TV and the mobile terminal, and displays the information to be displayed in the first current image as the target picture.

[0049] Optionally, the determining module is specifically used to:

[0050] determining three-dimensional coordinates of each skeletal point of the first user in the first current image according to the body proportion of the first user and the position of each skeletal point of the first user in the first current image;

[0051] determining three-dimensional coordinates of each skeletal point of the second user in the second current image according to the body proportion of the second user and the position of each skeletal point of the second user in the scaled image;

[0052] determining an action score of the second user according to the three-dimensional coordinates of each skeleton point of the first user and the three-dimensional coordinates of each skeleton point of the second user;

[0053] The action score of the second user is sent to the television for display.

[0054] Optionally, the determining module is further used for:

[0055] Acquire proportion information of each body part of the first user and positions of multiple associated skeleton points of the first user corresponding to each body part;

[0056] Determining the bending angle of each associated bone point according to the proportion information of the body part and the position of each associated bone point;

[0057] The three-dimensional coordinates of each associated bone point are determined according to the original position and bending angle of each associated bone point.

[0058] Optionally, the adjustment module is further used for:

[0059] Acquire a first upright image, wherein the first upright image includes the first user in an upright state;

[0060] Determine the position of each skeletal point of the first user in the first upright image;

[0061] The body proportions of the first user are determined according to positions of the skeleton points of the first user in the first upright image.

[0062] In a third aspect, the present application provides an electronic device comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method described in the first aspect above.

[0063] In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are executed.

[0064] The beneficial effect of the present application is that after receiving the first user image sent by the TV, the mobile terminal collects the second user image accordingly, and adjusts the height and skeleton points of the second user in the second user image according to the height and skeleton points of the first user in the above-mentioned first user image, obtains the target position of each skeleton point of the second user in the first current image, and then generates the information to be displayed according to the target position, so that the target picture displayed by the TV end includes the skeleton point connection line of the second user attached to the first current image. Since the data processing process is completed on the mobile terminal side, the limitations and low accuracy problems existing in the prior art can be solved. At the same time, since the mobile terminal adjusts the height of the second user according to the height of the first user and generates the information to be displayed, the target picture displayed by the TV end can include the skeleton point connection of the first user and the second user attached to the first user, so that the user can intuitively view the action of the second user on the image of the first user, thereby greatly improving the convenience of observation, and thus greatly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 is a schematic diagram of a scenario of a video processing method applied in an embodiment of the present application;

[0067] Figure 2 is a flow chart of a video processing method provided by an embodiment of the present application;

[0068] Figure 3 It is a flow chart of a method for determining a target position of a user's skeleton point in a video processing method provided in an embodiment of the present application;

[0069] Figure 4 is a schematic diagram of a first user skeleton point provided in an embodiment of the present application;

[0070] Figure 5 is a schematic diagram of a second user skeleton point provided in an embodiment of the present application;

[0071] Figure 6 is a flow chart of displaying an image to be displayed provided by an embodiment of the present application;

[0072] Figure 7 This is a flowchart for obtaining a score provided in an embodiment of the present application;

[0073] Figure 8 This is a flow chart of obtaining three-dimensional coordinates of a skeleton point provided by an embodiment of the present application;

[0074] Fig. 9 is a schematic diagram of calculating the three-dimensional coordinates of a bone point provided in an embodiment of the present application;

[0075] Fig.10 is a flow chart of a method for determining a body proportion of a first user provided in an embodiment of the present application;

[0076] Fig.11 is a schematic diagram of a video processing device provided in an embodiment of the present application;

[0077] Fig.12 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0079] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0080] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0081] The existing technology mainly uses TVs with built-in cameras or external cameras to complete the comparison and scoring of user actions with actions in teaching videos. However, most TVs do not have cameras, so there are limitations in the application of existing methods. In addition, the computing power of the central processing unit and image processor of the TV is poor, the ability to recognize the above-mentioned user and the actions in the teaching video is poor, and the accuracy of capturing the skeleton points of the action is low. Since the position of the TV camera is fixed, the user's activity area is also limited during the recording process. At the same time, since the TV screen is relatively large, the user needs to observe the left and right sides of the TV screen at the same time and then compare them during exercise, which will also cause inconvenience in observation when the user uses it.

[0082] Based on the above-mentioned problems caused by the processing by the TV end, this application proposes a video processing method, which transfers the data processing process to the mobile terminal through the interaction between the mobile terminal and the TV. The mobile terminal receives the first user image, obtains the second user's skeleton points by collecting the second user image, scales the above skeleton points to match the body proportions in the first user image, and sends the image composed of the matched first user image and the second user's skeleton points and their connecting lines to the TV, and then displays the user's actions and scores through the TV. This can solve the limitations, low accuracy, inconvenience of observation and other problems in the existing technology.

[0083] Figure 1 is a schematic diagram of a scenario of a video processing method used in an embodiment of the present application, such as Figure 1 As shown, the method involves a mobile terminal and a television. Specifically, an image acquisition device is provided on the mobile terminal, and the image acquisition device can be, for example, a camera, and the mobile terminal is connected to the television for communication. Take a fitness video teaching scene as an example, in this scene, the television sends the coach's teaching image to the mobile terminal, the mobile terminal collects the user image following the coach's practice, and receives the above-mentioned coach's teaching image, and adjusts the height and bone point coordinates of the user in the user image according to the height and body proportion of the coach in the above-mentioned teaching image. The mobile terminal then overlaps the image with the coach's image and the user's bone points and their connecting lines and sends it to the television, and then the television displays the overlapped image for the user to conveniently view the difference between his own movements and the coach's movements. Optionally, the teaching image and the user image can be each frame in the video, and the coach height and the user height can be obtained by the distance between the center point of the line connecting the user's foot bone points and the top of the head. Exemplarily, the above-mentioned mobile terminal can be a mobile phone and a tablet computer.

[0084] Figure 2 is a flow chart of a video processing method provided in an embodiment of the present application. The execution subject of the method is the above-mentioned mobile terminal. Figure 2 The embodiments of the present application are described as follows:

[0085] S201, receiving a first current image of a first user sent by a television, and acquiring a second current image of a second user captured by an image capture device.

[0086] Exemplarily, the mobile terminal and the TV can send data via a local area network, the mobile terminal can install a specific application (Application, APP for short), and the TV can install specific software, and through the interaction between the APP and the specific software, a channel for interactive information is formed between the mobile terminal and the TV.

[0087] Optionally, the first current image is a frame of image in the video of the first user played on the TV, and the second current image is a frame of image in the video of the second user captured by the mobile terminal. The first current image and the second current image are at the same moment. Exemplarily, assuming that the first current image is a frame of image played on the TV at 0:0:0, then the second current image is a frame of image of the second user captured by the mobile terminal at 0:0:0. What the mobile terminal captures and receives are clear user videos with obvious features of body parts. Taking the aforementioned fitness video teaching scenario as an example, in this scenario, the first user can be a coach, and the second user can be a user practicing with the coach. The first current image can be a frame of image at a certain moment in the coach's teaching video, and the second current image can be a frame of image of the user at that moment captured by the mobile terminal.

[0088] S202: Scale the second current image according to the height of the first user in the first current image to obtain a scaled image, wherein the height of the second user in the scaled image is the same as the height of the first user in the first current image.

[0089] Optionally, the mobile terminal scales the second current image as a whole according to the height of the first user in the first current image, so that the height of the scaled second user is the same as the height of the first user. Exemplarily, the height of the first user in the first current image is 1.8 meters, and the height of the second user in the second current image is 1.6 meters. The mobile terminal needs to scale the image of the second user as a whole so that the height of the second user is 1.8 meters.

[0090] Optionally, the scaling of the second user image is an overall scaling without causing deformation of the image. On this basis, the mobile terminal can make the first user image and the second user image completely overlap in height.

[0091] In addition, the mobile terminal can also determine the center point of the first user in the first current image and the center point of the second user in the second current image, and according to the center point of the first user, horizontally move the position of the second user in the scaled image so that the center points of the first user in the first current image and the second user in the second current image coincide with each other, and use the horizontally moved image as a new scaled image for the following step S203. Taking the first user as an example, the center point of the first user can be the center point of two bone points on both sides of the waist of the first user. The method for determining the center point of the second user is the same.

[0092] S203: Determine the original position of each skeleton point of the second user in the scaled image.

[0093] Optionally, the original position of each skeleton point of the second user in the zoomed image can be determined by a specific skeleton point determination algorithm. Wherein, the original position of each skeleton point can refer to the coordinates of each skeleton point in the zoomed image. On this basis, the overall skeleton framework of the second user can be obtained by connecting the lines of each corresponding skeleton point. It is worth noting that the skeleton framework in the zoomed image at this time is consistent with the current state of the second user. Exemplary, if it is found that the skeleton framework composed of the skeleton point connection lines is inconsistent with the second user state, the skeleton point coordinates need to be recollected.

[0094] S204: adjusting the original positions of the skeleton points of the second user according to the body proportions of the first user to obtain target positions of the skeleton points of the second user in the first current image.

[0095] Optionally, the body proportions of the first user can be obtained in advance by identifying the skeleton points of the first user and calculating them. The method for calculating the body proportions of the first user will be described in detail in the following embodiments. Among them, the body parts bound to the skeleton points of the first user can correspond to the body parts bound to the skeleton points of the second user, that is, the number of skeleton points is consistent. The mobile terminal obtains the distance between the skeleton points of the first user based on the skeleton points of the first user, and the distance between the skeleton points of the second user can be obtained through the position of the skeleton points of the second user, and the original skeleton points of the second user are adjusted according to the relationship between the distance between the skeleton points of the second user and the distance between the skeleton points of the first user.

[0096] Optionally, the target position of the second user's skeletal point in the first current image is the position of each skeleton of the second user after adjustment to adapt to the body proportion of the first user. Exemplarily, the length ratio of the upper arm and forearm of the first user is 5:5, and the length ratio of the upper arm and forearm of the second user is 4:6. After determining the original position of each skeletal point on the upper arm and forearm of the second user, the original position of each skeletal point on the upper arm and forearm of the second user can be adjusted according to the 5:5 ratio of the first user, so that the adjusted target position of the first user matches the length ratio of the upper arm and forearm of the second user, and then, when the skeletal points of the second user are attached to the first current image in the following step S205, the connection line of the skeletal points of the second user can be kept matching with the body of the first user.

[0097] S205, generating information to be displayed according to the target positions of the skeleton points of the second user in the first current image and the first current image, and sending the information to be displayed to the television, so that the television displays a target picture according to the information to be displayed, the target picture including the first current image and the skeleton points of the second user attached to the first current image and the lines between the skeleton points.

[0098] Optionally, the information to be displayed generated by the mobile terminal can be used to indicate the target position of each skeleton point of the second user in the first current image, and then the information to be displayed is sent to the TV. After receiving the information to be displayed, the TV can display the target picture according to the information to be displayed.

[0099] Optionally, the target image is the first current image and the skeleton points of the second user attached to the first current image and the lines between the skeleton points, wherein the skeleton points of the second user can be attached to the first user in the first current image based on specific points.

[0100] In this embodiment, after receiving the first user image sent by the TV, the mobile terminal collects the second user image accordingly, and adjusts the height and skeleton points of the second user in the second user image according to the height and skeleton points of the first user in the above-mentioned first user image, obtains the target position of each skeleton point of the second user in the first current image, and then generates the information to be displayed according to the target position, so that the target picture displayed by the TV end includes the skeleton point connection line of the second user attached to the first current image. Since the data processing process is completed on the mobile terminal side, the limitations and low accuracy problems existing in the prior art can be solved. At the same time, since the mobile terminal adjusts the height of the second user according to the height of the first user and generates the information to be displayed, the target picture displayed by the TV end can include the skeleton point connection of the first user and the second user attached to the first user, so that the user can intuitively view the action of the second user on the image of the first user, thereby greatly improving the convenience of observation, and then greatly improving the user experience.

[0101] Next, combine Figure 3 The above step S204 is described in which the original positions of the skeleton points of the second user are adjusted according to the body proportions of the first user to obtain the target positions of the skeleton points of the second user in the first current image. Figure 3 A flowchart of a method for determining a target position of a user's skeleton point in a video processing method provided in an embodiment of the present application, such as Figure 3 As shown, the above step S204 may include:

[0102] S301, obtaining proportion information of each body part of the first user and original positions of multiple associated skeleton points of the second user corresponding to each body part.

[0103] The multiple associated skeleton points of the second user corresponding to each body part can be pre-set. For example, it is assumed that the human body is pre-set to include 14 skeleton points. Figure 5 is a second user skeleton point schematic diagram, refer to Figure 5, the associated skeletal points of the "left arm" that can be pre-set for the body part "arm" include: wrist skeletal point A, elbow skeletal point B and shoulder skeletal point C. After the scaling process of the second current image is completed in the aforementioned step S203, the original positions of the associated skeletal points of each body part can be read from the scaled image in the order of the body parts of the second user. The original position can refer to the coordinates of the associated skeletal point in the second current image.

[0104] Optionally, the proportion information of each body part of the first user is obtained by obtaining the skeleton points on each body part of the first user. After the mobile terminal obtains the skeleton points of the first user, the distance between each skeleton point can be obtained, and the proportion information of each body part of the first user is obtained through the distance between each skeleton point. Figure 4 is a schematic diagram of the first user skeleton point, such as Figure 4 As shown, the distance between each bone point is obtained through the coordinates of the bone points. For example, the distance between point d and point e can be known from the coordinates of point d and point e. Similarly, the distance between each bone point of the user can be known. The ratio of the distances between each bone point is the proportion information of the first user.

[0105] S302: According to the proportion information of the body part and the preset association relationship between the associated bone points, the original position of at least one associated bone point is adjusted to obtain the target position of each associated bone point corresponding to the body part in the first current image.

[0106] Optionally, the mobile terminal adjusts the position of each bone point of the second user according to the preset association relationship based on the proportion information of each body part of the first user, wherein the preset association relationship can be used to indicate the adjustment priority between each associated bone point in the body part. Exemplarily, for the body part "left arm", the associated bone points include wrist bone point A, elbow bone point B and shoulder bone point C, then the association relationship between each associated bone point can be: the positions of elbow bone point B and shoulder bone point C remain fixed, and only the position of wrist bone point A is adjusted. On this basis, when making adjustments, the mobile terminal calculates the distance between each bone point of the second user at the original position, and obtains the ratio of the distance between the bone points of the first user and the distance between the corresponding bone points of the second user, that is, the ratio of the distance between each bone point of the first user after adjustment and the actual distance between each bone point. According to the above ratio and the above preset association relationship, the original position of at least one associated bone point is adjusted to obtain the target position of each associated bone point corresponding to the body part in the first current image. Exemplarily, with reference to Figure 4, the distance between the wrist bone point a and the elbow bone point b of the first user is ab, the distance between the elbow bone point b and the shoulder bone point c is bc, the distance between the wrist bone point A and the elbow bone point B of the second user is AB, and the distance between the elbow bone point B and the shoulder bone point C is BC. On the basis of calculating the ratio of ab to AB and the ratio of bc to BC, combined with the above-mentioned preset association relationship, at least one of point A and point B is moved according to the above-mentioned preset association relationship, so that the second user's bone point can be scaled according to the first user's body proportion.

[0107] It is worth mentioning that the mobile terminal adjusts at least one associated skeleton point, and the adjusted skeleton point may be one or all skeleton points.

[0108] In this embodiment, the original position of at least one skeleton point of the second user is adjusted based on a preset association relationship according to the body proportions of the first user, and the target position of each skeleton point of the second user in the first current image is obtained, wherein adjusting the position of the skeleton point of the second user according to the preset association relationship can obtain a more accurate skeleton point adapted to the proportion information of the first user, so that the skeleton point of the second user is attached to the first current image and the skeleton relationship is coherent, and the skeleton point can still display its own movement according to the coach's body proportion when the movement is inconsistent with the first user, avoiding the visual incoordination caused by the large difference in body shape between the first user and the second user.

[0109] In the above step S205, the target positions of the skeleton points of the second user in the first current image and the first current image generate information to be displayed, and the information to be displayed is sent to the TV, so that when the TV displays the target picture according to the information to be displayed, it can be determined in any of the following ways.

[0110] In the first way, after the information to be displayed is sent to the television via the mobile terminal, the television directly displays the information to be displayed. Figure 6 A flow chart for displaying an image to be displayed is provided, such as Figure 6 As shown, the above step S205 may include:

[0111] S601: superimpose skeleton point marks on each target position of the first current image, and add connection marks between adjacent skeleton point marks to obtain an attached image.

[0112] Optionally, the mobile terminal superimposes the adjusted skeleton points of the second user and the first user in the first current image, and adds connection marks between adjacent skeleton point marks, and each skeleton point and connection mark constitutes a skeleton framework of the user.

[0113] S602: Send the attached image as information to be displayed to the television, so that the television displays the information to be displayed as a target picture.

[0114] Optionally, the information to be displayed includes the first current image and the adjusted second user skeleton points attached to the first user and connection marks thereof.

[0115] In the above method, after the information to be displayed is sent to the TV via the mobile terminal, the TV directly displays the information to be displayed, thereby reducing the processing complexity of the TV and avoiding the picture delay caused by the poor computing power of the TV central processing unit and the image processor.

[0116] The second method is that after the mobile terminal sends the information to be displayed to the TV, the TV adjusts the information in the information to be displayed and then displays it. The method is:

[0117] The target positions of the second user's skeleton points in the first current image are sent to the TV as information to be displayed, so that the TV adjusts the information to be displayed according to the screen ratio of the TV and the mobile terminal, and displays the information to be displayed as a target picture attached to the first current image.

[0118] Optionally, the TV attaches the skeletal points of the second user to the body of the first user being displayed based on the received information to be displayed. It is worth noting that due to the different ratios between the mobile terminal screen and the TV screen, the skeletal points of the second user need to be scaled proportionally according to the above ratios.

[0119] The above method enables the TV to flexibly adjust the position of each skeleton point according to its own screen ratio, so it can be applied to various models of TVs, greatly improving flexibility.

[0120] As an optional implementation, the mobile terminal may also obtain the action score of the second user by analyzing and comparing the first current image and the second current image. This process is described below.

[0121] Figure 7 is a flow chart for obtaining scores, such as Figure 7 As shown, the above method also includes:

[0122] S701: Determine three-dimensional coordinates of each skeleton point of the first user in the first current image according to the body proportion of the first user and the position of each skeleton point of the first user in the first current image.

[0123] Optionally, the body proportions of the first user can be calculated in advance by the ratio of the distances between the skeletal points. For example, if the first user's thigh is 40 cm and the calf is 60 cm long, then the ratio of the length of the calf to the thigh is 2 / 3, and the body proportion of the leg is 2 / 3.

[0124] The first current image includes the two-dimensional coordinates of each bone point of the first user. Based on the two-dimensional coordinates and the body proportions of the first user, the curvature information of the first user's body can be determined, and the three-dimensional coordinates of each bone point of the first user can be determined accordingly.

[0125] S702: Determine three-dimensional coordinates of each skeleton point of the second user in the second current image according to the body proportion of the second user and the position of each skeleton point of the second user in the scaled image.

[0126] Optionally, the body proportions of the second user may also be calculated by the ratio of the distances between the skeletal points. The specific determination method is the same as that of the first user, and will not be repeated here.

[0127] S703: Determine an action score of the second user according to the three-dimensional coordinates of each skeleton point of the first user and the three-dimensional coordinates of each skeleton point of the second user.

[0128] Optionally, based on the three-dimensional coordinates of each skeletal point of the first user and the three-dimensional coordinates of each skeletal point of the second user determined by the above steps, the three-dimensional coordinates of the same skeletal points can be compared in sequence according to the order of the skeletal points to obtain the comparison results corresponding to the skeletal points. And according to the comparison results corresponding to the skeletal points, the action score of the second user is obtained. Among them, the comparison results corresponding to the skeletal points can refer to the difference between the position of the skeletal point of the second user and the position of the same skeletal point of the first user.

[0129] S704: Send the action score of the second user to the television for display.

[0130] Optionally, the action score can be in the form of a score, a percentage, or a graphic. For example, the action score can be 80 points, or 90% completion, or can be represented in the form of a six-dimensional radar chart.

[0131] In this embodiment, the mobile terminal scores the action of the second user by comparing the consistency of the three-dimensional coordinates of each skeleton point of the first user and the second user, wherein the three-dimensional coordinates are more precise than the traditional two-dimensional coordinates, and thus the action score is more accurate.

[0132] Figure 8 This is a flowchart for obtaining the three-dimensional coordinates of the bone points. Figure 8 The method for obtaining the three-dimensional coordinates of the first user skeleton point in step S701 is described, and the steps include:

[0133] S801. Obtain proportion information of each body part of the first user and positions of multiple associated skeleton points of the first user corresponding to each body part.

[0134] Optionally, the mobile terminal obtains the proportion information of the body part according to the position of the associated bone point. Exemplarily, the leg joint information of the first user is obtained, such as Figure 4 The left leg of the first user has a hip bone point i, a knee bone point k, and an ankle bone point m. When the first user is in an upright state, the ratio of the thigh to the calf can be obtained as ik / km. For example, ik is 40 cm and km is 60 cm, then the ratio of the user's thigh to the calf is 2 / 3.

[0135] S802: Determine the bending angle of each associated bone point according to the proportion information of the body part and the position of each associated bone point.

[0136] Optionally, when the joint of the associated bone point is bent, the change distances of the three dimensions of the corresponding bone points are all proportional, and the bending angle of each associated bone point can be obtained accordingly. Fig. 9 It is a schematic diagram for calculating the three-dimensional coordinates of bone points. Fig. 9 For example, when the skeleton point k is bent toward the z-axis, the ratio of y2-y1 to y1 is equal to ik / km, which is 2 / 3 of the above. Then, according to the ratio of y1 and km and the Pythagorean theorem, the length of the skeleton point k on the z-axis can be known. The bending angle of the skeleton point can be calculated by the length of the z-axis. The specific formula is as follows:

[0137] z 2 +y1 2 =(6y) 2

[0138] y1y2 2 +z 2 =(4y) 2

[0139] Wherein, z represents the bending length of bone point k in the Z axis, y1 represents the length of bone point k in the Y axis direction, y represents the unit number in the proportion, and y2 represents the distance between bone point i and bone point k in the Y axis direction.

[0140] S803: Determine the three-dimensional coordinates of each associated bone point according to the original position and bending angle of each associated bone point.

[0141] Optionally, the original position and bending angle of each associated bone point can be used to obtain the length of the above bone point in the X-axis, Y-axis, and Z-axis, and the three-dimensional coordinates of each bone point can be determined by the lengths in the three axes.

[0142] It is worth noting that the method for determining the three-dimensional coordinates of each skeleton point of the second user is the same as that of the first user, and can refer to this embodiment and will not be repeated here.

[0143] In this embodiment, the mobile terminal determines the three-dimensional coordinates of the skeleton points through the user's body proportions and the position of each skeleton point, and uses simple information to improve the accuracy of verification after processing, thereby avoiding the increase in costs caused by the user purchasing expensive three-dimensional somatosensory cameras to obtain the above-mentioned three-dimensional coordinates.

[0144] According to the following Fig.10 A method for determining the body proportions of a first user is described, and the steps include:

[0145] S1001. Acquire a first upright image, where the first upright image includes a first user in an upright state.

[0146] Optionally, the first upright image includes a full body image of the first user in an upright state.

[0147] S1002: Determine the positions of the skeleton points of the first user in the first upright image.

[0148] Exemplarily, the openpose algorithm may be used to identify and obtain the position of each skeleton point in the first upright image.

[0149] S1003: Determine the body proportions of the first user according to the positions of the skeleton points of the first user in the first upright image.

[0150] Optionally, the body proportions of the first user are obtained by the ratio of the distances between the bone points.

[0151] The body proportions include the proportions of various body parts. For example, for the head, the proportions may refer to the height of the head; for the arms, the proportions may refer to the length proportions of the upper arm and the forearm.

[0152] Taking the arm as an example, combined with Figure 4 and Figure 5 , the distance between the first user's wrist bone point a and elbow bone point b is ab, and the distance between the elbow bone point b and the shoulder bone point c is bc. The distance between the second user's wrist bone point A and elbow bone point B is AB, and the distance between the elbow bone point B and the shoulder bone point C is BC. The proportion of the arm is AB / BC.

[0153] In this embodiment, after obtaining the body proportions of the first user, the mobile terminal attaches the adjusted skeletal points of the second user to the body of the first user and sends the image to the television for display. This step provides the second user with a better viewing experience.

[0154] Based on the same inventive concept, a video processing device corresponding to the video processing method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned video processing method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0155] Reference Fig.11 , which is a schematic diagram of a video processing device provided in an embodiment of the present application, the device includes: a receiving module 1101, a scaling module 1102, a determining module 1103, an adjusting module 1104, and a sending module 1105; wherein:

[0156] The receiving module 1101 is used to receive a first current image of a first user sent by a television, and obtain a second current image of a second user acquired by an image acquisition device;

[0157] A scaling module 1102 is used to scale the second current image according to the height of the first user in the first current image to obtain a scaled image, wherein the height of the second user in the scaled image is the same as the height of the first user in the first current image;

[0158] A determination module 1103, used to determine the original position of each skeleton point of the second user in the scaled image;

[0159] An adjustment module 1104, configured to adjust the original positions of the skeleton points of the second user according to the body proportions of the first user, to obtain target positions of the skeleton points of the second user in the first current image;

[0160] The sending module 1105 is used to generate information to be displayed based on the target positions of the skeleton points of the second user in the first current image and the first current image, and send the information to be displayed to the TV so that the TV displays the target picture according to the information to be displayed. The target picture includes the first current image and the skeleton points of the second user attached to the first current image and the lines between the skeleton points.

[0161] Optionally, the adjustment module 1104 is specifically used for:

[0162] Acquire proportion information of each body part of the first user and original positions of multiple associated skeleton points of the second user corresponding to each body part;

[0163] According to the proportion information of the body part and the preset association relationship between the associated bone points, the original position of at least one associated bone point is adjusted to obtain the target position of each associated bone point corresponding to the body part in the first current image.

[0164] Optionally, the sending module 1105 is specifically used for:

[0165] Superimposing skeleton point marks on each target position of the first current image, and adding connection marks between adjacent skeleton point marks to obtain an attached image;

[0166] The attached image is sent to the television as information to be displayed, so that the television displays the information to be displayed as a target picture.

[0167] Optionally, the sending module 1105 is specifically used for:

[0168] The method generates information to be displayed according to the target positions of the skeleton points of the second user in the first current image and the first current image, and sends the information to be displayed to the television, so that the television displays the target picture according to the information to be displayed, including:

[0169] The target positions of the skeleton points of the second user in the first current image are sent to the TV as information to be displayed, so that the TV adjusts the information to be displayed according to the screen ratio of the TV and the mobile terminal, and displays the information to be displayed as the target picture in the first current image.

[0170] Optionally, the determination module 1103 is specifically configured to:

[0171] Determine the three-dimensional coordinates of each skeletal point of the first user in the first current image according to the body proportion of the first user and the position of each skeletal point of the first user in the first current image;

[0172] Determine the three-dimensional coordinates of each skeletal point of the second user in the second current image according to the body proportion of the second user and the position of each skeletal point of the second user in the scaled image;

[0173] Determining an action score of the second user according to the three-dimensional coordinates of each skeleton point of the first user and the three-dimensional coordinates of each skeleton point of the second user;

[0174] The second user's action score is sent to the television for display.

[0175] Optionally, the determination module 1103 is further configured to:

[0176] Acquire proportion information of each body part of the first user and positions of multiple associated skeleton points of the first user corresponding to each body part;

[0177] Determine the bending angle of each associated bone point according to the proportion information of the body parts and the position of each associated bone point;

[0178] The three-dimensional coordinates of each associated bone point are determined according to the original position and bending angle of each associated bone point.

[0179] Optionally, the adjustment module 1104 is further configured to:

[0180] Acquire a first upright image, where the first upright image includes a first user in an upright state;

[0181] Determine the position of each skeletal point of the first user in the first upright image;

[0182] The body proportions of the first user are determined according to the positions of the skeleton points of the first user in the first upright image.

[0183] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0184] The present application also provides an electronic device, such as Fig.12 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, including: a processor 1201, a memory 1202 and a bus. The memory 1202 stores machine-readable instructions executable by the processor 1201 (for example, Fig.11 In the device, the receiving module 1101, the scaling module 1102, the determining module 1103, the adjusting module 1104, the sending module 1105 and the corresponding execution instructions, etc., when the computer device is running, the processor 1201 communicates with the memory 1202 through the bus, and when the machine-readable instructions are executed by the processor 1201, the above-mentioned video processing method is performed.

[0185] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video processing method are executed.

[0186] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0187] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0188] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A video processing method, characterized in that: Applied to a mobile terminal, the mobile terminal is provided with an image acquisition device, the mobile terminal is communicatively connected with a television, and the method comprises: receiving a first current image of a first user sent by the television, and acquiring a second current image of a second user acquired by the image acquisition device; scaling the second current image according to the height of the first user in the first current image to obtain a scaled image, wherein the height of the second user in the scaled image is the same as the height of the first user in the first current image; Determining the original position of each skeleton point of the second user in the zoomed image; According to the body proportions of the first user, adjusting the original positions of the skeleton points of the second user to obtain target positions of the skeleton points of the second user in the first current image; generating information to be displayed according to target positions of the skeleton points of the second user in the first current image and the first current image, and sending the information to be displayed to the television, so that the television displays a target picture according to the information to be displayed, the target picture including the first current image and the skeleton points of the second user attached to the first current image and the lines between the skeleton points; The method further comprises: determining three-dimensional coordinates of each skeletal point of the first user in the first current image according to the body proportion of the first user and the position of each skeletal point of the first user in the first current image; determining three-dimensional coordinates of each skeletal point of the second user in the second current image according to the body proportion of the second user and the position of each skeletal point of the second user in the scaled image; determining an action score of the second user according to the three-dimensional coordinates of each skeleton point of the first user and the three-dimensional coordinates of each skeleton point of the second user; The action score of the second user is sent to the television for display.

2. The method according to claim 1, characterized in that The adjusting the original position of each skeleton point of the second user according to the body proportion of the first user to obtain the target position of each skeleton point of the second user in the first current image includes: Acquire proportion information of each body part of the first user and original positions of multiple associated skeleton points of the second user corresponding to each body part; According to the proportion information of the body part and the preset association relationship between the associated bone points, the original position of at least one associated bone point is adjusted to obtain the target position of each associated bone point corresponding to the body part in the first current image.

3. The method according to claim 1, characterized in that The generating information to be displayed according to the target position of each skeleton point of the second user in the first current image and the first current image, and sending the information to be displayed to the television so that the television displays a target picture according to the information to be displayed, comprises: Superimposing a skeleton point mark on each of the target positions of the first current image, and adding a connection mark between adjacent skeleton point marks, to obtain an attached image; The attached image is sent to the television as the information to be displayed, so that the television displays the information to be displayed as the target picture.

4. The method according to claim 1, characterized in that The generating information to be displayed according to the target position of each skeleton point of the second user in the first current image and the first current image, and sending the information to be displayed to the television so that the television displays a target picture according to the information to be displayed, comprises: The target positions of the skeleton points of the second user in the first current image are sent to the TV as the information to be displayed, so that the TV adjusts the information to be displayed according to the screen ratio of the TV and the mobile terminal, and displays the information to be displayed in the first current image as the target picture.

5. The method according to claim 1, characterized in that The determining, according to the body proportion of the first user and the position of each skeleton point of the first user in the first current image, the three-dimensional coordinates of each skeleton point of the first user in the first current image includes: Acquire proportion information of each body part of the first user and positions of multiple associated skeleton points of the first user corresponding to each body part; Determining the bending angle of each associated bone point according to the proportion information of the body part and the position of each associated bone point; The three-dimensional coordinates of each associated bone point are determined according to the original position and bending angle of each associated bone point.

6. The method according to any one of claims 1 to 4, characterized in that: Before adjusting the original positions of the skeleton points of the second user according to the body proportions of the first user, the method further includes: Acquire a first upright image, wherein the first upright image includes the first user in an upright state; Determine the position of each skeletal point of the first user in the first upright image; The body proportions of the first user are determined according to positions of the skeleton points of the first user in the first upright image.

7. A video processing device, characterized in that: include: A receiving module, used for receiving a first current image of a first user sent by a television, and acquiring a second current image of a second user acquired by an image acquisition device; a scaling module, configured to scale the second current image according to the height of the first user in the first current image to obtain a scaled image, wherein the height of the second user in the scaled image is the same as the height of the first user in the first current image; a determination module, configured to determine the original position of each skeleton point of the second user in the zoomed image; an adjustment module, configured to adjust the original positions of the skeleton points of the second user according to the body proportions of the first user, so as to obtain target positions of the skeleton points of the second user in the first current image; a sending module, configured to generate information to be displayed according to target positions of the skeleton points of the second user in the first current image and the first current image, and send the information to be displayed to the television, so that the television displays a target picture according to the information to be displayed, wherein the target picture includes the first current image and the skeleton points of the second user attached to the first current image and the lines between the skeleton points; The sending module is further used to determine the three-dimensional coordinates of each skeleton point of the first user in the first current image according to the body proportion of the first user and the position of each skeleton point of the first user in the first current image; determining three-dimensional coordinates of each skeletal point of the second user in the second current image according to the body proportion of the second user and the position of each skeletal point of the second user in the scaled image; determining an action score of the second user according to the three-dimensional coordinates of each skeleton point of the first user and the three-dimensional coordinates of each skeleton point of the second user; The action score of the second user is sent to the television for display.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to perform the steps of the video processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 6 are executed.

Images