Image processing method and device, image equipment and storage medium

Abstract

Embodiments of the present application disclose an image processing method and device, an image device and a storage medium. The image processing method comprises: acquiring an image of a target object; the target object comprises a limb of a body; acquiring a feature of the limb of the body based on the image; the limb comprises at least two parts and a connecting part connecting the two parts; determining second type motion information of the connecting part based on the feature of the limb; and controlling the motion of the limb of a controlled model according to the second type motion information of the connecting part.

Description

[0001] This application is a divisional application of the patent application No. 201910365188.2, with the application date of April 30, 2019, and the title of “Image processing method and device, image equipment and storage medium”. TECHNICAL FIELD

[0002] The present application relates to the field of information technology, in particular to an image processing method and device, image equipment and storage medium. BACKGROUND

[0003] With the development of information technology, users can record videos for online teaching, online anchors, and motion games. However, in some cases, such as motion games, users need to wear special motion devices to detect their own activities, such as body movements, to control game characters. When online teaching or online anchors, the user's appearance or body is fully exposed on the network, which may involve user privacy issues on the one hand, and information security issues on the other hand. In order to solve this privacy or security problem, the face image may be covered by mosaic or the like, but this will affect the video effect. SUMMARY

[0004] Therefore, the embodiments of the present application aim to provide an image processing method and device, image equipment and storage medium.

[0005] To achieve the above-mentioned purpose, the technical solution of the present application is as follows:

[0006] An image processing method, comprising:

[0007] Obtaining an image collected for a target object; the target object includes a body limb;

[0008] Obtaining a feature of the body limb based on the image; the limb includes at least two local parts and a connecting part connecting the two local parts;

[0009] Determining second type motion information of the connecting part based on the feature of the limb;

[0010] Controlling the motion of the limb of a controlled model according to the second type motion information of the connecting part.

[0011] Based on the above-mentioned solution, the method further comprises: determining first type motion information of the local part of the limb based on the feature of the limb; and controlling the motion of the limb of the controlled model according to the first type motion information.

[0012] Based on the above-mentioned solution, determining the second type motion information of the connecting part based on the feature of the limb comprises:

[0013] determine the second type motion information of the connection part based on the position information of the key points of the two local parts connected by the connection part.

[0014] According to the above scheme, the motion of the limb of the controlled model is controlled according to the second type motion information of the connection part, which includes:

[0015] According to the characteristics of the two local parts and the first motion constraint condition of the connection part, the second type motion information of the connection part is determined.

[0016] According to the second type motion information, the motion of the connection part of the controlled model is controlled.

[0017] According to the above scheme, the motion of the connection part of the controlled model is controlled according to the second type motion information, which includes:

[0018] According to the type of the connection part, the control mode of the connection part is determined.

[0019] According to the control mode of the connection part and the second type motion information, the motion of the connection part of the controlled model is controlled.

[0020] According to the above scheme, the control mode of the connection part is determined according to the type of the connection part, which includes:

[0021] In the case of the first type connection part, a first type control mode is determined to be used, wherein the first type control mode is used to directly control the first type controlled local part of the controlled model which is the same as the first type connection part.

[0022] According to the above scheme, the control mode of the connection part is determined according to the type of the connection part, which includes:

[0023] In the case of the second type connection part, a second type control mode is determined to be used, wherein the second type control mode is used to indirectly control the second type connection part by controlling the local part other than the second type connection part of the controlled model.

[0024] According to the above scheme, the motion of the connection part of the controlled model is controlled according to the control mode of the connection part and the second type motion information, which includes:

[0025] In the case of the second type control mode of the connection part, the second type motion information is decomposed to obtain the first type rotation information that the second type connection part is rotated by the traction part.

[0026] According to the first type rotation information, the motion information of the traction part is adjusted.

[0027] Control the movement of the traction part in the controlled model based on the adjusted movement information of the traction part, so as to indirectly control the movement of the second type of connecting part.

[0028] Based on the above scheme, the method further comprises:

[0029] Decompose the second type of movement information to obtain second type of rotation information of the second type of connecting part relative to the rotation of the traction part;

[0030] Control the rotation of the second type of connecting part relative to the traction part of the controlled model based on the second type of rotation information.

[0031] Based on the above scheme, the first type of connecting part comprises: elbow; knee;

[0032] And / or,

[0033] The second type of connecting part comprises: wrist; ankle.

[0034] Based on the above scheme, in the case that the second type of connecting part is a wrist, the traction part corresponding to the wrist comprises: upper arm;

[0035] And / or,

[0036] In the case that the second type of connecting part is an ankle, the traction part corresponding to the ankle comprises: lower leg.

[0037] An image processing device comprises:

[0038] A first acquisition module is configured to acquire an image collected for a target object; the target object comprises a limb of a body;

[0039] A second acquisition module is configured to acquire features of the limb of the body based on the image; the limb comprises at least two local parts and a connecting part connecting the two local parts;

[0040] A second determination module is configured to determine second type of movement information of the connecting part based on the features of the limb;

[0041] A control module is configured to control the movement of the limb of the controlled model according to the second type of movement information of the connecting part.

[0042] An image device comprises:

[0043] A memory;

[0044] A processor connected with the memory, configured to realize the image processing method provided in any of the above technical schemes by executing computer executable instructions located on the memory.

[0045] A computer storage medium stores computer executable instructions, and the computer executable instructions are executed by a processor to implement the image processing method provided in any of the technical solutions.

[0046] The image processing method provided in the embodiments of the present application can imitate the action of the limbs of the collection object through image collection, and then be used to control the action of the limbs of the controlled model. In this way, the controlled model simulates the motion of the user through the control of the action of the limbs of the controlled model, realizes video teaching, video speech, live broadcast or game control, and at the same time, the collected image is hidden by using the controlled model to replace the direct display of the collected image, so as to protect the privacy of the user and improve the information security. BRIEF DESCRIPTION OF DRAWINGS

[0047] The drawings incorporated in the specification and constituting a part of the specification illustrate embodiments consistent with the present application and, together with the specification, serve to explain the technical solutions of the present application. The drawings incorporated in the specification and constituting a part of the specification illustrate embodiments consistent with the present application and, together with the specification, serve to explain the technical solutions of the present application.

[0048] Figure 1 The flowchart of the first image processing method provided in the embodiments of the present application is shown in the figure.

[0049] Figure 2 The flowchart of the first type of motion information of the first local part provided in the embodiments of the present application is shown in the figure.

[0050] Figure 3 The key point diagram of the hand provided in the embodiments of the present application is shown in the figure.

[0051] Figure 4 The flowchart of the second image processing method provided in the embodiments of the present application is shown in the figure.

[0052] Figures 5A to 5C The change diagram of the hand motion of the collected user simulated by the controlled model provided in the embodiments of the present application is shown in the figure.

[0053] Figures 6A to 6C The change diagram of the trunk motion of the collected user simulated by the controlled model provided in the embodiments of the present application is shown in the figure.

[0054] Figure 7 The structural diagram of the image processing device provided in the embodiments of the present application is shown in the figure.

[0055] Figure 8A The key point diagram provided in the embodiments of the present application is shown in the figure.

[0056] Figure 8B A schematic diagram illustrating another key point provided in an embodiment of this application;

[0057] Figure 9 A schematic diagram illustrating the construction of a local coordinate system in an embodiment of this application;

[0058] Figure 10 This is a schematic diagram of the structure of an image device provided in an embodiment of this application. Detailed Implementation

[0059] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] like Figure 1 As shown, this embodiment provides an image processing method, including:

[0061] Step S110: Acquire the image;

[0062] Step S120: Obtain the features of the body's limbs based on the image, wherein the limbs include: upper limbs and / or lower limbs;

[0063] Step S130: Based on the features, determine the first type of motion information of the limb;

[0064] Step S140: Control the movement of the limbs of the controlled model based on the first type of motion information.

[0065] The image processing method provided in this embodiment can drive the motion of a controlled model through image processing.

[0066] The image processing method provided in this embodiment can be applied to image devices, which can be various electronic devices capable of image processing, such as electronic devices that perform image acquisition, image display, and image pixel recombination to generate images.

[0067] The imaging device includes, but is not limited to, various terminal devices, such as mobile terminals and / or fixed terminals; it may also include various image servers capable of providing image services.

[0068] The mobile terminal includes portable devices such as mobile phones or tablets that are easy for users to carry, as well as devices worn by users, such as smart bracelets, smartwatches, or smart glasses.

[0069] The fixed terminal includes a fixed desktop computer, etc.

[0070] In this embodiment, the image obtained in step S110 can be a 2D image or a 3D image.

[0071] The 2D image may include: red-green-blue (RGB) images captured by a monocular or multi-view camera, etc.

[0072] The manner of acquiring the image can include:

[0073] acquiring the image by using a camera of the image device itself;

[0074] and / or,

[0075] receiving the image from an external device;

[0076] and / or,

[0077] reading the image from a local database or a local storage.

[0078] The limb of the body can include: an upper limb and / or a lower limb.

[0079] In step S120, a feature of the upper limb or the lower limb can be detected, and after the feature is obtained, the first type of motion information is obtained in step S130. The first type of motion information at least represents a motion change of the limb.

[0080] In the embodiment, a deep learning model such as a neural network can be used to detect the image, thereby obtaining the feature.

[0081] The controlled model can be a model corresponding to the target. For example, if the target is a person, the controlled model is a body model, and if the target is an animal, the controlled model can be a body model of the corresponding animal.

[0082] In summary, in the embodiment, the controlled model is a model for the category to which the target belongs. The model can be predetermined. For example, the style of the controlled model can be determined based on a user instruction. The style of the controlled model can include multiple styles, such as a real person style simulating a real person, an animation style, a net celebrity style, and styles of different moods, such as an artistic style or a rock style, and a game style. In the game style, the controlled model can be a game character.

[0083] In the embodiment, through the execution of the above steps S110 to S140, the limb movement of the user can be directly migrated to the controlled model; in this way, through image acquisition by the camera, the user can easily control the limb movement of the controlled object without wearing a body sensing device.

[0084] In some cases, if the user does not want to use the real portrait of himself / herself when performing a network live broadcast or a speech, but wants the controlled model to simulate his / her own limb movement to enhance the effect of the network live broadcast and the appeal of the speech, the above method can not only protect the privacy of the user, but also ensure the desired effect of the user.

[0085] In some embodiments, the step S130 can include:

[0086] detecting position information of a key point of a limb in the image;

[0087] determining the first type of motion information according to the position information.

[0088] The position information of the key point of the limb includes, but is not limited to, position information of a skeleton key point and / or position information of a contour key point. The position information of the skeleton key point is information of a key point of a skeleton, and the position information of the contour key point is position information of a key point of an outer surface of the limb.

[0089] The position information can include coordinates.

[0090] The coordinates of a body in different body postures or different motion states relative to a reference posture or a reference motion state change, and thus the first type of motion information can be determined according to the coordinates.

[0091] In this embodiment, the first type of motion information can be determined simply and quickly through the position information of the key point, and the first type of motion information can be directly used as a motion parameter for driving a controlled model to move.

[0092] In some embodiments, the method further includes:

[0093] detecting position information of a key point of a body skeleton in the image;

[0094] The detecting of the position information of the key point of the limb in the image includes:

[0095] The position information of the key point of the limb is determined based on the position information of the key point of the body skeleton.

[0096] In this embodiment, the position information of the key point of the whole body of the body skeleton can be obtained by using a neural network or other deep learning model, and thus the position information of the key point of the limb can be determined based on the distribution between the position information of the key points of the whole body. After obtaining the position information of the key points of the whole body, the key points are connected to obtain a skeleton, and based on the relative distribution positions of the bones and joints in the skeleton, it can be determined which key points are the key points of the limb, and thus the position information of the key points of the limb is determined.

[0097] In some other embodiments, the position of the limb can be identified through human recognition, and the position information of the key point is extracted only for the position of the limb.

[0098] In some embodiments, as shown in Figure 2 The step S130 can include:

[0099] Step S131: determining a position box of the first part containing the limb in the image according to the position information of the key points of the limb;

[0100] Step S132: detecting the position information of the key points of the first part based on the position box.

[0101] Step S133: obtaining the first type motion information of the first part based on the position information.

[0102] In the embodiment, in order to accurately imitate the motion of the limb, taking the upper limb as an example, the motion of the upper limb includes not only the motion of the upper arm and the lower arm, but also the finger motion of the hand which is more subtle.

[0103] In the embodiment, the first part can be the hand or the foot.

[0104] Based on the obtained position box, the image region containing the first part can be framed in the image.

[0105] Further position information of the key points is obtained for the image region.

[0106] For example, the position information of the key points corresponding to the knuckles is extracted from the hand. Different gestures of the hand result in different states of the fingers, and the positions and distributions of the knuckles are different, which can be reflected by the key points of the knuckles.

[0107] In some embodiments, the step S131 can include:

[0108] The position box of the hand is determined in the image according to the position information of the key points of the hand.

[0109] The position box can be a rectangular box or a non-rectangular box.

[0110] For example, taking the hand as the first part, the position box can be a hand-shaped box which is suitable for the shape of the hand. For example, the hand-shaped position box can be generated by detecting the outer contour key points of the hand.

[0111] For another example, the skeleton key points of the hand are detected to obtain a circumscribed box of a rectangle containing all the skeleton key points. The circumscribed box is a regular rectangular box.

[0112] Further, the step S132 can include:

[0113] The position information of the key points corresponding to the knuckles of the hand and / or the position information of the key points corresponding to the fingertips of the fingers are detected based on the position box.

[0114] Figure 3 The figure shows a schematic diagram of the key points of a hand. InFigure 3 The 20 key points of the hand are shown in FIG. 1, which are the key points of the joints of the 5 fingers and the key points of the finger tips. These key points are respectively P1 to P20 in FIG. 1. Figure 3

[0115] The position information of the key points of the joints of the fingers can control the movement of each joint of the controlled model, and the position information of the key points of the finger tips can control the movement of the finger tips of the controlled model, thereby achieving more precise control in the process of limb migration.

[0116] In some embodiments, the step S130 can include: obtaining movement information of the fingers of the hand based on the position information.

[0117] In some embodiments, the determining the first type of movement information of the limb based on the features further includes:

[0118] determining the first type of movement information of the second part of the limb based on the position information of the key points of the limb.

[0119] The second part here is a part of the limb other than the first part. For example, taking the upper limb as an example, if the first part is the lower arm and / or the upper arm between the elbow joint and the shoulder joint.

[0120] The second part can directly obtain the first type of movement information based on the position information of the key points of the limb.

[0121] Therefore, in this embodiment, the first type of movement information of different parts of the limb is obtained in different ways according to the characteristics of different parts of the limb, so as to achieve precise control of different parts of the limb in the controlled model.

[0122] In some embodiments, the method further includes:

[0123] determining the second type of movement information of the connecting part based on the position information of the key points of the two parts connected by the connecting part in the limb.

[0124] In some embodiments, as shown in FIG. 2, the step S140 can include: Figure 4

[0125] Step S141: determining the second type of movement information of the connecting part of the at least two parts based on the features of the at least two parts and the first movement constraint condition of the connecting part;

[0126] Step S142: controlling the movement of the connecting part of the controlled model according to the second type of movement information.

[0127] ​​The connection part can be a part connecting two other parts, for example, in the case of a human body, the neck, wrist, ankle or waist are all connection parts connecting two parts.

[0128] For example, in the case of an upper limb of a human body, the connection part can include a wrist connecting a hand and a lower arm, an elbow connecting an upper arm and a lower arm, etc. For example, in the case of a lower limb of a human body, the connection part can include an ankle connecting a foot and a lower leg, a knee connecting a lower leg and a thigh.

[0129] The motion information of these connection parts (i.e., the second type of motion information) can not be directly detected or can be determined to some extent depending on its adjacent other parts, so the second type of motion information can be determined according to the other parts connected thereto.

[0130] In this embodiment, the S142 further includes: determining a control mode according to the type of the connection part, and controlling the motion of the connection part corresponding to the controlled model based on the control mode.

[0131] For example, the lateral rotation of the wrist, the lateral direction being the extension direction of the upper arm to the hand, is visually a lateral rotation of the wrist, but in essence caused by the rotation of the upper arm. For another example, the lateral rotation of the ankle, the lateral direction being the extension direction of the lower leg, is in essence caused by the rotation of the lower leg.

[0132] In other embodiments, the determining the control mode for controlling the connection part according to the type of the connection part includes:

[0133] In the case where the connection part is a first type of connection part, a first type of control mode is determined to be used, wherein the first type of control mode is used to directly control a first type of controlled part of the controlled model which is the same as the first type of connection part.

[0134] In this embodiment, the first type of connection part is a rotation of itself which is not caused by other parts.

[0135] The second type of connection part is a connection part other than the first type of connection part, and the rotation of the second type of connection part includes a rotation caused by other parts.

[0136] In some embodiments, the determining the control mode for controlling the connection part according to the type of the connection part includes: in the case where the connection part is a second type of connection part, a second type of control mode is determined to be used, wherein the second type of control mode is used to indirectly control the second type of connection part by controlling parts of the controlled model other than the second type of connection part.

[0137] The local part other than the second type of connection part includes, but is not limited to, a local part directly connected with the second type of connection part, or a local part indirectly connected with the second type of connection part. For example, when the wrist is laterally rotated, the entire upper limb can be in motion, and the shoulder and the elbow are both rotated. Thus, the rotation of the wrist can be indirectly controlled by controlling the lateral rotation of the shoulder and / or the elbow.

[0138] In some embodiments, the controlling the motion of the first type of local part according to the control mode and the second type of motion information comprises:

[0139] In the case where the control mode is the second type of control mode, the second type of motion information is decomposed to obtain the first type of rotation information of the connection part rotated by the traction part.

[0140] According to the first type of rotation information, the motion information of the traction part is adjusted.

[0141] The motion of the traction part in the controlled model is controlled by using the adjusted motion information of the traction part, so as to indirectly control the motion of the connection part.

[0142] In the present embodiment, the traction part is a local part directly connected with the second type of connection part. Taking the wrist as the second type of connection part, the traction part is the elbow or the upper arm above the wrist. Taking the ankle as the second type of connection part, the traction part is the knee or the thigh root above the ankle.

[0143] In the present embodiment, the first type of rotation information is not the rotation information generated by the motion of the second type of connection part itself, but the motion information generated by the motion of the other local part (i.e., the traction part) connected with the second type of connection part, which causes the second type of connection part to move relative to a specific reference point (e.g., the center of the human body) of the target.

[0144] In the present embodiment, the traction part is a local part directly connected with the second type of connection part. Taking the wrist as the second type of connection part, the traction part is the elbow or even the shoulder above the wrist. Taking the ankle as the second type of connection part, the traction part is the knee or even the thigh root above the ankle.

[0145] The lateral rotation of the wrist along the straight line direction of the shoulder, elbow to the wrist, can be driven by the shoulder or the elbow, and the motion information is detected by the motion of the wrist. Therefore, the lateral rotation information of the wrist should be assigned to the elbow or the shoulder, and the motion information of the elbow or the shoulder is adjusted through the assignment of the transmission; the adjusted motion information is used to control the motion of the elbow or the shoulder of the controlled model, so that the lateral rotation corresponding to the elbow or the shoulder is realized by the wrist of the controlled model from the effect column of the image; thereby, the precise simulation of the target motion of the controlled model is realized.

[0146] In some embodiments, the method further comprises:

[0147] Decomposing the second type of motion information to obtain second type of rotation information of the second type of connecting part relative to the traction part;

[0148] Using the second type of rotation information to control the rotation of the connecting part relative to the traction part of the controlled model.

[0149] In the present embodiment, the motion information of the second type of connecting part relative to the predetermined posture can be known by the characteristics of the second type of connecting part, such as 2D coordinates or 3D coordinates, which is called second motion information. The second type of motion information includes but is not limited to rotation information.

[0150] The first type of rotation information can be information obtained directly from the characteristics of the image by the information model for extracting the rotation information, and the second type of rotation information is the rotation information obtained by adjusting the first type of rotation information. In some embodiments, the first type of connecting part includes: elbow; knee; and / or, the second type of connecting part includes: wrist; ankle.

[0151] In other embodiments, if the second type of connecting part is a wrist, the traction part corresponding to the wrist includes: upper arm; and / or, if the second type of connecting part is an ankle, the traction part corresponding to the ankle includes: lower leg.

[0152] In some embodiments, the first type of connecting part includes the neck connecting the head and the torso.

[0153] In some embodiments, the second type of motion information of the connecting part of the at least two parts is determined according to the characteristics of the at least two parts and the first motion constraint condition of the connecting part, including:

[0154] The orientation information of the at least two parts is determined according to the characteristics of the at least two parts;

[0155] The alternative orientation information of the connecting part is determined according to the orientation information of the at least two parts;

[0156] According to the second type of motion information of the connection part in the alternative orientation information that meets the first motion constraint condition.

[0157] In some embodiments, the determining of the alternative orientation information of the connection part according to the orientation information of at least two parts comprises:

[0158] According to the orientation information of at least two parts, the first alternative orientation and the second alternative orientation of the connection part are determined.

[0159] Two angles can be formed between the orientation information of two parts, and in the embodiment, the angle that meets the first motion constraint condition is taken as the second type of motion information.

[0160] For example, two angles are formed between the orientation of the face and the orientation of the torso, and the sum of the two angles is 180 degrees; suppose the two angles are the first angle and the second angle respectively; and the first motion constraint condition of the neck connecting the face and the torso is -90 to 90 degrees, then the angle exceeding 90 degrees is excluded from the first motion constraint condition; in this way, the abnormal situation that the rotation angle exceeds 90 degrees clockwise or counterclockwise, for example, 120 degrees, 180 degrees, can be reduced in the process of simulating the target motion of the controlled model.

[0161] For example, taking the neck as an example, the face is right, and the corresponding orientation of the neck can be right 90 degrees or left 270 degrees, but according to the physiological structure of the human body, the change of the orientation of the neck of the human body can not be through left rotation 270 degrees to make the neck right. At this time, the orientation of the neck is right 90 degrees and left 270 degrees, which are alternative orientation information, and the orientation information of the neck needs to be further determined, which needs to be determined according to the first motion constraint condition. In this example, the orientation of the neck is right 90 degrees, which is the target orientation information of the neck, and according to the right 90 degrees of the neck, the second type of motion information of the neck relative to the front face is right rotation 90 degrees.

[0162] In some embodiments, the selecting of the second type of motion information of the connection part in the alternative orientation information that meets the first motion constraint condition comprises:

[0163] From the first alternative orientation information and the second alternative orientation information, the target orientation information within the orientation change constraint range is selected.

[0164] The second type of motion information is determined according to the target orientation information.

[0165] The target orientation information here is the information that meets the first motion constraint condition.

[0166] In some embodiments, the determining the orientation information of the at least two parts according to the features of the at least two parts comprises:

[0167] obtaining a first key point and a second key point of each of the at least two parts;

[0168] obtaining a first reference point of each of the at least two parts, wherein the first reference point is a predetermined key point in the target;

[0169] generating a first vector based on the first key point and the first reference point, and generating a second vector based on the second key point and the first reference point;

[0170] determining the orientation information of each of the at least two parts based on the first vector and the second vector.

[0171] If the first part is a shoulder of a human body, the first reference point of the first part can be a waist key point or a midpoint of two crotch key points of the target. The second part is a face, and the first reference point of the second part can be a connection point of the face and a neck and a shoulder.

[0172] In some embodiments, the determining the orientation information of the at least two parts based on the first vector and the second vector comprises:

[0173] cross-multiplying the first vector and the second vector of one part to obtain a normal vector of a plane where the corresponding part is located.

[0174] If the normal vector is determined, the orientation of the plane where the part is located is also determined.

[0175] In some embodiments, the determining the first type of motion information of the at least two parts based on the features comprises:

[0176] obtaining a third 3D coordinate of the first type of connection part relative to a second reference point;

[0177] obtaining absolute rotation information of the first type of connection part according to the third 3D coordinate;

[0178] the controlling the motion of the corresponding part of the controlled model according to the first type of motion information comprises:

[0179] controlling the motion of the corresponding first type of connection part of the controlled model based on the absolute rotation information.

[0180] In some embodiments, the first reference point can be one of the key points of the target's skeleton, and the second reference point can be a key point of a local part connected by the first type of connection. For example, taking the neck as an example, the second reference point can be a key point of the shoulder connected by the neck.

[0181] In other embodiments, the second reference point can be the same as the first reference point, for example, the first reference point and the second reference point can both be the root node of the human body, and the root node of the human body can be the midpoint of the line connecting the two key points of the hip of the human body. The root node includes but is not limited to Figure 8A The key point 0 is shown. Figure 8A It is a skeleton diagram of the human body, and Figure 8A It contains 17 skeleton key nodes with labels 0 to 16.

[0182] In other embodiments, the method further comprises:

[0183] According to the traction level relationship between the plurality of first type of connections in the target, the absolute rotation information is decomposed to obtain relative rotation information;

[0184] Based on the relative rotation information, the motion of the corresponding first type of connection in the controlled model is controlled.

[0185] In some embodiments, the method further comprises:

[0186] According to the second motion constraint condition, the relative rotation information is corrected;

[0187] The method further comprises:

[0188] Based on the corrected relative rotation information, the motion of the first type of connection in the controlled model is controlled.

[0189] In some embodiments, the second motion constraint condition includes the rotatable angle of the first type of connection.

[0190] In some embodiments, the method further comprises:

[0191] The second type of motion information is corrected for posture defects to obtain calibrated second type of motion information;

[0192] The method further comprises:

[0193] The motion of the connecting part of the controlled model is controlled using the calibrated second type of motion information.

[0194] For example, some users have issues with their posture and walking coordination. To ensure that the controlled model can directly mimic these unusual movements, this embodiment first performs posture defect correction on the second type of motion information to obtain calibrated second type of motion information.

[0195] like Figure 5A , Figure 5B and Figure 5C As shown, the smaller image on the left is the captured image, which includes a controlled model of the human body. The user's hand is moving, from... Figures 5A to 5B , and then from Figures 5B to 5C The user's hand moves, and the controlled model's hand follows suit. The user's hand movements are... Figures 5A to 5C The gestures change sequentially from clenching a fist, extending the palm, and then extending the index finger, while the controlled model mimics the user's gestures from clenching a fist, extending the palm, and then extending the index finger.

[0196] Figure 6A , Figure 6B and Figure 6C As shown, the smaller image on the left is the captured image, which includes a controlled model of the human body. Figures 6A to 6C The user steps to the right of the image, then to the left, and finally stands up straight; the controlled model also simulates the user's foot movements.

[0197] In some embodiments, step S120 may include: obtaining the first 2D coordinates of the limb based on the 2D image; step S130 may include: obtaining the first 3D coordinates corresponding to the first 2D coordinates based on the first 2D coordinates and the transformation relationship from 2D coordinates to 3D coordinates.

[0198] 2D coordinates are coordinates within a planar coordinate system, while 3D coordinates are coordinates within a 3D coordinate system. 2D coordinates represent the coordinates of key points in a plane, while 3D coordinates represent the coordinates in three-dimensional space.

[0199] The transformation relationship can be a pre-defined transformation function of various types. For example, taking the location of the image acquisition module as the virtual viewpoint, the virtual 3D space corresponding to the acquisition target and the image acquisition module at a predetermined distance is set. By projecting the 2D coordinates into the 3D space, the first 3D coordinates corresponding to the first 2D coordinates can be obtained.

[0200] In some other embodiments, step S120 may include: obtaining second 3D coordinates of key skeletal points of the target's limbs based on the 3D image; step S130 may include: obtaining third 2D coordinates based on the second 3D coordinates.

[0201] For example, the 3D image directly obtained in step S110 includes a 2D image and a depth image corresponding to the 2D image. The 2D image can provide coordinate values of the skeleton key points in the xoy plane, and the depth values in the depth image can provide coordinates of the skeleton key points on the z axis. The z axis is perpendicular to the xoy plane.

[0202] Based on the second 3D coordinates, the 3D coordinates of the skeleton key points corresponding to the occluded part of the limb in the 3D image are adjusted to obtain the third 2D coordinates.

[0203] Further, the third 2D coordinates obtained based on the second 3D coordinates include:

[0204] Based on the second 3D coordinates, the 3D coordinates of the skeleton key points corresponding to the occluded part of the limb in the 3D image are adjusted to obtain the third 2D coordinates.

[0205] For example, the user turns sideways to the image acquisition module, and the knee positions of the two legs have the same depth value in the depth image. At this time, the knee closer to the image acquisition module occludes the knee farther from the image acquisition module. In order to reduce the problem of inaccurate extraction of 3D coordinates in the depth image caused by such occlusion, the 3D coordinates of the skeleton key points are adjusted by using a deep learning model or a machine learning model, so as to reduce the fourth 3D coordinates obtained based on the second 3D coordinates to accurately represent the first type of motion information.

[0206] In some embodiments, the first type of motion information includes a quaternion.

[0207] The quaternion can be used to accurately represent the spatial position and / or rotation in each direction of the second type of local part.

[0208] In addition to using a quaternion to represent the first type of motion information, it can also be Euler coordinates or Lagrangian coordinates, etc.

[0209] Those skilled in the art can understand that in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0210] As shown in Figure 7 The embodiment provides an image processing apparatus, which comprises:

[0211] The first acquisition module 110 is configured to acquire an image.

[0212] The second acquisition module 120 is configured to acquire a feature of a limb of the body based on the image, wherein the limb includes an upper limb and / or a lower limb.

[0213] The first determination module 130 is configured to determine first motion information of the limb based on the feature.

[0214] The control module 140 is configured to control the motion of the limb of the controlled model according to the first motion information.

[0215] In some embodiments, the first acquisition module 110, the second acquisition module 120, the first determination module 130 and the control module 140 can be program modules, which can realize the speed of the image, the feature determination of the limb, the determination of the first motion information and the motion control of the controlled model after being executed by a processor.

[0216] In some other embodiments, the first acquisition module 110, the second acquisition module 120, the first determination module 130 and the control module 140 can be soft and hard combined modules; the soft and hard combined modules can include various programmable arrays; the programmable arrays include but are not limited to field programmable arrays or complex programmable arrays.

[0217] In some other embodiments, the first acquisition module 110, the second acquisition module 120, the first determination module 130 and the control module 140 can be pure hardware modules, which include but are not limited to application specific integrated circuits.

[0218] In some embodiments, the first determination module 130 is specifically configured to detect position information of a key point of the limb in the image; and determine the first motion information according to the position information.

[0219] In some embodiments, the apparatus further includes:

[0220] The detection module is configured to detect position information of a key point of a body skeleton in the image.

[0221] The second acquisition module 120 is specifically configured to determine the position information of the key point of the limb based on the position information of the key point of the body skeleton.

[0222] In some embodiments, the first determination module 130 is specifically configured to determine a position box of a first partial part containing the limb in the image according to the position information of the key point of the limb; detect position information of a key point of the first partial part based on the position box; and obtain first motion information of the first partial part based on the position information.

[0223] In some embodiments, the first determining module 130 is specifically configured to determine a position box containing the hand in the image according to the position information of the hand key points.

[0224] In some embodiments, the first determining module 130 is further configured to detect position information of key points corresponding to knuckles and / or position information of key points corresponding to finger tips on the hand based on the position box.

[0225] In some embodiments, the first determining module 130 is further configured to obtain motion information of the fingers of the hand based on the position information.

[0226] In some embodiments, the first determining module 130 is further configured to determine first type motion information of a second part of the limb according to the position information of the key points of the limb.

[0227] In some embodiments, the apparatus further comprises:

[0228] A second determining module configured to determine second type motion information of the connecting part based on position information of key points of two parts of the limb connected by the connecting part.

[0229] In some embodiments, the control module 140 is specifically configured to determine second type motion information of the connecting part of the at least two parts according to the features of the at least two parts and the first motion constraint condition of the connecting part, and control the motion of the connecting part of the controlled model according to the second type motion information.

[0230] In some embodiments, the control module 140 is further configured to determine a control mode for controlling the connecting part according to the type of the connecting part, and control the motion of the connecting part of the controlled model according to the control mode and the second type motion information.

[0231] In some embodiments, the control module 140 is further configured to determine that a first type control mode is used when the connecting part is a first type connecting part, wherein the first type control mode is used to directly control a first type controlled part of the controlled model which is the same as the first type connecting part.

[0232] In some embodiments, the control module 140 is further configured to determine that a second type control mode is used when the connecting part is a second type connecting part, wherein the second type control mode is used to indirectly control the second type connecting part by controlling parts of the controlled model other than the second type connecting part.

[0233] In some embodiments, the control module 140 is further configured to, in the case that the control mode is the second type of control mode, decompose the second type of motion information to obtain first type of rotation information of the connecting part being rotated by the pulling part; adjust the motion information of the pulling part according to the first type of rotation information; and control the motion of the pulling part in the controlled model by using the adjusted motion information of the pulling part, so as to indirectly control the motion of the connecting part.

[0234] In some embodiments, the control module 140 is further configured to decompose the second type of motion information to obtain second type of rotation information of the second type of connecting part relative to the pulling part; and control the rotation of the connecting part relative to the pulling part in the controlled model by using the second type of rotation information.

[0235] In some embodiments, the first type of connecting part includes: an elbow; a knee;

[0236] and / or,

[0237] The second type of connecting part includes: a wrist; an ankle.

[0238] In some embodiments, if the second type of connecting part is a wrist, the pulling part corresponding to the wrist includes: an upper arm or a forearm;

[0239] and / or,

[0240] If the second type of connecting part is an ankle, the pulling part corresponding to the ankle includes: a thigh or a shank.

[0241] In some embodiments, the second acquisition module 120 is specifically configured to acquire first 2D coordinates of the limb based on a 2D image.

[0242] The first determination module 130 is configured to obtain first 3D coordinates corresponding to the first 2D coordinates based on the first 2D coordinates and a conversion relationship from 2D coordinates to 3D coordinates.

[0243] In some embodiments, the second acquisition module 120 is specifically configured to acquire second 3D coordinates of the skeleton key points of the target limb based on a 3D image.

[0244] The first determination module 130 is specifically configured to obtain third 2D coordinates based on the second 3D coordinates.

[0245] In some embodiments, the first determination module 130 is specifically configured to adjust the 3D coordinates of the skeleton key points corresponding to the occluded part of the limb in the 3D image based on the second 3D coordinates, so as to obtain the third 2D coordinates.

[0246] In some embodiments, the first type of motion information comprises a quaternion.

[0247] Several specific examples are provided below in connection with any of the above embodiments:

[0248] Example 1:

[0249] The present example provides an image processing method, comprising:

[0250] The camera collects pictures one by one, the pictures are sent in, there is a portrait in the picture, the portrait is first detected, then the key points of the hand and wrist of the person are detected, and then the hand frame is detected, and then the 14 key points of the human body skeleton and the 63 contour points are obtained, mainly using the 14 points, after detecting the key points, the position of the hand is known, and then the hand frame is calculated. The hand frame here corresponds to the position frame mentioned above.

[0251] The goal of the hand frame is to put the hand frame in, generally the wrist frame will not be put in, but sometimes it is not so strict, such as when the position is tilted, it may include part of the wrist. The key points of the wrist are detected, and from different angles, they may be located at different positions of the wrist, but there is only one point, which is equivalent to the center point of the connection between the hand and the arm.

[0252] Now there are key points of the human body and the hand, and there is a network that can directly calculate 3D points from 2D points. The input of the network is 2D points, and the output is 3D points.

[0253] To drive a virtual model with this 3D skeleton, the angle parameters of the skeleton need to be obtained, and the key points cannot directly drive the controlled model. For example, the angle of the arm bending is calculated according to the positions of the three key points of the arm, and then the angle is assigned to the Avatar, so when you do an action, the Avatar will do the same action. There is a tool to express the angle: quaternion. Assign the quaternion to the Avatar model. The Avatar model here is one of the aforementioned controlled models.

[0254] For 3D cases, after adding Time Of Flight (TOF) information (TOF information is the original information of depth information), depth information, i.e. z-axis depth value, is added. Adding the original 2D image coordinate system plane xoy can find the corresponding z depth, so xyz is used as input. Here, xyz may have occlusions. After learning through a neural network or other depth model, the occluded points will be filled in, and the original depth can be directly used if possible. After learning through the network, a complete human body skeleton can be obtained. The effect will be better after using TOF, because RGB does not have depth perception, and after having depth information, the input information is stronger, and the accuracy is higher.

[0255] The controlled model in the example can be a game character in a game scene, a teacher model in a network education video in a network teaching scene, or a virtual anchor in a virtual anchor scene. In general, the controlled model is determined according to the application scene, and the application scene is different, and the model and / or appearance of the controlled model are different.

[0256] For example, in a traditional mathematics, physics, or other lecture scene, the teacher model is dressed in a relatively conservative manner, and may be a suit or other clothing that conforms to the identity of a teacher. For example, for a yoga or gymnastics teaching scene, the controlled model wears sports clothing.

[0257] Example 2

[0258] The present example provides an image processing method, comprising:

[0259] An image is collected, which includes a target, which includes but is not limited to a human body;

[0260] The torso key points and limb key points of the human body are detected, where the torso key points and / or limb key points can be 3D key points represented by 3D coordinates; the 3D coordinates can include 2D coordinates detected from a 2D image, and then obtained by using a 2D coordinate to 3D coordinate conversion algorithm; the 3D coordinates can also be 3D coordinates extracted from a 3D image collected by a 3D camera. The limb key points can include upper limb key points and / or lower limb key points; for example, taking a hand as an example, the hand key points of the upper limb key points include but are not limited to wrist joint key points, finger joint key points, finger key points, and fingertip key points; the positions of these key points can reflect the movement of the hand and the fingers.

[0261] The torso key points are converted into quaternions representing torso movement, which can be referred to as torso quaternions;

[0262] The limb key points are converted into quaternions representing limb movement, which can be referred to as limb quaternions;

[0263] The torso quaternions are used to control the torso movement of the controlled model;

[0264] The limb quaternions are used to control the limb movement of the controlled model.

[0265] For example, the face key points can include 106 key points;

[0266] The torso key points and limb key points can include 14 key points or 17 key points. Figure 8A 17 key points are shown in the example.

[0267] The controlled model in this example can be a game character in a game scene; a teacher model in a network education video in a network teaching scene; a virtual anchor in a virtual anchor scene. In general, the controlled model is determined according to the application scene, and the application scene is different, and the model and / or appearance of the controlled model are different.

[0268] For example, in a traditional mathematics, physics, etc. lecture scene, the teacher model is dressed in a more conservative manner, and may be a suit, etc. which is more suitable for the identity of a teacher. For example, for a yoga or gymnastics teaching scene, the controlled model wears sports clothes.

[0269] Example 3:

[0270] The example provides an image processing method, comprising:

[0271] Obtaining an image containing a target, which can be a human body;

[0272] According to the image, obtaining a 3D pose of the target in a three-dimensional space, which can be represented by 3D coordinates of the skeleton key points of the human body;

[0273] Obtaining the absolute rotation parameters of the joints of the human body in the camera coordinate system, which can be obtained by the coordinates in the camera coordinate system;

[0274] According to the coordinates, obtaining the coordinate direction of the joints;

[0275] According to the hierarchical relationship, determining the relative rotation parameters of the joints, which can specifically include determining the position of the key points of the joints relative to the root node of the human body; the relative rotation parameters can be represented by quaternions; the hierarchical relationship here can be the traction relationship between the joints, for example, the movement of the elbow joint will to some extent pull the movement of the wrist joint, and the movement of the shoulder joint will also pull the movement of the elbow joint, etc.; the hierarchical relationship can be determined in advance according to the joints of the human body.

[0276] Using the quaternions to control the rotation of the controlled model.

[0277] For example, the following is an example of a hierarchical relationship:

[0278] First level: pelvis;

[0279] Second level: waist;

[0280] Third level: thigh (for example, left thigh, right thigh);

[0281] Fourth level: calf (for example, left calf, right calf);

[0282] Fifth level: foot.

[0283] For another example, the following is another hierarchical relationship;

[0284] First level: chest;

[0285] Second level: neck;

[0286] Third level: head.

[0287] For further example, the following is yet another hierarchical relationship:

[0288] First level: collarbone, corresponding to shoulder;

[0289] Second level: upper arm;

[0290] Third level: forearm (also known as lower arm);

[0291] Fourth level: hand.

[0292] The first level to the fifth level are in a hierarchical relationship, in which the motion of a higher level part affects the motion of a lower level part. Therefore, the level of a pulling part is higher than the level of a connecting part.

[0293] In determining the second type of motion information, first, the motion information of the key points of each level is obtained, and then based on the hierarchical relationship, the motion information of the key points of a lower level relative to the key points of a higher level (i.e., the relative rotation information) is determined.

[0294] For example, if the motion information is represented by a quaternion, the relative rotation information can be represented by the following technical formula:

[0295] The rotation quaternion of each key point relative to the camera coordinate system is Q 18 , and then the rotation quaternion of each key point relative to the parent key point is calculated as q i

[0296]

[0297] The parent key point parent(i) is the key point of the previous level of the current key point i. Q i is the rotation quaternion of the current key point i relative to the camera coordinate system; is the inverse rotation parameter of the key point of the previous level. For example, Q parent(i) is the rotation parameter of the key point of the previous level, and the rotation angle is 90 degrees; then the rotation angle of q

[0298] The aforementioned use of the quaternion to control the motion of each joint of the controlled model can include: using q i to control the motion of each joint of the controlled model.

[0299] A further processing: the method further comprises:

[0300] converting the quaternion into first Euler angles;

[0301] transforming the first Euler angles to obtain second Euler angles within a constraint condition; the constraint condition can be an angle restriction on the first Euler angles;

[0302] obtaining a quaternion corresponding to the second Euler angles, and using the quaternion to control the rotation of the controlled model. The obtaining of the quaternion corresponding to the second Euler angles includes, for example, directly converting the second Euler angles into the quaternion.

[0303] For example, taking the human body as an example, 17 joint key points can be detected through human body detection, and 2 key points are set corresponding to the left hand and the right hand, totaling 19 key points. Figure 8A The skeleton diagram of 17 key points. Figure 8B The skeleton diagram of 19 key points. Figure 8A Forming Figure 8B The skeleton diagram shown. Figure 8B The skeleton shown can correspond to 19 key points, which respectively refer to the following bones:

[0304] pelvis, waist, left thigh, left lower leg, left foot; right thigh, right lower leg, right foot, chest, neck, head, left clavicle, right clavicle, right upper arm, right forearm, right hand, left upper arm, left forearm, left hand.

[0305] First, the coordinates of 17 key points in the image coordinate system can be obtained through the detection of the key points of the human body joints in the image, which can be specifically as follows:

[0306] S={(x0,y0,z0),(x i ,y i ,z i )…,(x 16 ,y 16 ,z 16 )}

[0307] (x i ,y i ,z i ) can be the coordinates of the i-th key point, where i takes a value from 0 to 16.

[0308] The coordinates of the 19 joint key points in the respective local coordinate systems can be defined as follows:

[0309] A={(p0,q0),…,(p 18 ,q 18 )}.

[0310] The process of calculating the quaternion of each joint corresponding key point can be as follows:

[0311] It needs to be determined that the coordinate axis direction of each node local coordinate system. For each bone, the direction from the child node to the parent node is the x axis; the vertical direction of the plane on which the two bones connected by a node is the z axis; if the rotation axis cannot be determined, the direction facing the human body is the y axis; Figure 9 The schematic diagram of the local coordinate system of the A node is shown.

[0312] This example uses the left-hand coordinate system for illustration, and the right-hand coordinate system can also be used in specific implementation.

[0313]

[0314] In the above table, (i-j) represents a vector pointing from i to j, and x represents a cross product.

[0315] The process of solving the first Euler angle is as follows:

[0316] Calculate the joint local rotation quaternion q i Then we first convert it to Euler angles, using the x-y-z order by default.

[0317] Let q i =(q0, q1, q2, q3), q0 is the real part; q1, q2, q3 are all imaginary parts.

[0318] X = atan2(2*(q0*q1-q2*q3), 1-2*(q1*q1-q2*q2))

[0319] Y = asin(2*(q1*q3-q0*q2)) and Y is between -1 and 1.

[0320] X = atan2(2*(q0*q3-q1*q2), 1-2*(q2*q2-q3*q3)).

[0321] X is the Euler angle in the first direction; Y is the Euler angle in the second direction; Z is the Euler angle in the third direction. Any two of the first, second and third directions are perpendicular.

[0322] Then the three angles (X, Y, Z) can be limited, and if they exceed the limit, they will be limited to the boundary value to obtain the adjusted second Euler angle (X', Y', Z'). Recover to the new local coordinate system rotation quaternion q i '.

[0323] The process of solving the first Euler angle is as follows:

[0324] A joint local rotation quaternion q is calculated i After that, we first convert it into Euler angles, using the x-y-z order by default.

[0325] Let q i =(q0, q1, q2, q3), q0 is the real part; q1, q2, q3 are all imaginary parts.

[0326] X = atan2(2*(q0*q1-q2*q3), 1-2*(q1*q1-q2*q2))

[0327] Y = asin(2*(q1*q3-q0*q2)) and Y is between -1 and 1.

[0328] X = atan2(2*(q0*q3-q1*q2), 1-2*(q2*q2-q3*q3)).

[0329] X is the Euler angle in the first direction; Y is the Euler angle in the second direction; Z is the Euler angle in the third direction. Any two of the first direction, the second direction and the third direction are perpendicular.

[0330] Then the three angles (X, Y, Z) can be limited, and if they exceed the limit, they will be limited to the boundary value to obtain the second Euler angle (X', Y', Z') after adjustment. Recovered into a new local coordinate system rotation quaternion q i '.

[0331] Another further processing: the method further comprises:

[0332] Optimizing the pose adjustment of the second Euler angle, for example, adjusting some of the angles in the second Euler angle, for example, based on a pre-set rule, adjusting to a pose-optimized Euler angle adjustment to obtain a third Euler angle; and the obtaining of the quaternion corresponding to the second Euler angle can comprise: converting the third Euler angle into a quaternion for controlling the controlled model.

[0333] Another further processing: the method further comprises:

[0334] After converting the second Euler angle into a quaternion, the converted quaternion data is subjected to pose optimization processing, for example, based on a pre-set rule to adjust to obtain an adjusted quaternion again, and the controlled model is controlled according to the finally adjusted quaternion.

[0335] In some embodiments, when adjusting the second Euler angle or the quaternion obtained by converting the second Euler angle, the adjustment can be based on a pre-set rule, or it can be optimized and adjusted by a deep learning model; there are many specific implementation ways, which are not limited here.

[0336] Pre-processing:

[0337] According to the size of the collected human body, the width of the controlled model's crotch and / or shoulder is modified;

[0338] Adjust the overall posture of the human body, for example, the standing posture of the human body; make a straightening adjustment, make an abdominal adjustment; for example, some people stand with their abdomen, and the controlled model can be made not to simulate the user's abdominal action through the abdominal adjustment. For example, some people stand with their backs hunched, and the controlled model can be made not to simulate the user's hunched back action through the hunched back adjustment, and so on.

[0339] Example 4:

[0340] The present example provides an image processing method, comprising:

[0341] Obtaining an image, the image containing a target, which can include at least one of a human body, an upper limb of a human body, and a lower limb of a human body;

[0342] Obtaining a coordinate system of a target joint according to position information of the target joint in an image coordinate system;

[0343] Obtaining a coordinate system of a local limb that will pull the movement of the target joint according to position information of the local limb in the image coordinate system;

[0344] Determining the rotation of the target joint relative to the local limb based on the coordinate system of the target joint and the coordinate system of the local limb, to obtain a rotation parameter; the rotation parameter includes a self-rotation parameter of the target joint and a pulled rotation parameter by the local limb;

[0345] Limiting the pulled rotation parameter by the local limb by using a first angle limit to obtain a final pulled rotation parameter;

[0346] Adjusting the rotation parameter of the local limb according to the pulled rotation parameter;

[0347] According to the relative rotation parameter of the local limb of the first limb and the adjusted rotation parameter;

[0348] Carrying out a second angle limit on the relative rotation parameter to obtain a limited relative rotation parameter;

[0349] Obtaining a quaternion from the limited rotation parameter;

[0350] Controlling the movement of the target joint of the controlled model according to the quaternion.

[0351] For example, if the human upper limb is processed, the coordinate system of the hand is obtained in the image coordinate system, and the coordinate system of the forearm and the coordinate system of the upper arm are obtained; at this time, the target joint is the wrist joint;

[0352] determine the rotation of the hand relative to the forearm, and obtain a relative rotation parameter;

[0353] pass the driven rotation to the forearm, specifically, the driven rotation is assigned to the rotation of the forearm in the corresponding direction; and limit the maximum rotation of the forearm by using the first angle limit of the forearm;

[0354] determine the rotation of the hand relative to the adjusted forearm, and obtain a relative rotation parameter;

[0355] perform a second angle limit on the relative rotation parameter, and obtain the rotation of the hand relative to the forearm.

[0356] If the lower limbs of the human body are processed, the coordinate system of the foot is obtained in the image coordinate system, the coordinate system of the lower leg is obtained, and the coordinate system of the upper leg is obtained; at this time, the target joint is the ankle joint.

[0357] determine the rotation of the foot relative to the lower leg, and decompose the rotation into self-rotation and driven rotation;

[0358] pass the driven rotation to the lower leg, specifically, the driven rotation is assigned to the rotation of the lower leg in the corresponding direction; and limit the maximum rotation of the lower leg by using the first angle limit of the lower leg;

[0359] determine the rotation of the foot relative to the adjusted lower leg, and obtain a relative rotation parameter;

[0360] perform a second angle limit on the relative rotation parameter, and obtain the rotation of the foot relative to the lower leg.

[0361] As shown in Figure 10 , the embodiment of the present application provides an image device, which comprises:

[0362] a memory for storing information;

[0363] a processor connected with the display and the memory respectively, for realizing the image processing method provided by one or more of the foregoing technical solutions by executing computer executable instructions stored in the memory, for example, the image processing method as shown in Figure 1 and / or Figure 2 .

[0364] The memory can be various types of memories, which can be random access memory, read-only memory, flash memory, etc. The memory can be used for information storage, for example, storing computer executable instructions, etc. The computer executable instructions can be various program instructions, for example, object program instructions and / or source program instructions, etc.

[0365] The processor can be various types of processors, such as a central processing unit, a microprocessor, a digital signal processor, a programmable array, a digital signal processor, an application specific integrated circuit, or an image processor, etc.

[0366] The processor can be connected with the memory through a bus. The bus can be an integrated circuit bus, etc.

[0367] In some embodiments, the terminal device can further include a communication interface, which can include a network interface, such as a local area network interface, a transceiving antenna, etc. The communication interface is also connected with the processor and can be used for information transceiving.

[0368] In some embodiments, the terminal device further includes a human-computer interaction interface, which can include various input and output devices, such as a keyboard, a touch screen, etc.

[0369] In some embodiments, the image device further includes a display, which can display various prompts, collected face images, and / or various interfaces.

[0370] Embodiments of the present application provide a computer storage medium, which stores computer executable code; the computer executable code is executed to implement the image processing method provided in one or more of the preceding technical solutions, for example, the image processing method shown in Figure 1 and / or Figure 4 .

[0371] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be mutually referred to, and for brevity, will not be repeated here.

[0372] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed various constituent parts can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0373] The units described as separate parts above can or can not be physically separate, the parts shown as units can or can not be physical units, that is, can be located in one place or distributed on multiple network units; part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0374] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can be separately implemented as a single unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional unit.

[0375] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps including the above method embodiments when executed; and the foregoing storage medium includes mobile storage device, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various storage program codes.

[0376] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that, include: Acquire images of a target object; the target object includes limbs of the body; Based on the image, the features of the limbs of the body are obtained; the limbs include at least two parts and a connecting portion connecting the two parts; Based on the characteristics of the limb, a second type of motion information of the connecting part is determined; Based on the second type of motion information from the connecting part, the movement of the limbs of the controlled model is controlled; The step of controlling the movement of the limbs of the controlled model based on the second type of motion information of the connecting part includes: Based on the features of the two local parts and the first motion constraint condition of the connecting part, the second type of motion information of the connecting part is determined; When the connecting part is a first type of connecting part, it is determined that a first type of control method is adopted, wherein the first type of control method is used to directly control the first type of controlled part of the controlled model that is the same as the first type of connecting part; When the connecting part is a second type of connecting part, it is determined that a second type of control method is adopted, wherein the second type of control method is used to indirectly control the second type of connecting part by controlling a part other than the second type of connecting part of the controlled model; The movement of the connecting part of the controlled model is controlled according to the control method of the connecting part and the second type of motion information.

2. The method according to claim 1, characterized in that, The method further includes: determining a first type of local motion information of the limb based on the characteristics of the limb; and controlling the movement of the limb of the controlled model according to the first type of motion information.

3. The method according to claim 1 or 2, characterized in that, The determination of the second type of motion information of the connecting part based on the characteristics of the limb includes: Based on the positional information of two key points in the limb connected by the connecting part, the second type of motion information of the connecting part is determined.

4. The method according to claim 1, characterized in that, The step of controlling the movement of the connecting part of the controlled model according to the control method of the connecting part and the second type of motion information includes: When the control mode of the connecting part is the second type of control mode, the second type of motion information is decomposed to obtain the first type of rotation information of the second type of connecting part being rotated by the traction part; The motion information of the traction unit is adjusted based on the first type of rotation information; By utilizing the adjusted motion information of the traction unit, the motion of the traction unit in the controlled model is controlled, thereby indirectly controlling the motion of the second type of connecting unit.

5. The method according to claim 4, characterized in that, The method further includes: Decompose the second type of motion information to obtain the second type of rotation information of the second type of connection relative to the traction part; Using the second type of rotation information, the rotation of the second type of connection portion of the controlled model relative to the traction portion is controlled.

6. The method according to claim 4 or 5, characterized in that, The first type of connecting part includes: elbow; knee; And / or, The second type of connecting part includes: wrist; ankle; In the case where the second type of connection is the wrist, the traction part corresponding to the wrist includes: the upper arm; And / or, In the case where the second type of connecting part is the ankle, the traction part corresponding to the ankle includes the lower leg.

7. An image processing apparatus, characterized in that, include: The first acquisition module is used to acquire images of a target object; the target object includes limbs of the body; The second acquisition module is used to acquire features of the limbs of the body based on the image; the limbs include at least two parts and a connecting part connecting the two parts; The second determining module is used to determine a second type of motion information of the connecting part based on the characteristics of the limb; The control module is used to control the movement of the limbs of the controlled model based on the second type of motion information of the connecting part; The control module is further configured to determine a second type of motion information of the connecting part based on the characteristics of the two local parts and the first motion constraint condition of the connecting part; if the connecting part is a first type of connecting part, determine to adopt a first type of control method, wherein the first type of control method is used to directly control the first type of controlled local part of the controlled model that is the same as the first type of connecting part; if the connecting part is a second type of connecting part, determine to adopt a second type of control method, wherein the second type of control method is used to indirectly control the second type of connecting part by controlling local parts of the controlled model other than the second type of connecting part; and control the motion of the connecting part of the controlled model based on the control method of the connecting part and the second type of motion information.

8. An imaging device, characterized in that, include: Memory; A processor, connected to the memory, is configured to implement the method provided by any one of claims 1 to 6 by executing computer-executable instructions located on the memory.

9. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions; when the computer-executable instructions are executed by a processor, they can implement the method provided by any one of claims 1 to 6.

Images