A pose determination method and apparatus

Abstract

The application discloses a pose determination method and an electronic device. Depth information of a target key point in a first image frame is obtained, the first image frame being a two-dimensional image containing a target object; first position information of the target key point in the first image frame is predicted based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame in time sequence; and pose information of a virtual object is determined based on the depth information and the first position information, the pose information of the virtual object being capable of representing a real action of the target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information processing, and in particular to a pose determination method and device. BACKGROUND

[0002] Currently, the development trend of the metaverse is rapid, and virtual people are a crucial existence factor. Virtual people driving schemes include inertial motion capture, optical motion capture, and visual motion capture schemes.

[0003] For the visual motion capture scheme, the position and state of the virtual person need to be determined, and there will be a large error if only a single camera is used to determine it. SUMMARY

[0004] Therefore, the present application provides a pose determination method and device, and the specific scheme is as follows:

[0005] A pose determination method comprises:

[0006] obtaining depth information of a target key point in a first image frame, the first image frame being a two-dimensional image containing a target object;

[0007] predicting first position information of the target key point in the first image frame based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame in time sequence;

[0008] determining pose information of a virtual object based on the depth information and the first position information, the pose information of the virtual object being capable of representing real actions of the target object.

[0009] Further, the obtaining of the depth information of the target key point in the first image frame comprises:

[0010] determining a set of to-be-extracted depth points of each target key point in the first image frame based on a set extraction radius, and obtaining a set of depth information of the to-be-extracted depth points;

[0011] performing filtering processing on the set of depth information to obtain the depth information of the target key point.

[0012] Further, the predicting of the first position information of the target key point in the first image frame based on the pose information of the target key point in the second image frame comprises:

[0013] obtaining a time interval between the first image frame and the second image frame;

[0014] predicting position information of the target key point after the time interval based on second position information and motion information of the target key point in the second image frame, and determining the predicted position information as the first position information.

[0015] Further, the determining the pose information of the virtual object based on the depth information and the first position information comprises:

[0016] obtaining third position information of a target key point of the virtual object based on the first image frame;

[0017] if a difference between the depth information and second position information of the target key point in the second image frame is greater than a first threshold, determining the pose information of the virtual object based on the third position information and the first position information;

[0018] if a difference between the first position information and the third position information is greater than a second threshold, determining the pose information of the virtual object based on the third position information;

[0019] if the difference between the depth information and the second position information is not greater than the first threshold, and the difference between the first position information and the third position information is not greater than the second threshold, determining the pose information of the virtual object based on the depth information and the third position information.

[0020] Further, the determining the pose information of the virtual object based on the third position information and the first position information comprises:

[0021] determining target depth information of a first target key point of the virtual object based on the third position information and the first position information;

[0022] obtaining three-dimensional coordinate information of the first target key point based on the target depth information and plane coordinate information of the first target key point;

[0023] determining the pose information of the virtual object based on the three-dimensional coordinate information and position information between the first target key point and at least one second target key point.

[0024] Further, the determining the pose information of the virtual object based on the depth information and the third position information comprises:

[0025] determining target depth information of a first target key point of the virtual object based on the third position information and the depth information, to obtain three-dimensional coordinate information of the first target key point;

[0026] determining the pose information of the virtual object based on the three-dimensional coordinate information and a positional relationship between the first target key point and at least one second target key point.

[0027] Further, the method further comprises:

[0028] The position change information of the target key point of the virtual object is used to determine a motion state of the virtual object.

[0029] Further, the position change information of the target key point of the virtual object is used to determine a motion state of the virtual object, including:

[0030] If the position change amount of the first key point of the virtual object reaches a third threshold value, and a motion parameter reaches a fourth threshold value, a motion trend of at least one second key point of the virtual object is determined.

[0031] If the at least one second key point is in a first motion trend and a motion parameter of the second key point satisfies a corresponding condition, it is determined that the virtual object is in a first motion state.

[0032] Further, the method further includes:

[0033] In response to determining that the virtual object is in the first motion state, a footstep reverse motion function node of the virtual object in a current frame is closed; and,

[0034] In response to determining that the virtual object is in a second motion state and a duration exceeds a fifth threshold value, a footstep reverse motion function node of the virtual object is opened.

[0035] A pose determination apparatus includes:

[0036] An obtaining unit is configured to obtain depth information of a target key point in a first image frame, the first image frame being a two-dimensional image containing a target object;

[0037] A prediction unit is configured to predict first position information of the target key point in the first image frame based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame in time sequence;

[0038] A pose determination unit is configured to determine pose information of a virtual object based on the depth information and the first position information, the pose information of the virtual object being capable of representing a real action of the target object.

[0039] An electronic device includes:

[0040] A processor is configured to obtain depth information of a target key point in a first image frame, the first image frame being a two-dimensional image containing a target object; predict first position information of the target key point in the first image frame based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame in time sequence; and determine pose information of a virtual object based on a comparison result between the depth information and the first position information, the pose information of the virtual object being capable of representing a real action of the target object.

[0041] a memory for storing programs for the processor to perform the above processes.

[0042] From the above technical solutions, the pose determination method and the electronic device disclosed by the present application can obtain the depth information of the target key point in the first image frame, the first image frame being a two-dimensional image containing a target object; predict the first position information of the target key point in the first image frame based on the pose information of the target key point in the second image frame, the second image frame being earlier than the first image frame in time sequence; determine the pose information of the virtual object based on the depth information and the first position information, the pose information of the virtual object being capable of representing the real action of the target object. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0044] Figure 1 a flow chart of a pose determination method disclosed by an embodiment of the present application;

[0045] Figure 2 a key point schematic diagram of a virtual object disclosed by an embodiment of the present application;

[0046] Figure 3 a flow chart of a pose determination method disclosed by an embodiment of the present application;

[0047] Figure 4 a flow chart of a pose determination method disclosed by an embodiment of the present application;

[0048] Figure 5 a flow chart of a pose determination method disclosed by an embodiment of the present application;

[0049] Figure 6 a structural schematic diagram of an electronic device disclosed by an embodiment of the present application;

[0050] Figure 7 a structural schematic diagram of a pose determination device disclosed by an embodiment of the present application. DETAILED DESCRIPTION

[0051] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0052] The present application discloses a pose determination method, the flow chart thereof is as shown in Figure 1 The present application discloses a pose determination method, the flow chart thereof is as shown in

[0053] Step S11, obtaining the depth information of the target key point in the first image frame, the first image frame being a two-dimensional image containing a target object;

[0054] Step S12, predicting the first position information of the target key point in the first image frame based on the pose information of the target key point in the second image frame, the second image frame being earlier than the first image frame at least in time sequence;

[0055] Step S13, determining the pose information of the virtual object based on the comparison result between the depth information and the first position information, the pose information of the virtual object being capable of representing the real action of the target object.

[0056] When determining the pose of the virtual object, if a single camera is used for visual motion capture, the determination accuracy of the depth information of the target key point is low; if a double camera scheme is used, the accuracy is high, but calibration is needed, which is not suitable for personal users; if a depth camera is used, the depth information of all points in the image frame needs to be calculated, and the frame rate of the collected frame cannot meet the requirement of real-time virtual object driving.

[0057] To solve the above problems, in the present scheme, while obtaining the image frame through a single camera, the depth information of the target key point in the image frame is obtained through a depth camera, which improves the accuracy of pose determination while ensuring timeliness.

[0058] The frame image in the movement process of the target object is obtained through the image acquisition device, and the position information of the target key point in the next frame image can be predicted through one frame image, that is, the first position information of the target key point in the first image frame is predicted based on the pose information of the target key point in the second image frame.

[0059] The pose information of the target key point in the second image frame is determined by the pose determination method disclosed in the present embodiment. The second image frame is earlier than the first image frame at least in time sequence, that is, the second image frame is obtained before the first image frame, and the position information of the target key point in the first image frame obtained thereafter can be predicted based on the determined pose information of the target key point in the second image frame.

[0060] After the first position information of the target key point in the first image frame is predicted through the pose information of the target key point in the second image frame, depth information of the target key point in the first image frame needs to be obtained based on the depth camera, so that the pose of the virtual object can be determined based on the predicted first position information of the target key point and the depth information of the target key point.

[0061] Wherein, the depth information of the target key point in the first image frame is obtained through the depth camera, and the depth information of the target key point in the first image frame can be obtained only through the depth camera, without the need to obtain the depth information of other points in the first image frame except the target point, so as to improve the detection speed and meet the timeliness.

[0062] Wherein, the target key point can be a specific point of the virtual object in the first image frame, such as a tailbone point of the virtual object, or a wrist key point of the virtual object, etc. As shown in Figure 2 , the tailbone point of the virtual object is P hip (x, y, z), and the wrist key point of the virtual object can be point 15 or point 16 in Figure 2 .

[0063] The pose determination method disclosed in this embodiment obtains the depth information of the target key point in the first image frame, and the first image frame is a two-dimensional image containing a target object; the first position information of the target key point in the first image frame is predicted based on the pose information of the target key point in the second image frame, and the second image frame is earlier than the first image frame at least in time sequence; the pose information of the virtual object is determined based on the depth information and the first position information, and the pose information of the virtual object can represent the real action of the target object. This scheme determines the pose information of the virtual object based on the predicted position information of the target key point in the image frame and the obtained depth information of the target key point, increases the determination of the depth information, and avoids the problem of large deviation caused by using a single camera for pose determination.

[0064] This embodiment discloses a pose determination method, and a flow chart thereof is shown in Figure 3 , which comprises:

[0065] Step S31, determining a set of to-be-extracted depth points of each target key point in the first image frame based on the set extraction radius, and obtaining a set of depth information of the to-be-extracted depth points;

[0066] Step S32, filtering the set of depth information to obtain the depth information of the target key point;

[0067] Step S33, predicting the first position information of the target key point in the first image frame based on the pose information of the target key point in the second image frame, and the second image frame is earlier than the first image frame at least in time sequence;

[0068] In step S34, the pose information of the virtual object is determined based on the depth information and the first position information, and the pose information of the virtual object can represent the real action of the target object.

[0069] In the process of obtaining the depth information of the target key point in the first image frame through the depth camera, the extraction radius can be set for the depth camera, so as to extract only the depth information of the key points within the extraction radius of the target key point, without the need to obtain the depth information of each point in the first image frame, thereby saving the extraction time of the depth information, avoiding the problem of time waste caused by the extraction of useless data, and improving the timeliness of the data. In addition, by setting the extraction radius and processing the depth information of the key points within the extraction radius, the accurate depth information of the target key point can be obtained, so as to improve the accuracy of the pose determination.

[0070] The extraction radius is set in the first image frame, and the extraction radius at least includes the target key point in the first image frame.

[0071] After the extraction radius is determined, first, it is determined which target key points in the first image frame are within the extraction radius, so as to set the to-be-extracted depth point set, which includes the information of the target key points within the extraction radius. After the depth information of the target key points is extracted, the depth information set corresponding to the to-be-extracted depth point set can be obtained, and the depth information set includes the depth information of each target key point in the to-be-extracted depth point set.

[0072] If there is only one target key point in the extraction radius, only the depth information of the target key point is extracted; if there are three target key points in the extraction radius, the depth information of the three target key points needs to be extracted.

[0073] After the depth information set is obtained, the depth information set is filtered to exclude the depth values with large deviations in the depth information set, so as to avoid extracting the depth values of the background in the first image frame, thereby obtaining the more accurate depth information of the target key points extracted by the depth camera.

[0074] Further, the first position information of the target key point in the first image frame is predicted based on the pose information of the target key point in the second image frame, including:

[0075] The time interval between the first image frame and the second image frame is obtained; the position information of the target key point after the time interval is predicted based on the second position information and the motion information of the target key point in the second image frame, and the predicted position information is determined as the first position information.

[0076] Since the second image frame is earlier in time sequence than the first image frame, the action of the virtual object in the second image frame is earlier than the action of the virtual object in the first image frame, and therefore, the time interval t between the first image frame and the second image frame needs to be determined.

[0077] If the second image frame and the first image frame are adjacent image frames, the time interval t between the second image frame and the first image frame is the time required for each frame of image; if the second image frame and the first image frame are non-adjacent image frames, the time interval t between the second image frame and the first image frame is a multiple of the time required for each frame of image.

[0078] The second position information of the target key point in the second image frame, i.e. the position information of the target key point in the second image frame determined when the pose of the virtual object in the second image frame is determined, is determined as the second position information d pre .

[0079] The motion information can be the velocity v pre and the acceleration a pre of the target key point in the second image frame, and based on the motion information and the second position information of the target key point in the second image frame, the first position information of the target key point in the first image frame can be predicted as:

[0080]

[0081] wherein d pnow is the first position information of the target key point in the first image frame, d pre is the second position information, v pre is the velocity of the target key point in the second image frame, a pre is the acceleration of the target key point in the first image frame, and t is the time interval between the first image frame and the second image frame.

[0082] When the first position information d pnow of the target key point in the first image frame and the depth information of the target key point in the first image frame are obtained, the pose information of the virtual object can be determined based on the first position information and the depth information, so as to realize the determination of the pose of the virtual object.

[0083] The pose determination method disclosed in the embodiment obtains depth information of a target key point in a first image frame, the first image frame being a two-dimensional image containing a target object; predicts first position information of the target key point in the first image frame based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame in time sequence; and determines pose information of a virtual object based on the depth information and the first position information, the pose information of the virtual object being capable of representing a real action of the target object. The scheme determines the pose information of the virtual object based on the predicted position information of the target key point in the image frame and the obtained depth information of the target key point, increases determination of the depth information, and avoids the problem of large deviation caused by pose determination by using a single camera.

[0084] The embodiment discloses a pose determination method, a flowchart of which is shown in Figure 4

[0085] In step S41, depth information of a target key point in a first image frame is obtained, the first image frame being a two-dimensional image containing a target object.

[0086] In step S42, first position information of the target key point in the first image frame is predicted based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame in time sequence.

[0087] In step S43, third position information of a target key point of a virtual object is obtained based on the first image frame.

[0088] In step S44, if a difference between the depth information and second position information of the target key point in the second image frame is greater than a first threshold value, pose information of the virtual object is determined based on the third position information and the first position information.

[0089] In step S45, if a difference between the first position information and the third position information is greater than a second threshold value, pose information of the virtual object is determined based on the third position information.

[0090] In step S46, if a difference between the depth information and the second position information is not greater than the first threshold value, and a difference between the first position information and the third position information is not greater than the second threshold value, pose information of the virtual object is determined based on the depth information and the third position information.

[0091] The second position information is position information d pre .

[0092] The third position information is position information of the target key point of the virtual object obtained based on the first image frame, that is, image recognition is performed on the first image frame by a depth model to obtain position information of the target key point in the first image frame, which is determined as the third position information d​ori .

[0093] To determine the pose information of the virtual object, in addition to the second position information d pre and the third position information d ori , the first position information d pnow and the depth information d now are also applied, wherein the depth information d now is the depth information of the target key point in the first image frame identified by the depth camera.

[0094] The position information mentioned in the embodiment refers to the depth of the target key point, i.e. the value of the z-axis in the spatial coordinate system, but does not include the value of the plane coordinate, i.e. the values of the x-axis and the y-axis.

[0095] Specifically, if the difference between the depth information and the second position information is greater than the first threshold, the pose information of the virtual object is determined based on the third position information and the first position information.

[0096] Comparing whether the difference between the depth information and the second position information is greater than the first threshold, since the depth information is the depth information of the target key point in the first image frame extracted by the depth camera, and the second position information is the position information of the target key point in the second image frame, comparing the depth information and the second position information is actually to determine whether the depth of the target key point between the two frames has a large span, thereby determining the pose information of the virtual object.

[0097] If the difference between the depth information d now and the second position information d pre is greater than the first threshold d y1 , the target depth information of the target key point of the virtual object is determined based on the third position information and the first position information, and then the pose information of the virtual object is determined based on the target depth information.

[0098] Specifically, if |d now -d pre |>d y1 , then

[0099]

[0100] wherein d is the target depth information, and V is the confidence of the third position information d ori of the target key point of the virtual object obtained according to the first image frame.

[0101] In addition, it is also necessary to determine whether there is a cumulative deviation between the front and rear frames, since the third position information d oriis a result predicted by the depth model, and does not have cumulative bias, while the first position information d pnow is calculated, and can have cumulative bias, so it is necessary to determine whether there is cumulative bias by comparing the first position information d pre with the third position information d pnow to improve the calculation accuracy. ori

[0102] Specifically, if |d pnow -d ori |>d y2 , that is, if the difference between the first position information d pnow and the third position information d ori is greater than the second threshold d y2 , it indicates that there is cumulative bias, and at this time, the pose information of the virtual object can be determined based on the third position information, that is, the third position information d ori is directly determined as the target depth information.

[0103] In addition, if the difference between the depth information d now and the second position information d pre is not greater than the first threshold d y1 , and the difference between the first position information d pnow and the third position information d ori is not greater than the second threshold d y2 , the pose information of the virtual object can be determined based on the depth information d now and the third position information d ori , that is, the target depth information is determined by using formula (3), and the pose information of the virtual object is further determined based on the target depth information.

[0104]

[0105] wherein d is the target depth information, and V is the confidence of the third position information d ori of the target key point of the virtual object obtained according to the first image frame.

[0106] Further, the pose information of the virtual object is determined based on the target depth information, including:

[0107] the three-dimensional coordinate information of the first target key point is obtained based on the target depth information of the first target key point and the plane coordinate information of the first target key point; and the pose information of the virtual object is determined based on the position relationship between the first target key point and at least one second target key point and the three-dimensional coordinate information.

[0108] ​Wherein, no matter whether the target depth information of the first target key point is calculated based on formula (2) or formula (3), after obtaining the target depth information, it is necessary to be combined with the plane coordinate information to determine the three-dimensional coordinate information of the target key point.

[0109] If the target depth information is obtained based on formula (2), it is actually the target depth information of the first target key point of the virtual object determined based on the third position information and the first position information; if the target depth information is obtained based on formula (3), it is actually the target depth information of the first target key point of the virtual object determined based on the third position information and the depth information.

[0110] Wherein, the target depth information is the value of the z-axis in the space coordinate system xyz, and the plane coordinate information is the value of xy in the space coordinate system xyz, only when both the plane coordinate information and the target depth information are included, the three-dimensional coordinate information in space is obtained.

[0111] When the three-dimensional coordinate information of a target key point is determined, the three-dimensional coordinate information of the target key point is combined with the three-dimensional coordinate information of other target key points, and the pose information of the virtual object based on these target key points can be obtained.

[0112] Wherein, the first target key point can be a coccyx point, and the at least one second target key point can be a wrist key point, a knee key point, etc.

[0113] The pose determination method disclosed in this embodiment obtains the depth information of the target key point in the first image frame, and the first image frame is a two-dimensional image containing a target object; predicts the first position information of the target key point in the first image frame based on the pose information of the target key point in the second image frame, and the second image frame is earlier than the first image frame in time sequence; determines the pose information of the virtual object based on the depth information and the first position information, and the pose information of the virtual object can represent the real action of the target object. This scheme determines the pose information of the virtual object based on the predicted position information of the target key point in the image frame and the obtained depth information of the target key point, increases the determination of the depth information, and avoids the problem of large deviation caused by pose determination using a single camera.

[0114] The pose determination method disclosed in this embodiment has a flowchart as shown in Figure 5 The pose determination method disclosed in this embodiment has a flowchart as shown in

[0115] Step S51, obtaining the depth information of the target key point in the first image frame, and the first image frame is a two-dimensional image containing a target object;

[0116] Step S52, predicting first position information of the target key point in the first image frame based on the pose information of the target key point in the second image frame, the second image frame being earlier than the first image frame in time sequence;

[0117] Step S53, determining the pose information of the virtual object based on the depth information and the first position information, the pose information of the virtual object being capable of representing the real action of the target object;

[0118] Step S54, determining the motion state of the virtual object based on the position information change of the target key point of the virtual object.

[0119] When the pose of the virtual object in each image frame is determined, the motion of the virtual object and the change of the motion state of the virtual object can be determined through the change of the pose or the change of the position between different image frames.

[0120] For example, it is determined that the virtual object is in a first pose and located at a first position in the second image frame, and it is determined that the virtual object is in a second pose and located at a second position in the first image frame, and the pose and the position of the virtual object both change, which indicates that the motion state of the first image frame relative to the second image frame changes.

[0121] Specifically, if the position change amount of the first key point of the virtual object reaches a third threshold value and the motion parameter reaches a fourth threshold value, the motion trend of at least one second key point of the virtual object is determined; if the at least one second key point is in a first motion trend and the motion parameter of the second key point satisfies a corresponding condition, it is determined that the virtual object is in a first motion state.

[0122] If the position change amount of the first key point of the virtual object reaches a third threshold value and the motion parameter reaches a fourth threshold value, it can be determined that the virtual object has a motion trend. The first key point can be a first target key point of the virtual object, such as a coccyx point, i.e. Figure 2 P hip (x, y, z) point.

[0123] The position change amount can be the position change amount of the first key point between two image frames, such as through comparison of the first image frame and the second image frame, it is determined that the position of the coccyx point P hip (x, y, z) changes from a first position to a second position, i.e. in the second image frame, the coccyx point P hip (x, y, z) is located at the first position, while in the first image frame, the coccyx point P hip (x, y, z) is located at the second position, the first position and the second position are different, and the position change amount of the second position relative to the first position reaches a third threshold value.

[0124] Motion parameters, such as: the speed or acceleration of the motion, or the angle of the motion, or the duration.

[0125] For example: by comparing the first image frame and the second image frame, the coccyx point P of the virtual object can be determined hip When the coccyx point P rises to a third threshold value, and the acceleration reaches a fourth threshold value, it is determined that the virtual object has a tendency to jump, and the motion trend of the second key point of the virtual object needs to be further determined.

[0126] The second key point can be a leg key point or a foot key point, etc. By determining the motion trend of the leg key point or the foot key point of the virtual object, it is determined whether the virtual object is currently in a jumping state or a non-jumping state, etc.

[0127] The determination of the motion trend of the second key point can be: if at least one second key point is in the first motion trend and the motion parameters of the second key point meet the corresponding conditions, it is determined that the virtual object is in the first motion state, wherein the first motion state is the take-off state. The plurality of leg key points or the plurality of foot key points can be determined as the second key points, and when the motion trend of the second key point is determined, the motion trend of the plurality of leg key points or the plurality of foot key points can be determined.

[0128] If the second key point is a leg key point, it can include leg key point 25, leg key point 26, leg key point 27, and leg key point 28 as shown in FIG. 25. Figure 2

[0129] The motion trend of the first image frame relative to the second image frame of the above-mentioned key points is determined. If it is determined that the motion trend of the first image frame relative to the second image frame of each second key point is downward motion, it indicates that at least one second key point is in the first motion trend.

[0130] If the second key point is a foot key point, it can include foot key point 29, foot key point 30, foot key point 31, and foot key point 32 as shown in FIG. 29. Figure 2

[0131] Specifically, according to the human kinematics law, the foot of a person will show a downward state for a period of time when jumping, and the maximum jumping height of a normal person is d max At this time, the rising speed of the coccyx is v1, and the angle range a of the vector direction downward formed by the foot key point 32 and the foot key point 30 is limited in combination with the rising distance d1 of the coccyx point x And the predicted duration t x may also be:

[0132] a x = d m ​​ / d max ×(π / 2)

[0133]

[0134] Among them, a x Let t be the downward angle of the vector formed by foot key points 32 and 30. x The predicted duration of the downward angle of the vector formed by foot keypoints 32 and 30, d. max The maximum jump height for a normal person, g is the acceleration due to gravity, and d m This represents the maximum upward distance for this jump.

[0135] Where, d m =(v1) 2 / 2g, where v1 is the rate of ascent of the coccyx point.

[0136] Based on the above formula, the downward angle α of the vector formed by the key points of the foot can be determined. x and predicted duration t x When the positional change and motion parameters of the coccyx point meet the corresponding conditions, and simultaneously, the vector formed by the key points of the foot points points downwards at an angle α... x and predicted duration t x When all the corresponding thresholds are met, the virtual object can be determined to be in the first motion state, that is, in the jumping state.

[0137] Furthermore, it may also include:

[0138] In response to determining that the virtual object is in the first motion state, the footstep reverse motion function node of the virtual object in the current frame is turned off; and in response to determining that the virtual object is in the second motion state and the duration exceeds the fifth threshold, the footstep reverse motion function node of the virtual object is turned on.

[0139] That is, as long as it is determined that the virtual object is in the jumping state, the foot reversal function node ik of the virtual object in the current frame will be turned off; and when it is determined that the virtual object has returned to the jumping position, or when the duration of the non-jumping state exceeds the fifth threshold, the foot reversal function node ik of the virtual object will be turned on again.

[0140] The second motion state refers to either the virtual object returning to its starting position or being in a non-jumping state. The fifth threshold can be related to the predicted duration t. x Related, for example: the fifth threshold is 2t x .

[0141] When it is determined that the virtual object is in the second motion state, the footstep reverse motion function node ik of the virtual object is started again, the current tailbone point of the virtual object is taken as the initial position of the first key point of the current frame, and subsequent pose or motion state determination is performed.

[0142] The pose determination method disclosed in the embodiment obtains depth information of a target key point in a first image frame, the first image frame being a two-dimensional image containing a target object; predicts first position information of the target key point in the first image frame based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame in time sequence; and determines pose information of a virtual object based on the depth information and the first position information, the pose information of the virtual object being capable of representing real actions of the target object. The scheme determines the pose information of the virtual object based on the predicted position information of the target key point in the image frame and the obtained depth information of the target key point, increases determination of the depth information, and avoids the problem of large deviation caused by pose determination by using a single camera.

[0143] The embodiment discloses an electronic device, a structural schematic diagram of which is shown in Figure 6 The electronic device comprises:

[0144] a processor 61 and a memory 62.

[0145] The processor 61 is configured to obtain depth information of a target key point in a first image frame, the first image frame being a two-dimensional image containing a target object; predict first position information of the target key point in the first image frame based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame in time sequence; and determine pose information of a virtual object based on a comparison result between the depth information and the first position information, the pose information of the virtual object being capable of representing real actions of the target object.

[0146] The memory 62 is configured to store a program for the processor to perform the above processing process.

[0147] The electronic device disclosed in the embodiment is implemented based on the above-mentioned pose determination method, and details are not repeated here.

[0148] The electronic device disclosed in the embodiment obtains depth information of a target key point in a first image frame, the first image frame being a two-dimensional image containing a target object; predicts first position information of the target key point in the first image frame based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame at least in time sequence; and determines pose information of a virtual object based on the depth information and the first position information, the pose information of the virtual object being capable of representing a real action of the target object. The scheme determines the pose information of the virtual object based on the predicted position information of the target key point in the image frame and the obtained depth information of the target key point, increases determination of the depth information, and avoids a large deviation caused by pose determination by using a single camera.

[0149] The pose determination apparatus disclosed in the embodiment has a structure diagram as shown in Figure 7 The pose determination apparatus comprises:

[0150] The pose determination apparatus comprises an obtaining unit 71, a predicting unit 72, and a pose determination unit 73.

[0151] The obtaining unit 71 is configured to obtain depth information of a target key point in a first image frame, the first image frame being a two-dimensional image containing a target object.

[0152] The predicting unit 72 is configured to predict first position information of the target key point in the first image frame based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame at least in time sequence.

[0153] The pose determination unit 73 is configured to determine pose information of a virtual object based on the depth information and the first position information, the pose information of the virtual object being capable of representing a real action of the target object.

[0154] The pose determination apparatus disclosed in the embodiment is implemented based on the pose determination method disclosed in the above embodiment, and thus will not be described herein.

[0155] The pose determination apparatus disclosed in the embodiment obtains depth information of a target key point in a first image frame, the first image frame being a two-dimensional image containing a target object; predicts first position information of the target key point in the first image frame based on pose information of the target key point in a second image frame, the second image frame being earlier than the first image frame at least in time sequence; and determines pose information of a virtual object based on the depth information and the first position information, the pose information of the virtual object being capable of representing a real action of the target object. The scheme determines the pose information of the virtual object based on the predicted position information of the target key point in the image frame and the obtained depth information of the target key point, increases determination of the depth information, and avoids a large deviation caused by pose determination by using a single camera.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0157] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0158] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0159] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pose determination method, comprising: Obtain depth information of target key points in a first image frame, wherein the first image frame is a two-dimensional image containing the target object; The first position information of the target key point in the first image frame is predicted based on the pose information of the target key point in the second image frame, wherein the second image frame is at least earlier than the first image frame in time. The pose information of the virtual object is determined based on the depth information and the first position information, and the pose information of the virtual object can represent the real action of the target object; The step of determining the pose information of the virtual object based on the depth information and the first position information includes: The third location information of the target key points of the virtual object is obtained based on the first image frame; If the difference between the depth information and the second position information of the target key points in the second image frame is greater than a first threshold, the pose information of the virtual object is determined based on the third position information and the first position information; or, If the difference between the first location information and the third location information is greater than a second threshold, the pose information of the virtual object is determined based on the third location information; or, If the difference between the depth information and the second position information is not greater than a first threshold, and the difference between the first position information and the third position information is not greater than a second threshold, the pose information of the virtual object is determined based on the depth information and the third position information.

2. The method according to claim 1, wherein, Obtaining the depth information of the target key points in the first image frame includes: Based on the set extraction radius, determine the set of depth points to be extracted for each target key point in the first image frame, and obtain the depth information set of the depth points to be extracted. The depth information set is filtered to obtain the depth information of the target key points.

3. The method according to claim 1, wherein, The prediction of the first position information of the target key points in the first image frame based on the pose information of the target key points in the second image frame includes: Obtain the time interval between the first image frame and the second image frame; Based on the second position information and motion information of the target key point in the second image frame, predict the position information of the target key point after the time interval, and determine the predicted position information as the first position information.

4. The method according to claim 1, wherein, Determining the pose information of the virtual object based on the third position information and the first position information includes: Based on the third location information and the first location information, determine the target depth information of the first target key point of the virtual object; Based on the target depth information and the planar coordinate information of the first target key point, the three-dimensional coordinate information of the first target key point is obtained; The pose information of the virtual object is determined based on the positional information between the first target key point and at least one second target key point and the three-dimensional coordinate information.

5. The method according to claim 1, wherein, Determining the pose information of the virtual object based on the depth information and the third position information includes: Based on the third location information and the depth information, the target depth information of the first target key point of the virtual object is determined, so as to obtain the three-dimensional coordinate information of the first target key point; The pose information of the virtual object is determined based on the positional relationship between the first target key point and at least one second target key point and the three-dimensional coordinate information.

6. The method according to claim 1, wherein, Also includes: The motion state of the virtual object is determined based on the position change information of the target key points of the virtual object.

7. The method according to claim 6, wherein, Determining the motion state of the virtual object based on the position change information of the target key points of the virtual object includes: If the position change of the first key point of the virtual object reaches the third threshold and the motion parameter reaches the fourth threshold, the motion trend of at least one second key point of the virtual object is determined. If at least one second key point is in a first motion trend and the motion parameters of the second key point meet the corresponding conditions, the virtual object is determined to be in a first motion state.

8. The method according to claim 6 or 7, wherein, Also includes: In response to determining that the virtual object is in a first motion state, the footstep reverse motion function node of the virtual object in the current frame is turned off; as well as, In response to determining that the virtual object is in a second motion state and the duration exceeds a fifth threshold, the footstep reverse motion function node of the virtual object is activated.

9. A pose determination device, comprising: The obtaining unit is used to obtain depth information of target key points in a first image frame, wherein the first image frame is a two-dimensional image containing the target object; The prediction unit is used to predict the first position information of the target key point in the first image frame based on the pose information of the target key point in the second image frame, wherein the second image frame is at least earlier than the first image frame in time. The pose determination unit is used to determine the pose information of a virtual object based on the depth information and the first position information, wherein the pose information of the virtual object can characterize the real action of the target object; The pose determination unit determines the pose information of the virtual object based on the depth information and the first position information, including: The third location information of the target key points of the virtual object is obtained based on the first image frame; If the difference between the depth information and the second position information of the target key points in the second image frame is greater than a first threshold, the pose information of the virtual object is determined based on the third position information and the first position information; or, If the difference between the first location information and the third location information is greater than a second threshold, the pose information of the virtual object is determined based on the third location information; or, If the difference between the depth information and the second position information is not greater than a first threshold, and the difference between the first position information and the third position information is not greater than a second threshold, the pose information of the virtual object is determined based on the depth information and the third position information.

Images