A video data processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a method, device, electronic device and storage medium for processing video data, which relates to the field of artificial intelligence technology, specifically to the technical fields of augmented reality AR, virtual reality VR, computer vision, augmented reality, deep learning, etc., and can be applied to scenes such as content generation based on artificial intelligence. The method includes: extracting the foot state of the target object and the attribute information of the body joints in the target object from the video data; based on the height of the preset ankle plane, according to the foot state of the target object in the video data and the height of the ankle joint, the body joints in the video data are moved in the height direction, so that the ankle joints in the ground contact state are within the preset ankle plane; constraining the toe joints in the ground contact state to the preset toe plane, and optimizing the angle of the ankle joints in the video data of the ground contact state.

Description

Technical Field

[0001] The present disclosure relates to the computer field, in particular to the field of artificial intelligence technology, specifically to augmented reality AR, virtual reality VR, computer vision, augmented reality, deep learning and other technical fields, and can be applied to scenes such as artificial intelligence-based content generation. Specifically, it relates to a video data processing method, device, electronic device and storage medium. Background Art

[0002] Video-based content generation involves multiple links such as video analysis, feature extraction, and content creation. For example, using motion capture data to drive digital objects can achieve simple, easy-to-use, and low-cost digital object motion generation compared to the traditional method of manually designing animation data by animators.

[0003] Motion capture sets collectors at key parts of moving objects to collect attribute information such as position, angle, etc. The accuracy of motion capture data has an important impact on the generation of digital objects. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device and storage medium for processing video data.

[0005] According to one aspect of the present disclosure, a method for processing video data is provided, comprising:

[0006] Extracting the foot state of the target object and the attribute information of the body joints in the target object from the video data;

[0007] Based on the height of the preset ankle plane, the body joints in the video data are moved in the height direction according to the foot state and the height of the ankle joint of the target object in the video data, so that the ankle joint in the ground contact state is within the preset ankle plane;

[0008] The toe joint in the ground contact state is constrained to a preset toe plane, and the angle of the ankle joint in the video data of the ground contact state is optimized.

[0009] According to one aspect of the present disclosure, there is provided a video data processing device, comprising:

[0010] An information extraction module, used for extracting the foot state of the target object and the attribute information of the body joints in the target object from the video data;

[0011] A joint moving module, for moving the body joints in the video data in the height direction based on the preset ankle plane height and the foot state of the target object in the video data and the height of the ankle joint, so that the ankle joint in the ground contact state is within the preset ankle plane;

[0012] The joint optimization module is used to constrain the toe joint in the ground contact state to a preset toe plane and optimize the angle of the ankle joint in the video data of the ground contact state.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method provided by any embodiment of the present disclosure.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by any embodiment of the present disclosure.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method provided according to any embodiment of the present disclosure is implemented.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 is a flowchart of a method for processing video data provided according to an embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of another method for processing video data provided according to an embodiment of the present disclosure;

[0023] Figure 3 is a flowchart of another method for processing video data provided according to an embodiment of the present disclosure;

[0024] Figure 4 is a structural schematic diagram of a video data processing device provided according to an embodiment of the present disclosure;

[0025] Figure 5 The block diagram is a block diagram of an electronic device for implementing the video data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Figure 1 1 is a flowchart of a method for processing video data according to an embodiment of the present disclosure. The method is applicable to the case of optimizing motion capture data. The method can be executed by a video data processing device, which can be implemented in software and / or hardware and can be integrated into an electronic device. Figure 1 As shown, the video data processing method of this embodiment may include:

[0027] S101, extracting the foot state of the target object and the attribute information of the body joints of the target object from the video data;

[0028] S102, based on the preset ankle plane height, according to the foot state of the target object in the video data and the height of the ankle joint, the body joints in the video data are moved in the height direction so that the ankle joint in the ground contact state is within the preset ankle plane;

[0029] S103, constraining the toe joint in the ground contact state to a preset toe plane, and optimizing the angle of the ankle joint in the video data of the ground contact state.

[0030] The video data may be a motion video collected during the motion of the target object, the foot state may be touching the ground or in the air, the body joints may include ankle joints, toe joints, root joints, etc., and the attribute information of the body joints may be the angle or position of the body joints. The angle and position of the body joints may be derived from each other based on the length of the bones, that is, the position of the body joints may be determined based on the relationship between the angle of the body joints and the length of the bones, and the angle of the body joints may be determined based on the relationship between the position of the body joints and the length of the bones.

[0031] During the movement of the target object, the movement video of the target object is collected as the video data to be processed, and the ground detection network can be used to extract the foot state of the target object from the video frame. Subsequently, the video frame of the foot touching the ground can be used as the ground frame, and the video frame of the foot in the air can be used as the air frame; and the body perception network can be used to extract the attribute information of the body joints of the target object from the video frame. The structure of the ground detection network and the body perception network is not specifically limited.

[0032] The ankle plane and toe plane may also be preset, with the height of the ankle plane set to H1 and the height of the toe plane set to H2, and the difference between H1 and H2 being a preset fixed value, for example, H1-H2=DH1, where DH1 is the height difference between the ankle and toes when the digital object stands on the ground.

[0033] In the disclosed embodiments, the body joints of the target object in the video frame can be adaptively processed in the ankle plane. Exemplarily, based on the height of the ankle plane, it can be judged whether the ankle joint of the target object in the video frame in the ground contact state is within the preset ankle plane according to the foot state of the target object in the video frame and the height of the ankle joint. If not, the body joints are moved in the height direction so that the ankle joints in the ground contact state are moved into the ankle plane, and the attribute information of each body joint that has been moved is obtained. By moving the body joints in the video frame in the height direction based on the preset ankle plane so that the ankle joints in the ground contact state fall within the ankle plane, not only can the problem of the target object fluctuating in height in different video frames be avoided, but also the position interval of the foot joints in the video frame can be narrowed, and the difficulty of subsequent optimization of the ankle joints can be reduced, thereby improving the efficiency and quality of the optimization of the ankle joints.

[0034] In the disclosed embodiment, the ankle joint of the target object in the video frame can also be adaptively processed in the toe plane. A touchdown frame in which the foot is in a touchdown state can be selected from the video data, the toe joint in the touchdown frame can be constrained to a preset toe plane, and the angle of the ankle joint can be optimized as a variable to obtain the optimized angle of the ankle joint. By constraining the toe joint in the touchdown state to the toe plane and optimizing the ankle joint in the touchdown state to obtain the attribute information of the optimized ankle joint, the foot in the touchdown state is on the ground as a whole instead of only the toes falling on the ground, avoiding abnormal problems such as ankle joint deviation, for example, avoiding the foot standing up and walking due to only the toes touching the ground. In the adaptive processing of the toe plane, by only processing the touchdown frame without processing the air frame, and only optimizing the angle of the ankle joint without optimizing the angles of other joints, the calculation amount of the adaptive processing of the toe plane can be reduced, and the optimization efficiency can be further improved.

[0035] In summary, the foot state of the target object in the video frame is obtained by performing foot touchdown detection on each video frame in the video data, and the video frame is divided into a touchdown frame and a flight frame; the attribute information of the body joints is obtained by performing body joint detection on each video frame. In addition, the attribute information of each body joint is obtained by performing adaptive processing of the ankle plane on both the touchdown frame and the flight frame; the angle of the toe joint and the angle of the ankle joint are processed by performing adaptive processing on the touchdown frame. Through the above processing, that is, by introducing the ankle plane and the toe plane, the processing granularity of the attribute information of the body joints is refined, and the data accuracy of the body joints is improved, thereby improving the quality of subsequent digital object generation and reducing the repair work and cost of digital objects.

[0036] The technical solution provided by the disclosed embodiment performs foot contact detection and body joint detection on the target object in the video frame; based on a preset ankle plane, the body joint is moved in the height direction so that the ankle joint in the contact state falls within the ankle plane; and the toe joint in the contact frame is constrained to the preset toe plane, and the angle of the ankle joint in the contact frame is optimized, thereby improving the data quality of the body joint, thereby improving the quality of subsequent digital object generation.

[0037] In an optional implementation, the method further includes: performing motion rendering on the digital object using the adjusted body joint angles of the target object.

[0038] The action of the digital object can be rendered and displayed by using the angle of the adjusted body joints of the target object. When the target object is a human body, a digital human can be generated through 3D modeling, texture mapping, lighting rendering and other technologies; when the target object is another animal, a corresponding digital animal can also be generated. The data quality of the body joints is improved through adaptive processing of the ankle plane and the toe plane, thereby improving the generation quality of the digital object, and enabling the digital object to step on the ground, avoiding abnormalities such as the digital object's feet being inserted into the ground or suspended in the air, that is, achieving the effect of the digital object stepping on the ground based on motion capture drive.

[0039] Figure 2 is a flowchart of another method for processing video data provided according to an embodiment of the present disclosure. Figure 2 The video data processing method of this embodiment may include:

[0040] S201, extracting the foot state of the target object and the attribute information of the body joints of the target object from the video data;

[0041] S202, determining the ankle height in the ground contact state according to the video data and the attribute information of the foot state of the target object and the body joints of the target object;

[0042] S203, according to the height of the ankle in the ground contact state and the height of the preset ankle plane, the body joints in the video data are moved in the height direction so that the ankle joints in the ground contact state fall within the preset ankle plane;

[0043] S204, constraining the toe joint in the ground contact state to a preset toe plane, and optimizing the angle of the ankle joint in the video data of the ground contact state.

[0044] The ankle height in the ground contact state is used to represent the actual ankle height of the target object in the ground contact state in the video frame, and the height of the preset ankle plane is used to represent the expected ankle height of the target object in the ground contact state. The body joints of the target object may include the ankle joints, toe joints, root joints, etc. of the target object. The root joint is located at the waist of the target object, and moving the root joint can cause other joints except the root joint to follow the movement.

[0045] For any video frame, the ankle height in the ground contact state can be determined for the video frame according to the foot state of the target object in the video frame and the attribute information of the body joints in the target object; the ankle height in the ground contact state is compared with the height of the preset ankle plane; and the body joints of the target object in the video frame are moved in the height direction in a height manner according to the comparison result to obtain the adjusted height of the body joints. Among them, the video frames can be divided into ground contact frames and airborne frames according to the foot state of the target object in each video frame. Both the ground contact frame and the airborne frame have the ankle height in the ground contact state, and the ankle height in the ground contact state corresponding to the airborne frame can refer to the actual ankle height corresponding to the foot switching from the airborne action to the ground contact action in the airborne frame. According to the height of the ankle in the ground contact state and the height of the preset ankle plane, the body joints in the video data are moved in the height direction so that the ankle joints in the ground contact state fall within the preset ankle plane, and the ankles of the subsequently generated digital objects fall within the ankle plane when the foot touches the ground; moreover, only moving the body joints in the height direction can also enable the subsequently generated digital objects to maintain the original motion trajectory in the video data.

[0046] In an optional embodiment, the method for determining the ankle height in the ground contact state based on the foot state of the target object and the attribute information of the body joints in the target object for the video data includes: for any video frame, obtaining the foot state in the video frame and the heights of the two ankle joints in the video frame; if the video frame is a ground contact frame, using the height of the lower ankle joint in the ground contact frame as the ankle height in the ground contact state; the foot in the ground contact frame is in the ground contact state; if the video frame is a flight frame, obtaining an adjacent ground contact frame, and using the height of the lower ankle joint in the adjacent ground contact frame as the ankle height in the ground contact state; the foot in the flight frame is in the flight state.

[0047] For any video frame, if the foot state of the target object in the video frame is a ground contact state, the video frame is taken as a ground contact frame; if the foot state of the target object in the video frame is a flying frame, the video frame is taken as a flying frame.

[0048] For any touchdown frame, the heights of the two ankle joints in the touchdown frame are obtained, and the height of the relatively lower ankle joint is used as the ankle height of the touchdown state of the touchdown frame. For any soaring frame, a touchdown frame relatively close to the soaring frame can be obtained, and the height of the ankle joint in the touchdown frame is used as the ankle height of the touchdown state of the soaring frame. For example, the height of the relatively lower ankle joint in the most recent touchdown frame can be used as the ankle height of the touchdown state of the soaring frame. By selecting the relatively lower ankle joint height as the ankle height of the touchdown state, it is also possible to avoid the foot of the subsequently generated digital object from intersecting the ground.

[0049] In an optional embodiment, the body joints in the video data are moved in the height direction according to the ankle height in the ground contact state and the height of a preset ankle plane so that the ankle joints in the ground contact state fall within the preset ankle plane, including: subtracting the height of the preset ankle plane from the ankle height in the ground contact state as the ankle height deviation in the ground contact state; if the ankle height deviation is a positive value, the ankle height deviation is used to lower the body joint in the height direction; if the ankle height deviation is a negative value, the ankle height deviation is used to raise the body joint in the height direction.

[0050] For any video frame, the ankle height deviation in the ground contact state can be determined as follows:

[0051] DH2 = H3-H1;

[0052] DH2 is the ankle height deviation in the ground contact state, H3 is the ankle height in the ground contact state, and H1 is the height of the preset ankle plane. Specifically, the height of the adjusted root joint can be obtained by subtracting the ankle height deviation from the value of the root joint in the height direction; and the height of other joints except the root joint can be adjusted using the height adaptability of the adjusted root joint.

[0053] If the ankle height deviation is a positive value, the height value of the root joint is reduced by subtracting the ankle height deviation from the value in the height direction of the root joint, thereby lowering the body joint of the target object in height. If the ankle height deviation is a negative value, the height value of the root joint is increased by subtracting the ankle height deviation from the value in the height direction of the root joint, thereby raising the body joint of the target object in height. By moving the body joint in the height direction using the ankle height deviation, that is, only changing the height of the body joint, the target object can maintain the original motion trajectory in the video data.

[0054] The technical solution provided by the embodiment of the present disclosure determines the ankle height in the ground contact state for the video frame, and according to the ankle height in the ground contact state and the height of the preset ankle plane, moves the body joints in the video frame in the height direction, so that the ankle joints in the ground contact state fall within the preset ankle plane, so that the ankles of the subsequently generated digital objects fall within the ankle plane when the foot touches the ground; and, only the body joints are moved in the height direction, and the subsequently generated digital objects are kept in the original motion trajectory in the video data.

[0055] Figure 3 is a flowchart of another method for processing video data provided according to an embodiment of the present disclosure. Figure 3 The video data processing method of this embodiment may include:

[0056] S301, extracting the foot state of the target object and the attribute information of the body joints of the target object from the video data;

[0057] S302, based on the preset ankle plane height, according to the foot state of the target object in the video data and the height of the ankle joint, the body joints in the video data are moved in the height direction so that the ankle joint in the ground contact state is within the preset ankle plane;

[0058] S303, constraining the positions of the toe joints in the adjacent touchdown frames to a preset toe plane as a touchdown plane constraint;

[0059] S304, constructing an optimization problem according to the angles of the ankle joints in adjacent contact frames;

[0060] S305 , solving the optimization problem under the constraint of the ground contact plane to obtain an optimized target angle of the ankle joint.

[0061] The video data may be a motion video collected during the movement of the target object, and the foot state may be touching the ground or in the air. The video frame in which the foot is in the touching state is used as the touching frame, and the video frame in which the foot is in the air is used as the air frame. The body joints may include ankle joints, toe joints, root joints, etc., and the attribute information of the body joints may be the angle or position of the body joints. In the process of adapting the ankle plane, the body joints in the video data may be moved in the height direction according to the foot state of the target object in the video data and the height of the ankle joint based on the height of the ankle plane, so that the ankle joint in the touching state is within the preset ankle plane.

[0062] In the process of adapting the toe plane, the angle of the ankle joint can be used as a variable to be optimized and the touchdown frame can be processed. For example, the position of the toe joint in the adjacent touchdown frames can be constrained to a preset toe plane as a touchdown plane constraint; that is, the position of the toe joint in the touchdown frame falls within the preset toe plane and can satisfy the following touchdown plane constraint relationship:

[0063] H_foot-H2=0, if contact=1;

[0064] Among them, H2 is the height of the preset toe plane, H_foot is the height of the plane where the toes are located, that is, the height of the toe joint, and contact represents the state of the foot, which takes a value of 1 when touching the ground and a value of 0 when in the air.

[0065] Among them, the angle of the ankle joint can be used as a variable, and the angle of the ankle joint in adjacent touchdown frames is used to construct an optimization problem, so that the ankle joint in adjacent touchdown frames remains relatively stable, and the angle change of the ankle joint in adjacent touchdown frames is minimized to obtain the optimization problem; under the constraint of the touchdown plane, the optimization problem is solved to obtain the optimized target angle of the ankle joint. By constraining the toe joint in the touchdown frame to the toe plane, the effect of the digital object stepping on the ground is not achieved, but often appears to be interspersed with the ground or suspended in the air, so that the effect of the digital object stepping on the ground is achieved in the subsequently generated digital object, avoiding the problem of the digital object's feet interspersed with the ground or suspended in the air.

[0066] In an optional embodiment, the optimization problem is constructed based on the angle of the ankle joint in adjacent touchdown frames, including: determining the angle deviation of the ankle joint based on the angle of the ankle joint in adjacent touchdown frames; performing a first-order difference on the angle of the ankle joint in adjacent touchdown frames to obtain a velocity deviation of the ankle joint; performing a second-order difference on the angle of the ankle joint in adjacent touchdown frames to obtain an acceleration deviation of the ankle joint; and accumulating the angle deviation of the ankle joint, the velocity deviation of the ankle joint and the acceleration deviation of the ankle joint as an optimization problem.

[0067] Among them, the angle deviation between the ankle joints in adjacent contact frames can be expressed as:

[0068] L1=|q k -q k-1 |;

[0069] Among them, q k ,q k-1 They are respectively the angle of the ankle joint in the k-1th touchdown frame, the angle of the ankle joint in the kth touchdown frame, and L1 is the angle deviation of the ankle joint.

[0070] The velocity deviation between the ankle joints in adjacent contact frames can be expressed as:

[0071]

[0072] in, They are respectively the velocity of the ankle joint in the k-1th touchdown frame, the velocity of the ankle joint in the kth touchdown frame, and L2 is the velocity deviation of the ankle joint.

[0073] Among them, the acceleration deviation between the ankle joints in adjacent contact frames can be expressed as:

[0074]

[0075] in, They are respectively the acceleration of the ankle joint in the k-1th touchdown frame, the acceleration of the ankle joint in the kth touchdown frame, and L3 is the acceleration deviation of the ankle joint.

[0076] In summary, the optimization objective can be: L=L1+L2+L3.

[0077] By using the angle error, velocity deviation and acceleration deviation of the ankle joint in the touchdown frame to construct the optimization target, the angle, velocity and acceleration of the ankle joint in adjacent touchdown frames all change less, that is, the ankle joint remains stable in adjacent touchdown frames, thereby improving the quality of the foot data.

[0078] The technical solution provided by the disclosed embodiment determines the foot state of the target object in the video data and introduces adaptive processing of the ankle plane to raise or lower each joint as a whole, which not only avoids the stability of the person's position in different frames, but also improves the efficiency and accuracy of solving the optimization problem in the subsequent adaptive processing of the toe plane. By introducing the adaptive processing of the toe plane, the position of the toes in adjacent ground contact frames is maintained in the toe plane, and the ankle joint angle is kept stable, thereby achieving the effect of stepping on the ground, thereby optimizing the effect of the digital object's stepping on the ground.

[0079] Figure 4 1 is a schematic diagram of the structure of a video data processing device provided according to an embodiment of the present disclosure. The device is suitable for optimizing motion capture data. The device can be executed by a video data processing device, which can be implemented in software and / or hardware, and can be integrated into an electronic device. Figure 4 As shown, the video data processing device 400 of this embodiment may include:

[0080] An information extraction module 410, for extracting the foot state of the target object and the attribute information of the body joints in the target object from the video data;

[0081] The joint moving module 420 is used to move the body joints in the video data in the height direction according to the foot state and the height of the ankle joint of the target object in the video data based on the preset ankle plane height, so that the ankle joint in the ground contact state is within the preset ankle plane;

[0082] The joint optimization module 430 is used to constrain the toe joint in the ground contact state to a preset toe plane and optimize the angle of the ankle joint in the video data of the ground contact state.

[0083] In an optional implementation, the joint movement module 420 includes:

[0084] An ankle height unit, used for determining the ankle height of the ground contact state according to the foot state of the target object and the attribute information of the body joints in the target object based on the video data;

[0085] The joint moving unit is used to move the body joints in the video data in the height direction according to the ankle height in the ground contact state and the height of the preset ankle plane, so that the ankle joint in the ground contact state falls within the preset ankle plane.

[0086] In an optional embodiment, the ankle height unit comprises:

[0087] An information acquisition subunit, for acquiring, for any video frame, a foot state in the video frame and heights of two ankle joints in the video frame;

[0088] A ground contact frame height subunit, for, if the video frame is a ground contact frame, using the height of the lower ankle joint in the ground contact frame as the ankle height in the ground contact state; the foot in the ground contact frame is in the ground contact state;

[0089] The soaring frame height subunit is used to obtain adjacent touchdown frames if the video frame is a soaring frame, and use the lower ankle joint height in the adjacent touchdown frames as the ankle height in the touchdown state; the foot in the soaring frame is in the soaring state.

[0090] In an optional embodiment, the joint movement unit includes:

[0091] A height deviation subunit, used for subtracting the height of a preset ankle plane from the ankle height in the ground contact state to obtain the ankle height deviation in the ground contact state;

[0092] The joint lowering subunit is used for lowering the body joint in the height direction by using the ankle height deviation if the ankle height deviation is a positive value;

[0093] The joint elevation subunit is used to elevate the body joint in the height direction by using the ankle height deviation if the ankle height deviation is a negative value.

[0094] In an optional implementation, the joint optimization module 430 includes:

[0095] A ground contact constraint unit, used for constraining the position of the toe joint in adjacent ground contact frames to a preset toe plane as a ground contact plane constraint;

[0096] An optimization problem unit, used for constructing an optimization problem according to angles of an ankle joint in adjacent contact frames;

[0097] The optimization solving unit is used to solve the optimization problem under the constraint of the ground contact plane to obtain the optimized target angle of the ankle joint.

[0098] In an optional implementation, the optimization problem unit includes:

[0099] an angle deviation subunit, configured to determine an angle deviation of the ankle joint according to angles of the ankle joint in adjacent contact frames;

[0100] A speed deviation subunit, used for performing a first-order difference on the angles of the ankle joint in adjacent contact frames to obtain a speed deviation of the ankle joint;

[0101] The acceleration deviation subunit is used to perform second-order difference on the angles of the ankle joint in adjacent contact frames to obtain the acceleration deviation of the ankle joint;

[0102] The optimization problem subunit is used for accumulating the angle deviation of the ankle joint, the velocity deviation of the ankle joint and the acceleration deviation of the ankle joint as an optimization problem.

[0103] In an optional implementation manner, the video data processing device 400 further includes:

[0104] The digital object module is used to perform motion rendering on the digital object by using the adjusted body joint angles of the target object.

[0105] The technical solution of the embodiment of the present disclosure, the technical solution provided by the embodiment of the present disclosure, by determining the foot state of the target object in the video data and introducing adaptive processing of the ankle plane, raising or lowering each joint as a whole, not only avoids the stability of the position of the person in different frames, but also improves the efficiency and accuracy of solving the optimization problem in the subsequent adaptive processing of the toe plane; by introducing the adaptive processing of the toe plane, the position of the toes in adjacent ground contact frames is maintained in the toe plane, and the ankle joint angle is kept stable, thereby achieving the effect of stepping on the ground, and thus optimizing the effect of the digital object stepping on the ground.

[0106] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0108] Figure 5 The block diagram is a block diagram of an electronic device for implementing the video data processing method according to an embodiment of the present disclosure.

[0109] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0110] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0111] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0112] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), an image processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a processing method for video data. For example, in some embodiments, the processing method for video data may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the processing method for video data described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the processing method for video data in any other appropriate manner (e.g., by means of firmware).

[0113] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0114] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0115] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), an image processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a processing method for video data. For example, in some embodiments, the processing method for video data may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the processing method for video data described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the processing method for video data in any other appropriate manner (e.g., by means of firmware).

[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0123] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0124] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for processing video data, comprising: Extracting the foot state of the target object and the attribute information of the body joints in the target object from the video data; Based on the height of a preset ankle plane, the body joints in the video data are moved in the height direction according to the foot state of the target object in the video data and the height of the ankle joint, so that the ankle joint in the ground contact state is within the preset ankle plane, including: for the video data, determining the ankle height in the ground contact state according to the foot state of the target object and the attribute information of the body joint in the target object; according to the ankle height in the ground contact state and the height of the preset ankle plane, the body joints in the video data are moved in the height direction, so that the ankle joint in the ground contact state falls within the preset ankle plane; The toe joint in the ground contact state is constrained to a preset toe plane, and the angle of the ankle joint in the video data of the ground contact state is optimized; The preset ankle plane and the preset toe plane are determined based on the height difference between the ankle and the toes when the digital object stands on the ground.

2. The method according to claim 1, wherein: The method of determining the ankle height in the ground contact state according to the foot state of the target object and the attribute information of the body joints of the target object based on the video data includes: For any video frame, obtain the foot state and the heights of the two ankle joints in the video frame; If the video frame is a ground contact frame, the height of the lower ankle joint in the ground contact frame is used as the ankle height in the ground contact state; the foot in the ground contact frame is in the ground contact state; If the video frame is a flying frame, the adjacent touching frame is obtained, and the height of the lower ankle joint in the adjacent touching frame is used as the ankle height in the touching state; the foot in the flying frame is in the flying state.

3. The method according to claim 1, wherein: The method of moving the body joints in the video data in the height direction according to the ankle height in the ground contact state and the height of the preset ankle plane so that the ankle joints in the ground contact state fall within the preset ankle plane includes: Subtract the height of the preset ankle plane from the ankle height in the ground contact state to obtain the ankle height deviation in the ground contact state; If the ankle height deviation is a positive value, the ankle height deviation is used to pull the body joint down in the height direction; If the ankle height deviation is a negative value, the ankle height deviation is used to raise the body joint in the height direction.

4. The method according to claim 1, wherein: The method of constraining the toe joint in the ground contact state to a preset toe plane and optimizing the angle of the ankle joint in the video data of the ground contact state includes: Constrain the position of the toe joint in the adjacent touchdown frames to the preset toe plane as the touchdown plane constraint; The optimization problem is formulated based on the angles of the ankle joint in adjacent touchdown frames; Under the constraint of the contact plane, the optimization problem is solved to obtain the optimized target angle of the ankle joint.

5. The method according to claim 4, wherein: The optimization problem is constructed according to the angles of the ankle joints in adjacent contact frames, including: determining an angle deviation of the ankle joint based on the angle of the ankle joint in adjacent contact frames; Perform the first-order difference of the angles of the ankle joint in adjacent contact frames to obtain the velocity deviation of the ankle joint; Perform the second-order difference of the angles of the ankle joint in adjacent contact frames to obtain the acceleration deviation of the ankle joint; The angle deviation of the ankle joint, the velocity deviation of the ankle joint and the acceleration deviation of the ankle joint are accumulated as an optimization problem.

6. The method according to claim 1, further comprising: The motion of the digital object is rendered using the adjusted body joint angles of the target object.

7. A video data processing device, comprising: An information extraction module, used for extracting the foot state of the target object and the attribute information of the body joints in the target object from the video data; A joint moving module, for moving the body joints in the video data in the height direction based on the preset ankle plane height and the foot state of the target object in the video data and the height of the ankle joint, so that the ankle joint in the ground contact state is within the preset ankle plane; A joint optimization module, used for constraining the toe joint in the ground contact state to a preset toe plane, and optimizing the angle of the ankle joint in the video data of the ground contact state; The joint movement module comprises: An ankle height unit, used for determining the ankle height of the ground contact state according to the foot state of the target object and the attribute information of the body joints in the target object based on the video data; A joint moving unit, used for moving the body joints in the video data in the height direction according to the ankle height in the ground contact state and the height of the preset ankle plane, so that the ankle joint in the ground contact state falls within the preset ankle plane; The preset ankle plane and the preset toe plane are determined based on the height difference between the ankle and the toes when the digital object stands on the ground.

8. The device according to claim 7, wherein: The ankle height unit comprises: An information acquisition subunit, for acquiring, for any video frame, a foot state in the video frame and heights of two ankle joints in the video frame; A ground contact frame height subunit, for, if the video frame is a ground contact frame, using the height of the lower ankle joint in the ground contact frame as the ankle height in the ground contact state; the foot in the ground contact frame is in the ground contact state; The soaring frame height subunit is used to obtain adjacent touchdown frames if the video frame is a soaring frame, and use the lower ankle joint height in the adjacent touchdown frames as the ankle height in the touchdown state; the foot in the soaring frame is in the soaring state.

9. The device according to claim 7, wherein: The joint moving unit comprises: A height deviation subunit, used for subtracting the height of a preset ankle plane from the ankle height in the ground contact state to obtain the ankle height deviation in the ground contact state; The joint lowering subunit is used for lowering the body joint in the height direction by using the ankle height deviation if the ankle height deviation is a positive value; The joint elevation subunit is used to elevate the body joint in the height direction by using the ankle height deviation if the ankle height deviation is a negative value.

10. The device according to claim 7, wherein: The joint optimization module includes: A ground contact constraint unit, used for constraining the position of the toe joint in adjacent ground contact frames to a preset toe plane as a ground contact plane constraint; An optimization problem unit, used for constructing an optimization problem according to angles of an ankle joint in adjacent contact frames; The optimization solving unit is used to solve the optimization problem under the constraint of the ground contact plane to obtain the optimized target angle of the ankle joint.

11. The device according to claim 10, wherein: The optimization problem unit includes: an angle deviation subunit, configured to determine an angle deviation of the ankle joint according to angles of the ankle joint in adjacent contact frames; A speed deviation subunit, used for performing a first-order difference on the angles of the ankle joint in adjacent contact frames to obtain a speed deviation of the ankle joint; The acceleration deviation subunit is used to perform second-order difference on the angles of the ankle joint in adjacent contact frames to obtain the acceleration deviation of the ankle joint; The optimization problem subunit is used for accumulating the angle deviation of the ankle joint, the velocity deviation of the ankle joint and the acceleration deviation of the ankle joint as an optimization problem.

12. The apparatus according to claim 7, further comprising: The digital object module is used to perform motion rendering on the digital object by using the adjusted body joint angles of the target object.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images