Motion capture method and related device

Abstract

The invention discloses a motion capture method and a related device, and relates to the technical field of virtual reality, and the method comprises the steps: for each coordinate component, solving a coefficient of each monomial expression in a preset interpolation function based on a first position sequence of a key joint point and a preset constraint condition, and obtaining a target interpolation function; and obtaining a second position sequence of the key joint points based on the target interpolation function of each coordinate component. And acquiring motion data of the target motion video according to the second position sequence of each key joint point. The interpolation function is a polynomial taking a frame sequence as an independent variable and a coordinate value as a dependent variable, and the coefficient of each monomial in the interpolation function is a coefficient function taking the frame sequence as an independent variable. The coefficient of each monomial is related to the order, so that the accuracy of the second position sequence of each key joint point obtained by sampling based on the target interpolation function is high, and the motion capture continuity and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a motion capture method and related devices. Background Art

[0002] In current technical practice, skeletal motion capture can be performed on filmed character action videos to generate motion data for controlling the virtual character to produce corresponding actions. However, due to the limitations of video shooting quality and complex environments, the motion capture effect of traditional methods of motion capture through video is relatively low. Summary of the invention

[0003] In view of the above problems, the present application provides a motion capture method and related devices to achieve the purpose of improving the accuracy and continuity of motion capture. The specific scheme is as follows:

[0004] The first aspect of the present application provides a motion capture method, comprising:

[0005] Extracting key frames from the target action video to obtain a key frame sequence of the target action video, wherein the key frame sequence includes a plurality of key frames arranged in frame order;

[0006] Respectively acquiring position information of key joint points preset in each of the key frames to obtain a first position sequence of each of the key joint points, wherein the first position sequence is obtained by arranging the position information of the key joint points in the key frames in a frame order, and the position information includes coordinate values ​​of each coordinate component;

[0007] For each of the coordinate components, the coefficients of each monomial in the preset interpolation function are solved based on the first position sequence of the key joint points and the preset constraint conditions to obtain a target interpolation function; the interpolation function is a polynomial with a frame sequence as an independent variable and a coordinate value as a dependent variable, wherein the coefficients of each monomial in the interpolation function are coefficient functions with a frame sequence as an independent variable;

[0008] For each of the key joint points, obtaining a second position sequence of the key joint points based on the target interpolation function of each of the coordinate components, wherein the second position sequence is obtained by arranging position information of the key joint points in the video frame in frame order;

[0009] According to the second position sequence of each of the key joints, the action data of the target action video is obtained, and the action data includes the corresponding relationship between the identification, frame sequence, and position information of each of the key joints.

[0010] In a possible implementation, extracting key frames from a target action video includes:

[0011] Parsing the target action video to obtain a video frame sequence of the target action video, where the video frame sequence includes all video frames arranged in frame order;

[0012] For each video frame in the video frame sequence, obtaining an inter-frame difference between the video frame and a previous video frame;

[0013] Obtaining an average inter-frame difference of each video frame sequence, and calculating an adaptive threshold of the video frame based on the average inter-frame difference;

[0014] For each video frame in the video frame sequence, if the inter-frame difference of the video frame is greater than the adaptive threshold, the video frame is taken as a key frame and extracted; if the inter-frame difference of the video frame is not greater than the adaptive threshold, the video frame is taken as a non-key frame.

[0015] In a possible implementation, for each of the coordinate components, solving coefficients of each monomial in a preset interpolation function based on the first position sequence of the key joint points and preset constraints to obtain a target interpolation function includes:

[0016] Obtaining a system of linear equations constructed according to the constraint conditions, wherein the constraint conditions include an interpolation condition, a continuity condition, a first-order derivative continuity condition, and a second-order derivative continuity condition;

[0017] For each of the key frames, the frame sequence of the key frame corresponding to the position information in the first position sequence is used as the value of the independent variable, the coordinate value of the coordinate component is used as the function value of the target interpolation function, and is substituted into the linear equation group to solve the coefficients of each monomial in the interpolation function to obtain the target coefficient of each monomial;

[0018] The target interpolation function is obtained, where the target interpolation function is a polynomial with a frame sequence as an independent variable and a coordinate value as a dependent variable, wherein the coefficients of each monomial in the target interpolation function are target coefficients of the monomial.

[0019] In a possible implementation, for each of the key joint points, acquiring a second position sequence of the key joint point based on the target interpolation function of each of the coordinate components includes:

[0020] For each of the coordinate components, substituting the missing frame sequence into the target interpolation function to obtain a function value of the target interpolation function as a coordinate value of the coordinate component of the key joint point in the video frame corresponding to the missing frame sequence, wherein the missing frame sequence includes a frame sequence of missing key frames of a preset number of sampling frames, and the missing key frames include key frames without position information;

[0021] Combining the coordinate values ​​of all the coordinate components, obtaining the position information of the key joint point in the video frame corresponding to the missing frame sequence;

[0022] The position information of the key joint points in the video frames corresponding to the target frame sequence is sorted in ascending order according to the frame sequence to obtain a second position sequence of the key joint points, and the target frame sequence includes the frame sequence of all key frames.

[0023] In a possible implementation, obtaining the action data of the target action video according to the second position sequence of each of the key joint points includes:

[0024] According to the second position sequence of each of the key joint points, angle information of each of the key joint points in each sampled video frame is obtained, where the angle information is used to indicate the angle movement of the key joint point in the sampled video frame, and the sampled video frame includes a video frame corresponding to the target frame sequence;

[0025] Acquire the motion data of the target motion video, wherein the motion data also includes the corresponding relationship between the identification of each key joint point, the frame sequence and the angle information;

[0026] Generate a prefabricated skeleton file, and input the prefabricated skeleton file into a preset skeleton driving model, so that the skeleton driving model drives the key joints of the motion capture object based on the motion driving data of the key joints in the prefabricated skeleton file; wherein the motion driving data of the first type of key joints includes angle information and position information sorted in frame order, and the motion driving data of the second type of key joints includes angle information sorted in frame order.

[0027] In a possible implementation, obtaining angle information of each of the key joint points in each video frame according to the second position sequence of each of the key joint points includes:

[0028] Based on the position information of the key joint point in the video frame and the preceding video frame of the video frame, obtaining the displacement vector of the key joint point in the video frame;

[0029] Obtaining the displacement vector of the key joint point in the video frame and the angles of each coordinate axis;

[0030] The displacement vectors of the key joint points in the video frame and the angles of each coordinate axis are summarized to obtain the angle information of the key joint points in the video frame.

[0031] A second aspect of the present application provides a motion capture device, comprising:

[0032] A frame sampling unit, used to extract key frames in the target action video to obtain a key frame sequence of the target action video, wherein the key frame sequence includes a plurality of key frames arranged in frame order;

[0033] A joint point positioning unit, used to respectively obtain the position information of the key joint points preset in each of the key frames, and obtain a first position sequence of each of the key joint points, wherein the first position sequence is obtained by arranging the position information of the key joint points in the key frames in a frame order, and the position information includes the coordinate value of each coordinate component;

[0034] An interpolation function determination unit is used to solve the coefficients of each monomial in a preset interpolation function for each of the coordinate components based on the first position sequence of the key joint points and the preset constraint conditions to obtain a target interpolation function; the interpolation function is a polynomial with a frame sequence as an independent variable and a coordinate value as a dependent variable, wherein the coefficients of each monomial in the interpolation function are coefficient functions with a frame sequence as an independent variable;

[0035] an interpolation unit, configured to obtain, for each of the key joint points, a second position sequence of the key joint points based on the target interpolation function of each of the coordinate components, wherein the second position sequence is obtained by arranging position information of the key joint points in the video frame in frame order;

[0036] The motion capture unit is used to obtain the motion data of the target motion video according to the second position sequence of each of the key joints, and the motion data includes the corresponding relationship between the identification, frame sequence, and position information of each of the key joints.

[0037] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the motion capture method of the first aspect or any implementation of the first aspect.

[0038] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to execute the computer program so that the electronic device can implement the motion capture method of the first aspect or any implementation manner of the first aspect.

[0041] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the motion capture method of the first aspect or any implementation of the first aspect.

[0042] By means of the above technical scheme, the present application provides a motion capture method and related devices, which extract key frames in a target action video and obtain a key frame sequence of the target action video. The position information of the preset key joint points in each key frame is obtained respectively to obtain the first position sequence of each key joint point. For each coordinate component, the coefficients of each monomial in the preset interpolation function are solved based on the first position sequence of the key joint point and the preset constraint conditions to obtain the target interpolation function. For each key joint point, the second position sequence of the key joint point is obtained based on the target interpolation function of each coordinate component. According to the second position sequence of each key joint point, the motion data of the target action video is obtained. Among them, the motion data includes the identification, frame sequence, and the corresponding relationship of the position information of each key joint point. Since the interpolation function is a polynomial with the frame sequence as the independent variable and the coordinate value as the dependent variable, the coefficients of each monomial in the interpolation function are coefficient functions with the frame sequence as the independent variable. Therefore, based on the first position sequence of key joints, the coefficients of each monomial can be solved, and the coefficients are related to the sequence. Therefore, the second position sequence of each key joint point obtained by sampling based on the target interpolation function has high accuracy, avoiding discontinuous or inaccurate motion capture of the target action video caused by missing position information, that is, improving the continuity and accuracy of motion capture. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A schematic diagram of a system architecture provided for this application;

[0045] Figure 2 A schematic diagram of the hardware structure of an optional terminal 100 provided in this application;

[0046] Figure 3 A schematic diagram of an optional hardware structure of a server 200 provided for this application;

[0047] Figure 4 A flowchart of a specific implementation method of a motion capture method provided in an embodiment of the present application;

[0048] Figure 5 A flowchart of an optional key frame extraction method provided in an embodiment of the present application;

[0049] Figure 6 A specific method for obtaining angle information provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of a flow chart of a motion capture method provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of a motion capture device provided in an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0054] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0055] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0056] In the field of virtual reality technology, the traditional motion capture method is to capture the player's motion data in real time through motion capture devices (such as wearable virtual reality devices). However, the cost of motion capture devices is high and the application scenarios are limited, and the efficiency of motion capture is low. Therefore, motion capture through video is currently used to improve the efficiency of motion capture and reduce the cost of motion capture. Specifically, the action video of the object (usually a person) is captured, and the object's action in the video is identified to achieve object motion capture. However, due to the limitations of video shooting quality and complex environment, the motion capture effect of the traditional method of motion capture through video needs to be improved.

[0057] It should be noted that research has found that the main reasons for poor motion capture effects include:

[0058] 1. The image depth information of the video material is incomplete, resulting in low accuracy in the coordinate positioning of the point during the process of identifying the point through the video.

[0059] 2. In complex or low-quality video environments, such as occlusion, lighting changes, motion blur, etc., the accuracy of key point detection is low. For example, when the character's movements are occluded, the inability to identify and predict the occluded limb movements will cause dislocation and bone jumping in the motion capture results, seriously affecting the capture effect.

[0060] 3. Due to differences in video frame rates and timestamp misalignment, there are misalignment and missed detection problems between the video motion capture results and the original video motion.

[0061] In order to solve the above problems, the present application provides a motion capture method. Specifically, the motion capture method provided by the present application is applied to a scene where the motion of an object is captured by shooting a motion video of the object and motion capture result data is generated, aiming to improve the capture effect of motion capture.

[0062] This application can be applied to, but is not limited to, applications with video motion capture functions or cloud services provided by cloud-side servers, etc., which are introduced below:

[0063] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0064] Among them, a motion capture application can be installed on the terminal 100. The above application and web page can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the motion capture interface and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters and return the processing results to the terminal 100.

[0065] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0066] Next describe Figure 1 The product form of the mid-terminal 100;

[0067] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0068] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0069] refer to Figure 2 As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0070] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0071] Among them, the input device 132 can receive input data and the like.

[0072] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display and / or playback of any multimedia file. In an embodiment of the present application, the display unit 140 may be used to display an interface of motion capture, processing results, etc.

[0073] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0074] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0075] Among them, the memory 120 can be used to store software codes related to the motion capture method, the processor 170 can execute the steps of the motion capture method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to achieve corresponding functions.

[0076] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0077] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0078] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0079] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0080] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0081] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0082] Next describe Figure 1 The product form of the server 200;

[0083] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0084] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0086] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0087] The memory 204 may be used to store software codes related to the motion capture method, and the processor 202 may execute the steps of the motion capture method of the chip, and may also schedule other units to implement corresponding functions.

[0088] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0089] A specific implementation of the motion capture method in an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0090] Reference Figure 4 , Figure 4 A flowchart of a specific implementation method of a motion capture method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, a data processing method provided in an embodiment of the present application may include steps S401 to S408, and these steps are described in detail below.

[0091] S401: Acquire a target action video with depth information.

[0092] In this embodiment, an action video captured by a target shooting device that meets the video shooting standards can be obtained as a target action video, wherein the target shooting device includes but is not limited to a 3D camera, a depth camera, a mobile phone equipped with a dual-camera or multi-camera system, a mobile phone with an integrated depth sensor (TOF sensor or structured light sensor), and other shooting devices that can obtain depth information.

[0093] In this embodiment, the video shooting standard includes: the exposure setting parameters are a preset aperture, a preset shutter speed, and a preset IOS sensitivity; the frame number setting parameters are a preset frame number, wherein the various parameters in the exposure setting parameters are parameter values ​​obtained through debugging so that the light ratio of the shooting picture is reasonable, overexposure does not occur, and the outline of the object is clear and distinct; the frame number parameter is 25 frames or 30 frames, so that the normal state of the object's movement can be captured.

[0094] It should be noted that, first of all, in the process of shooting the video, it is necessary to obtain the depth information of the video material as much as possible, because in the case of conventional video materials, the plane position information of the video can usually be obtained through machine algorithms, that is, the two-dimensional plane data of each feature point, but motion capture requires sampling with three-dimensional space data, and we need to obtain its depth information to obtain complete three-dimensional coordinate data. Therefore, this step establishes a video shooting standard, uses a camera device that can obtain image depth information to shoot action videos, and sets specific parameters of the camera device in order to provide high-quality video acquisition samples.

[0095] S402: extract key frames in the target action video to obtain a key frame sequence of the target action video.

[0096] In this embodiment, the key frame sequence includes multiple key frames arranged in frame order. In this embodiment, the frame order of the key frame is also the frame number, which is used to indicate the order of the key frame in the target action video, and the order of the key frame in the key frame sequence is recorded as the key order.

[0097] In this embodiment, there may be multiple specific methods for extracting key frames in the target action video.

[0098] In a possible implementation, an adaptive threshold may be determined based on the inter-frame difference of each video frame in the target action video, and key frames in the target action video may be extracted based on the adaptive threshold.

[0099] Figure 5 An optional key frame extraction method provided in the embodiment of the present application is as follows: Figure 5 As shown, the key frame extraction method specifically includes:

[0100] S501: Obtain a video frame sequence of a target action video.

[0101] In this embodiment, the video image sequence of the target action video includes continuous video frames arranged in frame order, that is, the frame number of the t-th video frame in the video frame sequence is t.

[0102] S502: For each video frame in the video frame sequence, obtain an inter-frame difference between the video frame and a previous video frame.

[0103] In this embodiment, the preceding video frame of the video frame refers to the previous frame of the video frame. The inter-frame difference between the video frame and the preceding video frame is the sum of the pixel differences between the video frame and the preceding video frame at each pixel point.

[0104] Take the target video frame t as an example, the previous video frame of the t frame is the t-1 frame, and the calculation formula of the inter-frame difference D(t-1,t) between the t frame and the t-1 frame is as shown in formula (1):

[0105] D(t-1, t) = (1);

[0106] Where W and H represent the resolution width and height of the video frame. and They represent the pixel feature values ​​of the t-th frame and the t-1-th frame at the pixel point (x, y) respectively. Optionally, the pixel feature value is the pixel value, that is, D(t-1, t) is the sum of the pixel differences of all pixels between the t-1-th frame and the t-th frame.

[0107] S503: Calculate the average inter-frame difference of the video frame sequence.

[0108] In this embodiment, the average inter-frame difference of the video frame sequence is the average value of the inter-frame difference between each video frame and the previous video frame.

[0109] Taking a video frame sequence including T video frames as an example, the average inter-frame difference of the video frame sequence is The calculation formula is shown in Formula 2:

[0110] = (2)

[0111] S504: Calculate an adaptive threshold based on the average inter-frame difference.

[0112] In this embodiment, the adaptive threshold is positively correlated with the average inter-frame difference.

[0113] In a possible implementation, the formula for calculating the adaptive threshold based on the mean is as follows:

[0114] =a + b(3);

[0115] in, is the adaptive threshold of the video frame, and a and b are both pre-configured non-negative constants.

[0116] Optionally, a is a multiplier factor for indicating the degree of influence of the average inter-frame difference on the adaptive threshold of the video frame, that is, the larger a is, the higher the degree of influence of the average inter-frame difference on the adaptive threshold of the video frame is. b is an offset for fine-tuning the adaptive threshold of the video frame based on the average inter-frame difference, and optionally, the size of b is inversely correlated with the rate of change of the inter-frame difference, that is, the larger the rate of change of the inter-frame difference is, the smaller the adaptive threshold is, or the size of b is inversely correlated with the difference between the current video frame and the average inter-frame difference, that is, the larger the difference between the current video frame and the average inter-frame difference is, the smaller the adaptive threshold is.

[0117] Specifically, the initial values ​​of a and b are set based on experience or preliminary experiments, and the optimized values ​​of a and b are adjusted by iterative optimization. Optionally, a and b are optimized through training or testing, and after the optimization algorithm obtains a and b, the effects of the adaptive thresholds achieved by different combinations of a and b parameters are evaluated (such as the number and quality of key frames or the accuracy of the representation of video content) to select the best parameters.

[0118] In another possible implementation, the adaptive threshold is equal to the average inter-frame difference.

[0119] S505: If the inter-frame difference between the video frame and the previous video frame is greater than the adaptive threshold, extract the video frame as a key frame.

[0120] In this embodiment, the video frame is taken as the tth frame of the target video frame, and the inter-frame difference between the video frame and the previous video frame is compared with the adaptive threshold. If the inter-frame difference is greater than the adaptive threshold, the binarization parameter B(t) of the video frame is set to 1. If the inter-frame difference is not greater than the adaptive threshold, the binarization parameter B(t) of the video frame is set to 0. When B(t) is 1, the video frame t is extracted as a key frame and the frame number t is recorded and added to the key frame sequence. As shown in Formula 4:

[0121] B(t)= (4);

[0122] S506: Arrange the key frames in ascending order to obtain a key frame sequence.

[0123] In this embodiment, the key frame sequence includes key frames sorted in frame order, wherein the frame order of the key frame is the order of the key frame in the entire target action video (that is, the video frame sequence), and the key order of the key frame is the order of the key frame in the key frame sequence. For example, the key frame sequence G of the target action video is , where N is the number of key frames, Indicates the key frame with the order i in the key frame sequence, that is, the i-th key frame.

[0124] Depend on Figure 5It can be seen from the scheme shown that the motion capture method provided by the embodiment of the present application dynamically calculates a suitable threshold value when extracting key frames in the target action video according to the characteristics of the currently processed data or frames. This threshold value can change with the change of the video content, so as to more accurately reflect the key information in the video. Because this threshold value is dynamic, it is necessary to analyze the image of each frame, read frame by frame to calculate the difference between two adjacent frames, and dynamically adjust the threshold value according to the size of the difference value. For example, when the difference between frames suddenly increases, it may mean that the scene has changed significantly. At this time, the threshold value can be appropriately lowered to capture more key frames; conversely, when the difference between frames is small, the threshold value can be appropriately increased to reduce the extraction of redundant frames.

[0125] In summary, video frame sampling is performed according to the motion rules of the action in the target action video, that is, key frames are extracted to retain the complete motion effect as much as possible, so that the position information of the key joint points collected based on the key frames can better express the motion effect of the character's action in the video.

[0126] S403: Acquire position information of key joint points preset in each key frame to obtain a first position sequence of each key joint point.

[0127] In this embodiment, key joint points include but are not limited to joint points corresponding to the head, neck, shoulders, elbows, wrists, hips, knees and ankles.

[0128] In this embodiment, the first position sequence of the key joint points includes position information arranged in a frame sequence, wherein the position information of the key joint points includes the three-dimensional coordinates of the key joint points. Taking the elbow joint points as an example, the three-dimensional coordinates of the elbow joint points in each key frame are obtained, and the three-dimensional coordinates of the elbow joint points are arranged according to the frame sequence of the key frames to obtain the position sequence of the elbow joint points.

[0129] It should be noted that, when identifying the position information of key joint points in key frames based on depth information, there is a situation where the position information of key joint points in key frames is missing.

[0130] In the absence of missing information, that is, the position information of the key joint point O is identified in each key frame, the first position sequence W includes the position information of the key joint point O in the key frame arranged in order, that is, W={ },in, Represents the i-th position information in the first position sequence, that is, the key joint point O in the i-th key frame The location information in.

[0131] In the case of missing, that is, there is at least one missing key frame in which the position information of the key joint point O cannot be identified, the first position sequence W includes the position information of the key joint point O in the key frame arranged in order, and the position information corresponding to the missing key frame is empty.

[0132] S404 . For each coordinate component, based on the first position sequence of the key joint points and the preset constraint conditions, solve the coefficients of each monomial in the preset interpolation function to obtain a target interpolation function.

[0133] In this embodiment, the position information includes the coordinate values ​​of the three-dimensional coordinate components as an example, and the three-dimensional coordinate components include the X coordinate, the Y coordinate, and the Z coordinate. is the coordinate value of the X coordinate in the i-th position information, is the coordinate value of the Y coordinate in the i-th position information, is the coordinate value of the Z coordinate in the i-th position information.

[0134] In this embodiment, the interpolation function is a polynomial with the frame sequence t as the independent variable and the coordinate value as the dependent variable, wherein the coefficient of each monomial in the interpolation function is a coefficient function with the frame sequence t as the independent variable.

[0135] Optionally, the polynomial may be a cubic polynomial. Specifically, the interpolation function is: Taking the first coordinate component X of the key joint point O as an example, the interpolation function (t) is a cubic polynomial with the preset parameter t as the independent variable, as shown in formula (3):

[0136] (t)= (t) + (t) + (t)t+ (t) , ≤t≤ ; (3)

[0137] in, is the frame sequence corresponding to key sequence i. (t) (t) (t), and (t) are all functions of the frame sequence t corresponding to the first coordinate component X.

[0138] In this embodiment, the constraint conditions include an interpolation condition, a continuity condition, a first-order derivative continuity condition, and a second-order derivative continuity condition.

[0139] The interpolation condition indicates that when the frame sequence is equal to the key sequence, the function value of the interpolation function is equal to the coordinate value of the X coordinate in the position information on the key sequence. Optionally, the interpolation condition is as shown in formula (4):

[0140] ( )= (4);

[0141] The continuity condition means that the polynomial is equal to the given function value at the endpoints of the interval, so that the two points (X, X+1) can be connected and the continuity condition is satisfied. Among them, X and X+1 in the polynomial are used to describe the relationship between the x coordinates of two adjacent points.

[0142] Optionally, the continuity condition is as shown in formula (5):

[0143] ( +1)= ( +1) (5);

[0144] The continuity condition of the first-order derivative is: represents the slope of the curve, ensuring that the curve is at t= +1 for the smooth connection.

[0145] Optionally, the first-order derivative continuity is as shown in formula (6):

[0146] ( +1)= ( +1) (6);

[0147] The continuity condition of the second-order derivative is: represents the rate of change of the slope of the curve, that is, the curvature of the curve, ensuring that the curve is at t= The +1 position is not only smooth but also has continuous curvature.

[0148] Optionally, the second-order derivative continuity is as shown in formula (7):

[0149] ( +1)= ( +1) (7).

[0150] Based on formula (4) to formula (5), a system of linear equations is obtained.

[0151] Substitute the key frame sequence and coordinate values ​​corresponding to the X coordinate value of the key joint point O into the linear equation system and solve it to get (t) (t).

[0152] S405 . For each key joint point, obtain a second position sequence of the key joint point based on the target interpolation function.

[0153] In this embodiment, the function value of the target interpolation function corresponding to the missing frame sequence is calculated to obtain the position information of the key joint points in the video frame corresponding to the missing frame sequence, and the missing frame sequence is substituted into the interpolation function to calculate the function value, and the function value is used as the coordinate value of the corresponding coordinate component. Thus, the position information corresponding to the video frame of the missing frame sequence is obtained. Among them, the missing frame sequence is the frame sequence of the missing key frame, the missing key frame is the key frame whose position information is not identified, and the position information corresponding to the video frame of the missing frame sequence includes the three-dimensional coordinates of the key joint points in the video frame (non-key frame) of the missing frame sequence. The position information corresponding to the video frame of the key frame sequence includes the three-dimensional coordinates of the key joint points in the key frame.

[0154] It should be noted that this step uses the position information of each key joint point identified from the key video frame to determine the interpolation function, thereby the motion trajectory of the key joint points, and uses the motion trajectory to interpolate the point data that cannot be sampled due to occlusion, so as to ensure the integrity and continuity of the data.

[0155] In this embodiment, the position information corresponding to all target frame sequences are arranged from small to large according to the frame sequence to obtain a second position sequence, and the target frame sequence includes the frame sequence of all key frames.

[0156] S406 , obtaining angle information of each key joint point in each sampled video frame according to the second position sequence of each key joint point.

[0157] In this embodiment, the sampled video frame is a video frame corresponding to the target frame sequence, and the angle information of the key joint point in the target video frame is used to indicate the angle movement of the key joint point in the target video frame, and the angle information of the key joint point includes the rotation angle of the key joint point around each coordinate axis. Taking the tth frame as an example, the angle information of the key joint point in the tth frame is used to indicate the angle movement of the key joint point from the t-1th frame to the tth frame.

[0158] Take obtaining the angle information of the target video frame as an example. Figure 6 A specific method for obtaining angle information provided in an embodiment of the present application is as follows: Figure 6 As shown, the method includes:

[0159] S601 . Obtain a displacement vector of a key joint point in a target video frame based on the three-dimensional coordinates of the key joint point in a target video frame and a preceding video frame of the target video frame.

[0160] In this embodiment, taking the target video frame as the t-th frame as an example, the displacement vector of the target video frame is equal to the three-dimensional coordinate P ( , , ) and the three-dimensional coordinates P of the key joint points in the t-1th frame ( , , ) coordinate difference. Specifically, the displacement vector of the tth frame =( , , ).

[0161] S602: Obtain the displacement vector of the key joint point in the target video frame and the angle of each coordinate axis.

[0162] In this embodiment, the target coordinate axis is an arbitrary coordinate axis, and the method for obtaining the angle between the displacement vector of the key joint point in the target video frame and the target coordinate axis includes:

[0163] Obtain the modulus of the displacement vector, obtain the dot product of the displacement vector and the unit vector of the target coordinate axis, calculate the cosine of the angle based on the dot product and the modulus of the displacement vector, perform an inverse cosine function on the cosine of the angle, and obtain the angle between the displacement vector of the target video frame and the target coordinate axis.

[0164] Specifically, the displacement vector of the key joint point O in the tth frame is For example, the process of obtaining the displacement vector of the key joint point in the tth frame and the angles of each coordinate axis includes:

[0165] 1. Calculate the modulus of the displacement vector: ∣=

[0166] 2. Calculate the angle between the displacement vector and the X-axis, which is the first angle:

[0167] The unit vector of the X axis is =(1,0,0), the dot product of the displacement vector and the unit vector of the target coordinate axis is: =( )×1+( )×0+( )×0= .

[0168] Calculate the cosine of the first angle: cosα = =

[0169] Calculate the first angle: α=arccos( )×

[0170] 3. Calculate the angle between the displacement vector and the Y axis, which is the second angle:

[0171] The unit vector of the Y axis is =(0,1,0), the dot product of the displacement vector and the unit vector of the target coordinate axis is: =( )× 0+( )×1+( )×0= .

[0172] Calculate the cosine of the second angle: cosβ = =

[0173] Calculate the second angle: β = arccos( )×

[0174] 4. Calculate the angle between the displacement vector and the Z axis, which is the third angle:

[0175] The unit vector of the Z axis is , the dot product of the displacement vector and the unit vector of the target coordinate axis is: =( )×0+( )×0+( )×1= .

[0176] Calculate the cosine of the third angle: cosγ = cosγ= =

[0177] Calculate the third angle: γ=arccos( )×

[0178] S603 , summarizing the displacement vectors of the key joint points in the target video frame and the angles of each coordinate axis to obtain angle information of the key joint points in the target video frame.

[0179] In this embodiment, the angle information of the key joint point in the target video frame includes the angles of the displacement vector of the key joint point in the target video frame and each coordinate axis. For example, the angle information of the key joint point in the tth frame of the target video frame is .

[0180] S407: Obtain action data of the target action video.

[0181] In this embodiment, the motion data of the target motion video includes the corresponding relationship between the identification, frame sequence, and position information of each key joint point, and also includes the corresponding relationship between the identification, frame sequence, and angle information of each key joint point.

[0182] S408, generate a prefabricated skeleton file, and input the prefabricated skeleton file into the skeleton driving model, so that the skeleton driving model drives the key joint points of the motion capture object based on the action driving data of the key joint points in the prefabricated skeleton file, so that the motion capture object produces joint movements that match the target action video.

[0183] In this embodiment, the first type of key joint points are preset root nodes, and the motion driving data of the first type of key joint points in the prefabricated skeleton file includes angle information and position information sorted in frame order. The second type of key joint points are key joint points other than the root nodes, and the motion driving data of the second type of key joint points in the prefabricated skeleton file includes angle information sorted in frame order.

[0184] It should be noted that in order to obtain usable bone data, the obtained point data needs to be recorded on a prefabricated bone model. Usually, the movement of bones is mainly based on angle changes and position changes. Except for the root bone, which requires position information for displacement, other bone movements are angular movements. Therefore, we need to convert the obtained position information into angle information to drive the preset bone model to produce movement.

[0185] It can be seen from the above technical solutions that the motion capture method provided by the embodiment of the present application at least achieves the following technical effects:

[0186] 1. Use a camera with depth information to shoot the character's movements so that the sampling can obtain the three-dimensional coordinate information of the point. At the same time, ensure that the exposure setting is reasonable during the shooting process so that the character's movement can be clearly seen. The video frame rate is increased as much as possible to ensure the clarity and completeness of the captured video and reduce sampling errors.

[0187] 2. The shot video is composed of countless frames, and the complete frames need to be extracted. The extraction method is to extract the frames of the character's actions through an adaptive threshold algorithm. The number of frame extractions will be increased for those parts with large action frequency changes, and the number of frame extractions will be reduced for those parts with small change frequency, to ensure that the action image is extracted completely and accurately.

[0188] 3. Each joint point will have three to five tracking points collected, and the sampled points will be interpolated by drawing a smooth curve trajectory. The points that cannot be sampled or missed for various reasons will be interpolated through the motion trajectory to obtain data. It is worth mentioning that such a trajectory needs to be drawn by computer calculation, which is a cubic polynomial function, so code needs to be written to calculate the result.

[0189] Fourth, the obtained sampling data is converted into angle information, and then given to the prefabricated skeleton model to make it move with the change of unit time. The angle information we get here is the angle information presented between the points and the XYZ axis of the space in three-dimensional space, so that the skeleton motion capture is completed.

[0190] It should be noted that the above-mentioned embodiments only provide a specific implementation of a motion capture method provided in the embodiments of the present application, and the present application also includes many other optional specific implementations.

[0191] For example, S401~S402 is an optional specific method for extracting key frames from a target action video, wherein, when extracting key frames, the key frames in the target action video can be extracted based on a preset sampling frequency, wherein the sampling frequency is used to indicate the number of key frames extracted from the target action video per unit time, and the unit is sample / second, i.e., Hertz hz. For example, the target action video is subjected to key frame extraction at a first sampling frequency of 10 Hz, and 10 key frames are extracted from the target action video per second with equal frame spacing, and the key frame sequence of the obtained target action video includes key frames arranged according to the frame sequence. Among them, equal frame spacing means that the distance between the key frames is the same.

[0192] For example, S405 provides an optional specific method for obtaining the second position sequence of the key joint points based on the target interpolation function of each coordinate component. In another possible implementation, the missing frame sequence can also include a frame sequence of at least one non-key frame, that is, increasing the number of sampling points.

[0193] For another example, S406 to S408 provide an optional specific method for obtaining the motion data of the target motion video based on the second position sequence of each of the key joint points, wherein S406 is an optional step.

[0194] In summary, the motion capture method provided in the embodiment of the present application is summarized as follows: Figure 7 The method shown, such as Figure 7 As shown, this method specifically includes:

[0195] S701: extract key frames in a target action video to obtain a key frame sequence of the target action video.

[0196] In this embodiment, the key frame sequence includes a plurality of key frames arranged in frame order.

[0197] S702: Obtain position information of key joint points preset in each key frame respectively to obtain a first position sequence of each key joint point.

[0198] In this embodiment, the first position sequence is obtained by arranging the position information of the key joint points in the key frames in frame order, and the position information includes the coordinate value of each coordinate component.

[0199] S703 . For each coordinate component, solve the coefficients of each monomial in the preset interpolation function based on the first position sequence of the key joint points and the preset constraint conditions to obtain a target interpolation function.

[0200] In this embodiment, the interpolation function is a polynomial with the frame sequence as the independent variable and the coordinate value as the dependent variable, wherein the coefficient of each monomial in the interpolation function is a coefficient function with the frame sequence as the independent variable.

[0201] S704 . For each key joint point, obtain a second position sequence of the key joint point based on the target interpolation function of each coordinate component.

[0202] In this embodiment, the second position sequence is obtained by arranging the position information of the key joint points in the video frames in frame order.

[0203] S705: Acquire action data of the target action video according to the second position sequence of each key joint point.

[0204] In this embodiment, the motion data includes the corresponding relationship between the identification of each key joint point, the frame sequence, and the position information.

[0205] It can be seen from the above technical scheme that an action capture method provided by an embodiment of the present application extracts key frames in a target action video to obtain a key frame sequence of the target action video. The position information of the preset key joint points in each key frame is obtained respectively to obtain the first position sequence of each key joint point. For each coordinate component, the coefficients of each monomial in the preset interpolation function are solved based on the first position sequence of the key joint point and the preset constraint conditions to obtain the target interpolation function. For each key joint point, the second position sequence of the key joint point is obtained based on the target interpolation function of each coordinate component. According to the second position sequence of each key joint point, the action data of the target action video is obtained. Among them, the action data includes the identification, frame sequence, and the corresponding relationship of the position information of each key joint point. Since the interpolation function is a polynomial with the frame sequence as the independent variable and the coordinate value as the dependent variable, the coefficients of each monomial in the interpolation function are coefficient functions with the frame sequence as the independent variable. Therefore, based on the first position sequence of key joints, the coefficients of each monomial can be solved, and the coefficients are related to the sequence. Therefore, the second position sequence of each key joint point obtained by sampling based on the target interpolation function has high accuracy, avoiding discontinuous or inaccurate motion capture of the target action video caused by missing position information, that is, improving the continuity and accuracy of motion capture.

[0206] A motion capture method provided by an embodiment of the present application is introduced above, and a device for executing the above motion capture method will be introduced below.

[0207] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a motion capture device provided in an embodiment of the present application. Figure 8 As shown, the motion capture device 800 includes:

[0208] The frame sampling unit 801 is used to extract key frames in the target action video to obtain a key frame sequence of the target action video, wherein the key frame sequence includes multiple key frames arranged in frame order;

[0209] The joint point positioning unit 802 is used to respectively obtain the position information of the key joint points preset in each of the key frames, and obtain a first position sequence of each of the key joint points, wherein the first position sequence is obtained by arranging the position information of the key joint points in the key frames in a frame order, and the position information includes the coordinate value of each coordinate component;

[0210] The interpolation function determination unit 803 is used to solve the coefficients of each monomial in the preset interpolation function for each coordinate component based on the first position sequence of the key joint points and the preset constraint conditions to obtain a target interpolation function; the interpolation function is a polynomial with the frame sequence as an independent variable and the coordinate value as a dependent variable, wherein the coefficients of each monomial in the interpolation function are coefficient functions with the frame sequence as an independent variable;

[0211] An interpolation unit 804 is used to obtain, for each of the key joint points, a second position sequence of the key joint point based on the target interpolation function of each of the coordinate components, where the second position sequence is obtained by arranging position information of the key joint point in the video frame in frame order;

[0212] The motion capture unit 805 is used to obtain motion data of the target motion video according to the second position sequence of each of the key joints, and the motion data includes the corresponding relationship between the identification, frame sequence, and position information of each of the key joints.

[0213] In a possible implementation, when the frame sampling unit is used to extract the key frames in the target action video, it is specifically used to:

[0214] Parsing the target action video to obtain a video frame sequence of the target action video, where the video frame sequence includes all video frames arranged in frame order;

[0215] For each video frame in the video frame sequence, obtaining an inter-frame difference between the video frame and a previous video frame;

[0216] Obtaining an average inter-frame difference of each video frame sequence, and calculating an adaptive threshold of the video frame based on the average inter-frame difference;

[0217] For each video frame in the video frame sequence, if the inter-frame difference of the video frame is greater than the adaptive threshold, the video frame is taken as a key frame and extracted; if the inter-frame difference of the video frame is not greater than the adaptive threshold, the video frame is taken as a non-key frame.

[0218] In a possible implementation, the interpolation function determination unit is used to solve the coefficients of each monomial in the preset interpolation function for each of the coordinate components based on the first position sequence of the key joint points and the preset constraint conditions to obtain the target interpolation function, specifically for:

[0219] Obtaining a system of linear equations constructed according to the constraint conditions, wherein the constraint conditions include an interpolation condition, a continuity condition, a first-order derivative continuity condition, and a second-order derivative continuity condition;

[0220] For each of the key frames, the frame sequence of the key frame corresponding to the position information in the first position sequence is used as the value of the independent variable, the coordinate value of the coordinate component is used as the function value of the target interpolation function, and is substituted into the linear equation group to solve the coefficients of each monomial in the interpolation function to obtain the target coefficient of each monomial;

[0221] The target interpolation function is obtained, where the target interpolation function is a polynomial with a frame sequence as an independent variable and a coordinate value as a dependent variable, wherein the coefficients of each monomial in the target interpolation function are target coefficients of the monomial.

[0222] In a possible implementation, the interpolation unit is used to obtain, for each of the key joint points, the second position sequence of the key joint point based on the target interpolation function of each of the coordinate components, specifically to:

[0223] For each of the coordinate components, substituting the missing frame sequence into the target interpolation function to obtain a function value of the target interpolation function as a coordinate value of the coordinate component of the key joint point in the video frame corresponding to the missing frame sequence, wherein the missing frame sequence includes a frame sequence of missing key frames of a preset number of sampling frames, and the missing key frames include key frames without position information;

[0224] Combining the coordinate values ​​of all the coordinate components, obtaining the position information of the key joint point in the video frame corresponding to the missing frame sequence;

[0225] The position information of the key joint points in the video frames corresponding to the target frame sequence is sorted in ascending order according to the frame sequence to obtain a second position sequence of the key joint points, and the target frame sequence includes the frame sequence of all key frames.

[0226] In a possible implementation, when the motion capture unit is used to obtain the motion data of the target motion video according to the second position sequence of each of the key joint points, it is specifically used to:

[0227] According to the second position sequence of each of the key joint points, angle information of each of the key joint points in each sampled video frame is obtained, where the angle information is used to indicate the angle movement of the key joint point in the sampled video frame, and the sampled video frame includes a video frame corresponding to the target frame sequence;

[0228] Acquire the motion data of the target motion video, wherein the motion data also includes the corresponding relationship between the identification of each key joint point, the frame sequence and the angle information;

[0229] Generate a prefabricated skeleton file, and input the prefabricated skeleton file into a preset skeleton driving model, so that the skeleton driving model drives the key joints of the motion capture object based on the motion driving data of the key joints in the prefabricated skeleton file; wherein the motion driving data of the first type of key joints includes angle information and position information sorted in frame order, and the motion driving data of the second type of key joints includes angle information sorted in frame order.

[0230] In a possible implementation, when the motion capture unit is used to obtain the angle information of each of the key joint points in each video frame according to the second position sequence of each of the key joint points, it is specifically used to:

[0231] Based on the position information of the key joint point in the video frame and the preceding video frame of the video frame, obtaining the displacement vector of the key joint point in the video frame;

[0232] Obtaining the displacement vector of the key joint point in the video frame and the angles of each coordinate axis;

[0233] The displacement vectors of the key joint points in the video frame and the angles of each coordinate axis are summarized to obtain the angle information of the key joint points in the video frame.

[0234] The present application also provides an electronic device in an embodiment. Fig. 9 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0235] like Figure 6As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0236] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a memory card, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0237] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any motion capture method provided in the embodiment of the present application.

[0238] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any motion capture method provided in the embodiment of the present application.

[0239] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0240] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0241] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0242] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A motion capture method, characterized in that: include: Extracting key frames from the target action video to obtain a key frame sequence of the target action video, wherein the key frame sequence includes a plurality of key frames arranged in frame order; Respectively acquiring position information of key joint points preset in each of the key frames to obtain a first position sequence of each of the key joint points, wherein the first position sequence is obtained by arranging the position information of the key joint points in the key frames in a frame order, and the position information includes coordinate values ​​of each coordinate component; For each of the coordinate components, the coefficients of each monomial in the preset interpolation function are solved based on the first position sequence of the key joint points and the preset constraint conditions to obtain a target interpolation function; the interpolation function is a polynomial with a frame sequence as an independent variable and a coordinate value as a dependent variable, wherein the coefficients of each monomial in the interpolation function are coefficient functions with a frame sequence as an independent variable; For each of the key joint points, obtaining a second position sequence of the key joint points based on the target interpolation function of each of the coordinate components, wherein the second position sequence is obtained by arranging position information of the key joint points in the video frame in frame order; According to the second position sequence of each of the key joints, the action data of the target action video is obtained, and the action data includes the corresponding relationship between the identification, frame sequence, and position information of each of the key joints.

2. The motion capture method according to claim 1, characterized in that: The step of extracting key frames from the target action video includes: Parsing the target action video to obtain a video frame sequence of the target action video, where the video frame sequence includes all video frames arranged in frame order; For each video frame in the video frame sequence, obtaining an inter-frame difference between the video frame and a previous video frame; Obtaining an average inter-frame difference of each video frame sequence, and calculating an adaptive threshold of the video frame based on the average inter-frame difference; For each video frame in the video frame sequence, if the inter-frame difference of the video frame is greater than the adaptive threshold, the video frame is taken as a key frame and extracted; if the inter-frame difference of the video frame is not greater than the adaptive threshold, the video frame is taken as a non-key frame.

3. The motion capture method according to claim 1, characterized in that: For each of the coordinate components, solving the coefficients of each monomial in the preset interpolation function based on the first position sequence of the key joint points and the preset constraint conditions to obtain the target interpolation function includes: Obtaining a system of linear equations constructed according to the constraint conditions, wherein the constraint conditions include an interpolation condition, a continuity condition, a first-order derivative continuity condition, and a second-order derivative continuity condition; For each of the key frames, the frame sequence of the key frame corresponding to the position information in the first position sequence is used as the value of the independent variable, the coordinate value of the coordinate component is used as the function value of the target interpolation function, and is substituted into the linear equation group to solve the coefficients of each monomial in the interpolation function to obtain the target coefficient of each monomial; The target interpolation function is obtained, where the target interpolation function is a polynomial with a frame sequence as an independent variable and a coordinate value as a dependent variable, wherein the coefficients of each monomial in the target interpolation function are target coefficients of the monomial.

4. The motion capture method according to claim 3, characterized in that: For each of the key joint points, obtaining a second position sequence of the key joint point based on the target interpolation function of each of the coordinate components includes: For each of the coordinate components, substituting the missing frame sequence into the target interpolation function to obtain a function value of the target interpolation function as a coordinate value of the coordinate component of the key joint point in the video frame corresponding to the missing frame sequence, wherein the missing frame sequence includes a frame sequence of missing key frames of a preset number of sampling frames, and the missing key frames include key frames without position information; Combining the coordinate values ​​of all the coordinate components, obtaining the position information of the key joint point in the video frame corresponding to the missing frame sequence; The position information of the key joint points in the video frames corresponding to the target frame sequence is sorted in ascending order according to the frame sequence to obtain a second position sequence of the key joint points, and the target frame sequence includes the frame sequence of all key frames.

5. The motion capture method according to claim 4, characterized in that: Acquiring the action data of the target action video according to the second position sequence of each of the key joint points includes: According to the second position sequence of each of the key joint points, angle information of each of the key joint points in each sampled video frame is obtained, where the angle information is used to indicate the angle movement of the key joint point in the sampled video frame, and the sampled video frame includes a video frame corresponding to the target frame sequence; Acquire the motion data of the target motion video, wherein the motion data also includes the corresponding relationship between the identification of each key joint point, the frame sequence and the angle information; Generate a prefabricated skeleton file, and input the prefabricated skeleton file into a preset skeleton driving model, so that the skeleton driving model drives the key joints of the motion capture object based on the motion driving data of the key joints in the prefabricated skeleton file; wherein the motion driving data of the first type of key joints includes angle information and position information sorted in frame order, and the motion driving data of the second type of key joints includes angle information sorted in frame order.

6. The motion capture method according to claim 3, characterized in that: According to the second position sequence of each of the key joint points, the angle information of each of the key joint points in each video frame is obtained, including: Based on the position information of the key joint point in the video frame and the preceding video frame of the video frame, obtaining the displacement vector of the key joint point in the video frame; Obtaining the displacement vector of the key joint point in the video frame and the angles of each coordinate axis; The displacement vectors of the key joint points in the video frame and the angles of each coordinate axis are summarized to obtain the angle information of the key joint points in the video frame.

7. A motion capture device, characterized in that: include: A frame sampling unit, used to extract key frames in the target action video to obtain a key frame sequence of the target action video, wherein the key frame sequence includes a plurality of key frames arranged in frame order; A joint point positioning unit, used to respectively obtain the position information of the key joint points preset in each of the key frames, and obtain a first position sequence of each of the key joint points, wherein the first position sequence is obtained by arranging the position information of the key joint points in the key frames in a frame order, and the position information includes the coordinate value of each coordinate component; An interpolation function determination unit is used to solve the coefficients of each monomial in a preset interpolation function for each of the coordinate components based on the first position sequence of the key joint points and the preset constraint conditions to obtain a target interpolation function; the interpolation function is a polynomial with a frame sequence as an independent variable and a coordinate value as a dependent variable, wherein the coefficients of each monomial in the interpolation function are coefficient functions with a frame sequence as an independent variable; an interpolation unit, configured to obtain, for each of the key joint points, a second position sequence of the key joint points based on the target interpolation function of each of the coordinate components, wherein the second position sequence is obtained by arranging position information of the key joint points in the video frame in frame order; The motion capture unit is used to obtain the motion data of the target motion video according to the second position sequence of each of the key joints, and the motion data includes the corresponding relationship between the identification, frame sequence, and position information of each of the key joints.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the motion capture method as claimed in any one of claims 1 to 5.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the motion capture method as described in any one of claims 1 to 5.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the motion capture method as described in any one of claims 1 to 5.

Images