Motion search method, device, electronic device and storage medium

Abstract

The present disclosure relates to an action search method, device, electronic device and storage medium, wherein the method includes receiving video data shot during an object performing an action; extracting object action data from the video data; calculating the similarity between the object action data and preset action template data in a preset action template database; and outputting the target preset action template data with the greatest similarity in the preset action template database. In the present disclosure, each preset action template data in the preset action template database is different from each other, so the target preset action template data can be accurately determined by directly using the object action data containing the target preset action to perform a query operation in the preset action template database. Compared with the query scheme using keyword technology, since no further manual screening is required, the search efficiency can be greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to an action search method, device, electronic device and storage medium. Background Art

[0002] During the animation or game production process, it is necessary to make the virtual character show a certain animation effect. In this case, a preset action template data can be searched from the preset action template database (the preset action may include forward, jumping, rolling, etc., and the preset action template data is a template data corresponding to a preset action and can be applicable to various characters), and then the preset action template data is adapted to the virtual character to achieve the animation effect of the virtual character moving according to the preset action.

[0003] Each preset action template data in the preset action template database has a name. Currently, a keyword matching method is often used to search for a preset action template data whose name matches the keyword from the preset action template database.

[0004] However, with the gradual increase in the types and number of preset action template data in the preset action template database, there are more and more preset action template data containing the same keywords in the names. It is often the case that a lot of preset action template data appear when searching for a keyword, so further manual effort is needed to spend time and energy to screen out the target preset action template data.

[0005] Therefore, in the related art, it is difficult to efficiently retrieve the target preset action template data by searching the preset action template data through keywords. Summary of the invention

[0006] The present disclosure provides an action search method, device, electronic device and storage medium to at least solve the problem in the related art that it is difficult to efficiently retrieve target preset action template data in a preset action template database. The technical solution of the present disclosure is as follows:

[0007] According to a first aspect of the present disclosure, there is provided an action search method, comprising:

[0008] receiving video data captured during the subject's performance of an action;

[0009] extracting object motion data from the video data;

[0010] Calculate the similarity between the object action data and the preset action template data in the preset action template database;

[0011] Output the target preset action template data with the greatest similarity in the preset action template database.

[0012] The step of extracting object motion data from the video data comprises:

[0013] Inputting a plurality of continuous video frames in the video data into a motion capture model to perform motion capture operations, and obtaining a plurality of posture joint point features corresponding to the plurality of continuous video frames one by one;

[0014] According to the plurality of posture joint point features, obtaining a first posture joint point feature sequence for representing the object motion data;

[0015] Before receiving the video data captured by the object during the action, the method further includes:

[0016] Inputting each preset action template data in the preset action template database into the motion capture model to perform a motion capture operation, and obtaining multiple posture joint point features of each preset action template data;

[0017] According to the plurality of posture joint point features of the respective preset action template data, a second posture joint point feature sequence for representing the respective preset action template data is obtained.

[0018] The calculating of the similarity between the object action data and the preset action template data in the preset action template database comprises:

[0019] For each preset action template data in the preset action template database: based on the first posture joint point feature sequence and the second posture joint point feature sequence, calculating the distance between the preset action template data and the object action data;

[0020] The distance between the preset action template data and the object action data is used as the similarity between the preset action template data and the object action data.

[0021] Wherein, the video data further includes object types, and the preset action template database includes preset action template data of different object types;

[0022] Before calculating the similarity between the object action data and the preset action template data in the preset action template database, the method further includes:

[0023] According to the object type in the video data, a screening operation is performed on the preset action template database to obtain a preset action template data set that conforms to the object type;

[0024] Then, calculating the similarity between the object action data and the preset action template data in the preset action template database includes:

[0025] For each preset action template data in the preset action template data set: based on the first posture joint point feature sequence and the second posture joint point feature sequence, calculating the distance between the preset action template data and the object action data;

[0026] The distance between the preset action template data and the object action data is used as the similarity between the preset action template data and the object action data.

[0027] Wherein, the outputting of the target preset action template data with the greatest similarity in the preset action template database comprises:

[0028] Performing a sorting operation on the distances between the preset action template data and the object action data;

[0029] Determine the preset action template data corresponding to the minimum distance as the target preset action template data with the greatest similarity in the preset action template database;

[0030] Output the target preset action template data.

[0031] Wherein, the first posture joint feature sequence of the object motion data adopts A={a1, a2, ...a i ...a k} represents; k represents that the video data contains k frames; a i Represents the posture joint point features of the i-th frame in the video data;

[0032] The second posture joint feature sequence of the preset action template data adopts B={b1, b2, ...b i ...b l} represents; k≤l, l represents that the preset action template data includes l frames; b i Represents the posture joint point features of the i-th frame in the preset action template data;

[0033] Among them, a i =(x root , v root , p0, p1...., p n ), x root Indicates the position of the root node of the object in three-dimensional space, v root represents the linear velocity vector and angular velocity vector of the root node of the object in the three-dimensional space, p0 represents the rotation vector of the root node of the object in the three-dimensional space, p1...., p n Represents the rotation vectors of the other n joint points of the object in three-dimensional space;

[0034] Among them, b i =(x′root , v′ root ,p′0,p′1....,p′ n ), x′ root Indicates the position of the root node of the preset action template data in the three-dimensional space, v′ root represents the linear velocity and angular velocity vector of the root node of the preset action template data in the three-dimensional space, p′0 represents the rotation vector of the root node of the preset action template data in the three-dimensional space, p′1...., p′ n Represents the rotation vectors of the other n joint points of the preset action template data in the three-dimensional space;

[0035] The root node is the root node in the object joint point structure represented by the tree structure in the motion capture model, and n is the total number of joint points remaining after excluding the root node among all joint points pre-set in the motion capture model.

[0036] Wherein, the calculating the distance between the preset action template data and the object action data based on the first posture joint point feature sequence and the second posture joint point feature sequence includes:

[0037] Based on the first posture joint feature sequence A={a1,a2,...a i ...a k} and the second posture joint feature sequence B = {b1, b2, ... b i ...b l}, the distance d(A, B) between the preset action template data and the object action data is calculated using the following formula;

[0038]

[0039]

[0040] Among them, E(a, b) represents the distance between the posture joint point feature a in A and the posture joint point feature b in B; t represents the starting sequence number of k frames intercepted from B.

[0041] Wherein, after outputting the target preset action template data with the greatest similarity in the preset action template database, the method further includes:

[0042] receiving a distribution instruction for distributing the target preset action template data to the virtual character;

[0043] Adapting the target preset action template data to the virtual character to obtain a target action video;

[0044] Display the target action video.

[0045] According to a second aspect of the present disclosure, there is provided an action search device, comprising:

[0046] A receiving unit, configured to receive video data captured during the object performing an action;

[0047] An extraction unit, configured to extract object motion data from the video data;

[0048] A calculation unit, configured to calculate the similarity between the object action data and the preset action template data in the preset action template database;

[0049] The output unit is configured to execute outputting the target preset action template data with the greatest similarity in the preset action template database.

[0050] Wherein, the extraction unit comprises:

[0051] A capture unit is configured to input a plurality of continuous video frames in the video data into a motion capture model to perform a motion capture operation, and obtain a plurality of posture joint point features corresponding to the plurality of continuous video frames one by one;

[0052] A combining unit is configured to obtain a first posture joint point feature sequence for representing the object motion data according to the plurality of posture joint point features;

[0053] Before the receiving unit, it also includes:

[0054] The capture unit is further configured to input each preset action template data in the preset action template database into the action capture model to perform a motion capture operation, and obtain multiple posture joint point features of each preset action template data;

[0055] According to the plurality of posture joint point features of the respective preset action template data, a second posture joint point feature sequence for representing the respective preset action template data is obtained.

[0056] Wherein, the computing unit comprises:

[0057] A distance calculation unit is configured to perform, for each preset action template data in the preset action template database: calculating the distance between the preset action template data and the object action data based on the first posture joint point feature sequence and the second posture joint point feature sequence;

[0058] The distance between the preset action template data and the object action data is used as the similarity between the preset action template data and the object action data.

[0059] Wherein, the video data further includes object types, and the preset action template database includes preset action template data of different object types;

[0060] Before the computing unit, it also includes:

[0061] A screening unit is configured to perform a screening operation on the preset action template database according to the object type in the video data to obtain a preset action template data set that conforms to the object type;

[0062] Then the calculation unit comprises:

[0063] A distance calculation unit, for each preset action template data in the preset action template data set: based on the first posture joint point feature sequence and the second posture joint point feature sequence, calculates the distance between the preset action template data and the object action data;

[0064] The distance between the preset action template data and the object action data is used as the similarity between the preset action template data and the object action data.

[0065] Wherein, the output unit comprises:

[0066] A sorting unit, configured to perform a sorting operation on the distances between the respective preset action template data and the object action data;

[0067] A target determination unit is configured to determine the preset action template data corresponding to the minimum distance as the target preset action template data with the greatest similarity in the preset action template database;

[0068] The target output unit is configured to execute and output the target preset action template data.

[0069] Wherein, the first posture joint feature sequence of the object motion data adopts A={a1, a2, ...a i ...a k} represents; k represents that the video data contains k frames; a i Represents the posture joint point features of the i-th frame in the video data;

[0070] The second posture joint feature sequence of the preset action template data adopts B={b1, b2, ...b i ...b l} represents; k≤l, l represents that the preset action template data includes l frames; b i Represents the posture joint point features of the i-th frame in the preset action template data;

[0071] Among them, ai =(x root , v root , p0, p1...., p n ), x root Indicates the position of the root node of the object in three-dimensional space, v root represents the linear velocity vector and angular velocity vector of the root node of the object in the three-dimensional space, p0 represents the rotation vector of the root node of the object in the three-dimensional space, p1...., p n Represents the rotation vectors of the other n joint points of the object in three-dimensional space;

[0072] Among them, b i =(x′ root , v′ root ,p′0,p′1....,p′ n ), x′ root Indicates the position of the root node of the preset action template data in the three-dimensional space, v′ root represents the linear velocity and angular velocity vector of the root node of the preset action template data in the three-dimensional space, p′0 represents the rotation vector of the root node of the preset action template data in the three-dimensional space, p′1...., p′ n Represents the rotation vectors of the other n joint points of the preset action template data in the three-dimensional space;

[0073] The root node is the root node in the object joint point structure represented by the tree structure in the motion capture model, and n is the total number of joint points remaining after excluding the root node among all joint points pre-set in the motion capture model.

[0074] Wherein, the distance calculation unit includes:

[0075] Based on the first posture joint feature sequence A={a1,a2,...a i ...a k} and the second posture joint feature sequence B = {b1, b2, ... b i ...b l}, the distance d(A, B) between the preset action template data and the object action data is calculated using the following formula;

[0076]

[0077]

[0078] Among them, E(a, b) represents the distance between the posture joint point feature a in A and the posture joint point feature b in B; t represents the starting sequence number of k frames intercepted from B.

[0079] Wherein, after the output unit, it also includes:

[0080] A receiving unit, configured to execute and receive a dispatching instruction for dispatching the target preset action template data to the virtual character;

[0081] An adapting unit is configured to adapt the target preset action template data to the virtual character to obtain a target action video;

[0082] The display unit is configured to display the target action video.

[0083] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0084] processor;

[0085] a memory for storing instructions executable by the processor;

[0086] Wherein, the processor is configured to execute the instructions to implement the action search method of the first aspect.

[0087] According to a fourth aspect of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the action search method of the first aspect.

[0088] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0089] The present disclosure proposes video data captured during the process of an object performing an action, that is, utilizing a real object to actually perform the desired target preset action, and capturing video data during the process of the object performing the target preset action, and extracting object action data from the video data, that is, the object action data includes the target preset action.

[0090] Then, the similarity between the object action data and the preset action template data in the preset action template database is calculated, that is, the preset action template database is queried to see whether the target preset action template data corresponding to the target preset action is included.

[0091] Finally, the target preset action template data with the greatest similarity in the preset action template database is output, that is, the target preset action template data corresponding to the target preset action is found in the preset action template database, and the target preset action template data is output.

[0092] It is understandable that each preset action template data in the preset action template database is different, so directly using the object action data containing the target preset action to perform a query operation in the preset action template database can accurately determine the target preset action template data.

[0093] Compared with the query solution using keyword technology, the search efficiency can be greatly improved because no further manual screening is required.

[0094] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0096] Figure 1 is a flow chart of a first embodiment of an action search method according to an exemplary embodiment;

[0097] Figure 2 is a flow chart of a second embodiment of an action search method according to an exemplary embodiment;

[0098] Figure 3 is a flow chart of a fourth embodiment of an action search method according to an exemplary embodiment;

[0099] Figure 4 is a schematic diagram of a motion capture model for capturing human motion according to an exemplary embodiment;

[0100] Figure 5 is a flow chart of a fifth embodiment of an action search method according to an exemplary embodiment;

[0101] Figure 6 is a flow chart of a sixth embodiment of an action search method according to an exemplary embodiment;

[0102] Figure 7 is a block diagram of an action search device according to an exemplary embodiment;

[0103] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0104] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0105] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0106] Figure 1 FIG. 1 is a flow chart of a first embodiment of an action search method according to an exemplary embodiment. Figure 1 As shown, the following steps are included.

[0107] In step S101, video data captured when an object is performing an action is received.

[0108] The electronic device can receive video data sent by other devices other than itself, and can also use the camera component of the electronic device to shoot and collect video data shot during the subject's performance. In this embodiment, the process and method of the electronic device obtaining video data are not limited.

[0109] In this embodiment, the subject actually performs the desired target preset action, and the video data is obtained by shooting the process of the subject performing the action, so the video data contains the target preset action.

[0110] In step S102, object motion data is extracted from the video data.

[0111] The electronic device pre-stores a motion capture model corresponding to the object type in the preset motion template database. The object type can be a human type or an animal type, etc. Since the motion capture models of different object types are different, the electronic device stores the motion capture model corresponding to the object type.

[0112] For example, when the preset action template database is a preset action template database containing human actions, the electronic device also stores a motion capture model that can be used to capture human actions.

[0113] In the case where the preset action template database is a preset action template database containing animal actions, the electronic device also stores an action capture model that can be used to capture animal actions.

[0114] The electronic device uses a pre-stored motion capture model to extract object motion data from the video data. The object motion data can represent the target preset motion at the data level.

[0115] In step S103, the similarity between the object action data and the preset action template data in the preset action template database is calculated.

[0116] The object action data is used to perform a search operation in the preset action template database, that is, the similarity between the object action data and each preset action template data in the preset action template database is calculated.

[0117] In step S104, the target preset action template data with the greatest similarity in the preset action template database is output.

[0118] A sorting operation is performed according to the similarity between the object action data and each preset action template data, and the target preset action template data with the greatest similarity in the preset action template database is output. The target preset action template data is also the template data required to correspond to the target preset action.

[0119] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0120] Since each preset action template data in the preset action template database is different, the target preset action template data can be accurately determined by directly using the object action data containing the target preset action to perform a query operation in the preset action template database. Compared with the query solution using keyword technology, since no manual further screening is required, the search efficiency can be greatly improved.

[0121] Figure 2 FIG. 1 is a flow chart of a second embodiment of an action search method according to an exemplary embodiment. Figure 2 As shown, the following steps are included.

[0122] In step S201, each preset action template data in the preset action template database is respectively input into the motion capture model to perform a motion capture operation, so as to obtain a plurality of posture joint point features of each preset action template data.

[0123] In order to calculate the similarity between the object motion data and each preset motion template data, each preset motion template data in the preset motion template database is respectively input into the motion capture model to perform motion capture operation, and multiple posture joint point features of each preset motion template data are obtained.

[0124] In step S202, a second posture joint point feature sequence for representing each preset action template data is obtained according to a plurality of posture joint point features of each preset action template data.

[0125] Taking a preset action template data as an example, multiple posture joint point features of the preset action template data are combined into a posture joint point feature sequence for the preset action template data. For the sake of distinction, the posture joint point feature sequence of the object action data is called the first posture joint point feature sequence, and the posture joint point feature sequence of the preset action template data is called the second posture joint point feature sequence.

[0126] In step S203, video data captured when the object is performing an action is received.

[0127] In step S204, object motion data is extracted from the video data.

[0128] Multiple continuous video frames in the video data are respectively input into the motion capture model to perform motion capture operations, and multiple posture joint point features corresponding to the multiple continuous video frames are obtained; based on the multiple posture joint point features, a first posture joint point feature sequence for representing the object motion data is obtained.

[0129] It should be noted that the motion capture model in step S201 and the motion capture model in step S204 are consistent.

[0130] In step S205, the similarity between the object action data and the preset action template data in the preset action template database is calculated.

[0131] The similarity between the first posture joint point feature sequence and each second posture joint point feature sequence is calculated as the similarity between the object action data and each preset action template data in the preset action template database.

[0132] In step S206, the target preset action template data with the greatest similarity in the preset action template database is output.

[0133] For details about steps S203 to S206, please refer to Figure 1 Steps S101 to S104 in the embodiment will not be described in detail here.

[0134] Based on the beneficial effects of the first embodiment, the embodiments of the present disclosure also have the following beneficial effects:

[0135] This embodiment converts video data into a first posture joint point feature sequence, converts preset action template data into a second posture joint point feature sequence, converts two different modalities of data, object action data and preset action template data, into data of the same modality, and links two different modalities of data, object action data and preset action template data, together, thereby realizing cross-modal search operations.

[0136] According to the third embodiment of the action search method provided by the present disclosure, a specific implementation of step S205 is provided:

[0137] For each preset action template data in the preset action template database: based on the first posture joint point feature sequence and the second posture joint point feature sequence, calculate the distance between the preset action template data and the object action data; use the distance between the preset action template data and the object action data as the similarity between the preset action template data and the object action data.

[0138] Then step S206 outputs the target preset action template data with the greatest similarity in the preset action template database, specifically including:

[0139] Performing a sorting operation on the distances between the preset action template data and the object action data;

[0140] Determine the preset action template data corresponding to the minimum distance as the target preset action template data with the greatest similarity in the preset action template database;

[0141] Output the target preset action template data.

[0142] Based on the beneficial effects of the second embodiment, the embodiments of the present disclosure also have the following beneficial effects:

[0143] This embodiment provides a specific implementation method for calculating the similarity between the preset action template data and the object action data, that is, calculating the distance between the preset action template data and the object action data. It can be understood that the greater the distance, the lower the similarity, and the smaller the distance, the higher the similarity. Therefore, the preset action template data corresponding to the minimum distance is determined as the target preset action template data with the greatest similarity in the preset action template database.

[0144] Figure 3 FIG. 4 is a flow chart of a fourth embodiment of an action search method according to an exemplary embodiment. Figure 3 As shown, the following steps are included. In this embodiment, the preset action template database includes preset action template data of different object types.

[0145] In step S301, each preset action template data in the preset action template database is respectively input into the motion capture model to perform a motion capture operation, so as to obtain a plurality of posture joint point features of each preset action template data.

[0146] In order to calculate the similarity between the object motion data and each preset motion template data, each preset motion template data in the preset motion template database is respectively input into the motion capture model to perform motion capture operation, and multiple posture joint point features of each preset motion template data are obtained.

[0147] In step S302, a second posture joint point feature sequence for representing each of the preset action template data is obtained according to a plurality of posture joint point features of each of the preset action template data.

[0148] Taking a preset action template data as an example, multiple posture joint point features of the preset action template data are combined into a posture joint point feature sequence for the preset action template data. For the sake of distinction, the posture joint point feature sequence of the object action data is called the first posture joint point feature sequence, and the posture joint point feature sequence of the preset action template data is called the second posture joint point feature sequence.

[0149] In step S303, video data captured when the object is performing an action is received.

[0150] The video data includes a plurality of consecutive video frames and an object type.

[0151] In step S304, object motion data is extracted from the video data.

[0152] Multiple continuous video frames in the video data are respectively input into the motion capture model to perform motion capture operations, and multiple posture joint point features corresponding to the multiple continuous video frames are obtained; based on the multiple posture joint point features, a first posture joint point feature sequence for representing the object motion data is obtained.

[0153] It should be noted that the motion capture model in step S301 is consistent with the motion capture model in step S304.

[0154] Step S305: performing a screening operation on the preset action template database according to the object type in the video data to obtain a preset action template data set that conforms to the object type.

[0155] Since the preset action template database includes multiple object types, the preset action template database can be screened using the object types in the video data to obtain a preset action template data set that meets the object type. The screening can greatly reduce the amount of data when calculating similarity.

[0156] In step S306, the similarity between the object action data and the preset action template data in the preset action template data set is calculated.

[0157] For each preset action template data in the preset action template data set: based on the first posture joint point feature sequence and the second posture joint point feature sequence, calculating the distance between the preset action template data and the object action data;

[0158] The distance between the preset action template data and the object action data is used as the similarity between the preset action template data and the object action data.

[0159] In step S307, the target preset action template data with the greatest similarity in the preset action template data set is output.

[0160] Performing a sorting operation on the distances between the preset action template data and the object action data; determining the preset action template data corresponding to the minimum distance as the target preset action template data with the greatest similarity in the preset action template database; and outputting the target preset action template data.

[0161] On the basis of the beneficial effects of the third embodiment, the embodiments of the present disclosure also have the following beneficial effects:

[0162] In this embodiment, the video data includes object types, and the preset action template database includes preset action template data of different object types. Therefore, before calculating the similarity between the preset action template data and the object action data, the preset action template database is first filtered using the object type to filter out only the preset action template data sets that match the object type.

[0163] Since the number of preset action template data in the preset action template data set is greatly reduced, the amount of calculation of the electronic device when calculating the similarity is greatly reduced, thereby improving the calculation efficiency.

[0164] Figure 4 The figure is a schematic diagram of a motion capture model for capturing human motion according to an exemplary embodiment.

[0165] pass Figure 4 It can be seen that the motion capture model for human motion capture mainly represents human motion through the root node and other nodes in the three-dimensional space. It should be pointed out that the motion capture model for animal motion capture also represents animal motion through the root node and other nodes in the three-dimensional space, but the difference is that the number of other nodes is slightly different.

[0166] Regardless of whether it is a human action or an animal action, the video data includes a plurality of continuous video frames, and then extracting the object action data from the video data includes:

[0167] Multiple continuous video frames in the video data are respectively input into the motion capture model to perform motion capture operations, and multiple posture joint point features corresponding to the multiple continuous video frames are obtained; based on the multiple posture joint point features, a first posture joint point feature sequence for representing the object motion data is obtained.

[0168] Among them, the first posture joint feature sequence of the object action adopts A = {a1, a2, ...a i ...a k} represents; k represents that the video data contains k frames; a i Represents the posture joint point features of the i-th frame in the video data.

[0169] Among them, a i =(x root , v root , p0, p1...., p n ), x root Indicates the position of the root node of the object in three-dimensional space, v root represents the linear velocity and angular velocity vector of the root node of the object in the three-dimensional space, p0 represents the rotation vector of the root node of the object in the three-dimensional space, p1...., p n Represents the rotation vectors of the other n joint points of the object in three-dimensional space.

[0170] The root node is the root node in the object joint point structure represented by the tree structure in the motion capture model, and n is the total number of joint points remaining after excluding the root node among all joint points preset in the motion capture model.

[0171] After determining the motion capture model, each preset action template data in the preset action template database is respectively input into the motion capture model to perform the motion capture operation, and a plurality of posture joint point features of each preset action template data are obtained; and based on the plurality of posture joint point features of each preset action template data, a second posture joint point feature sequence for representing each preset action template data is obtained.

[0172] The second posture joint feature sequence of the preset action template data adopts B={b1, b2, ...b i ...b l} represents; k≤l, l represents that the preset action template data contains l frames; b i Represents the posture joint point features of the i-th frame in the preset action template data.

[0173] Among them, b i =(x′ root , v′ root ,p′0,p′1....,p′n ), x′ root Indicates the position of the root node of the preset action template data in the three-dimensional space, v′ root represents the linear velocity vector and angular velocity vector of the root node of the preset action template data in the three-dimensional space, p′0 represents the rotation vector of the root node of the preset action template data in the three-dimensional space, p′1...., p′ n Represents the rotation vectors of the other n joint points of the preset action template data in three-dimensional space.

[0174] The root node is the root node in the object joint point structure represented by the tree structure in the motion capture model, and n is the total number of joint points remaining after excluding the root node among all joint points pre-set in the motion capture model.

[0175] The specific implementation of the action search method is introduced below. Figure 5 is a flowchart of a fifth embodiment of an action search method according to an exemplary embodiment. Figure 5 The steps shown include the following steps.

[0176] Step S501: receiving video data captured when an object performs an action.

[0177] The electronic device can receive video data sent by other devices other than itself, and can also use the camera component of the electronic device to shoot and collect video data shot during the subject's performance. In this embodiment, the process and method of the electronic device obtaining video data are not limited.

[0178] The video data includes a plurality of continuous video frames, or the video data includes a plurality of continuous video frames and a search keyword.

[0179] Step S502: extracting object motion data from the video data.

[0180] The electronic device pre-stores a motion capture model corresponding to the object type.

[0181] For example, when the preset action template database is a preset action template database containing human actions, the electronic device also stores a motion capture model that can be used to capture human actions.

[0182] In the case where the preset action template database is a preset action template database containing animal actions, the electronic device also stores an action capture model that can be used to capture animal actions.

[0183] Step S503: Calculate the similarity between the object action data and the preset action template data in the preset action template database.

[0184] The first implementation method: directly perform a search operation on the preset action template database.

[0185] Calculate the similarity between the object action data and each preset action template data in the preset action template database. In the present application, a distance method is used to calculate the similarity: determine each preset action template data from the preset action template database; for each preset action template data: based on the first posture joint feature sequence and the second posture joint feature sequence, calculate the distance between the preset action template data and the object action data.

[0186] The distance can reflect the similarity. The smaller the distance, the greater the similarity, and the larger the distance, the smaller the similarity.

[0187] Based on the first posture joint feature sequence A={a1,a2,...a i ...a k} and the second posture joint feature sequence B = {b1, b2, ...b i ...b l}, the distance d(A, B) between the object action and the preset action template data in the preset action template data set is calculated using the following formula;

[0188]

[0189]

[0190] Among them, E(a, b) represents the distance between the posture joint point feature a in A and the posture joint point feature b in B; t represents the starting sequence number of k frames intercepted from B.

[0191] Formula meaning: Since the video data contains k frames and the preset action template data contains l frames, the number is not equal. In order to calculate the similarity, the preset action template data is divided into multiple groups, each containing k frames. For example, the video data contains 5 frames and the preset action template data contains 7 frames, then the preset action template data 1 to 5 are grouped as one, 2 to 6 as one group, and 3 to 7 as one group, so as to achieve the equal number.

[0192] t represents the starting number of each group in the preset action template data, and the distance is calculated for each group in the preset action template data. Then, the minimum distance among multiple group corresponding distances is selected as d(A, B).

[0193] The present invention converts video data into a first posture joint point feature sequence, and provides a similarity measurement formula between the first posture joint point feature sequence and the second posture joint point feature sequence, so that two different modalities of data, namely object action data and abstract preset action template data in a preset action template database, can be linked together, thereby realizing cross-modal search operations.

[0194] The second implementation method: the video data includes a plurality of continuous video frames and a search keyword, and the search keyword is first used to perform a filtering operation on a preset database.

[0195] A screening operation is performed on the preset action template database according to the search keyword to obtain a preset action template data set whose name contains the search keyword; and a similarity between the object action and the preset action template data in the preset action template data set is calculated.

[0196] Since the amount of calculation for directly using the first implementation is large, for this reason, the preset action template database can be first screened using the search keyword, and this screening operation can filter out most of the irrelevant preset action template data, and then calculate the similarity between the object action data and the preset action template data in the preset action template data set related to the search keyword. This can greatly reduce the amount of calculation.

[0197] In this application, the distance method is used to calculate the similarity: based on the first posture joint point feature sequence and the second posture joint point feature sequence, the similarity between the object action and the preset action template data in the preset action template data set is calculated; if the distance is smaller, the similarity is greater, and if the distance is larger, the similarity is smaller. The calculation process can be seen in the first implementation method, which will not be repeated here.

[0198] Step S504: outputting the target preset action template data with the greatest similarity in the preset action template database.

[0199] The first implementation method corresponding to the first implementation method in step S503:

[0200] The distances between the object action and the preset action template data in the preset action template database are sorted, and the preset action template data corresponding to the minimum distance is output. The electronic device can display the target preset action template data with the greatest similarity in the preset action template database.

[0201] A second implementation method corresponding to the second implementation method in step S503:

[0202] The distances between the object action and the preset action template data in the preset action template data set are sorted, and the preset action template data corresponding to the minimum distance is output. The electronic device can display the target preset action template data with the greatest similarity in the preset action template database.

[0203] It can be understood that, before the electronic device outputs the target preset action template data with the greatest similarity in the preset action template database, the electronic device further includes:

[0204] Determine whether the maximum similarity is greater than a preset threshold. The preset threshold is used to measure whether the preset action template data is similar to the object action. If the maximum similarity is greater than the threshold, it is determined that the preset action template database has preset action template data similar to the object action, and the target preset action template data with the greatest similarity in the preset action template database is output.

[0205] Otherwise, it means that the preset action template database does not have preset action template data similar to the object action, and outputting the target preset action template data with the greatest similarity in the preset action template database is not executed.

[0206] The technical solution provided in this embodiment brings at least the following beneficial effects:

[0207] In the preset action template database of the present disclosure, each preset action template data is different from each other, so directly using the object action data containing the target preset action to perform a query operation in the preset action template database can accurately determine the target preset action template data. Compared with the query solution using only keyword technology, since no further manual screening is required, the search efficiency can be greatly improved.

[0208] The present invention converts video data into a first posture joint point feature sequence, and provides a similarity measurement formula between the first posture joint point feature sequence and the second posture joint point feature sequence, so that two different modalities of data, namely object action data and abstract preset action template data in a preset action template database, can be linked together, thereby realizing cross-modal search operations.

[0209] Figure 6 is a flowchart of a sixth embodiment of an action search method according to an exemplary embodiment. Figure 6 The steps shown include the following steps.

[0210] Step S601: receiving a distribution instruction for distributing target preset action template data to a virtual character.

[0211] After the electronic device determines the target preset action template data, the target preset action template data can be displayed. If the animation or game production personnel are satisfied with the target preset action template data, a dispatch instruction can be issued.

[0212] Step S602: adapting the target preset action template data to the virtual character to obtain a target action video.

[0213] The electronic device adapts the target preset action template data to the virtual character, adapts the size and each joint point of the virtual character to the target preset action template data, and obtains the target action video.

[0214] Step S603: display the target action video.

[0215] In this embodiment, the purpose of searching for the target preset action template data is to add the target preset action template data to the virtual character. Therefore, after searching and obtaining the target preset action template data, the virtual character can be adapted to the target preset action template data to obtain the target action video of the virtual character moving according to the target preset action template data.

[0216] Figure 7 is a block diagram of an action search device according to an exemplary embodiment. Figure 7 The device includes a receiving unit 61, an extracting unit 62, a calculating unit 63 and an output unit 64.

[0217] The receiving unit 61 is configured to receive video data captured during the object's action execution;

[0218] An extraction unit 62 is configured to extract object motion data from the video data;

[0219] A calculation unit 63, configured to calculate the similarity between the object action data and the preset action template data in the preset action template database;

[0220] The output unit 64 is configured to execute outputting the target preset action template data with the greatest similarity in the preset action template database.

[0221] Wherein, the extraction unit 62 includes:

[0222] The capture unit 621 is configured to input a plurality of continuous video frames in the video data into a motion capture model to perform a motion capture operation, and obtain a plurality of posture joint point features corresponding to the plurality of continuous video frames one by one;

[0223] The combining unit 622 is configured to obtain a first posture joint point feature sequence for representing the object motion data according to the plurality of posture joint point features;

[0224] Before the receiving unit 61, the capturing unit 621 is further configured to input each preset action template data in the preset action template database into the action capture model to perform the action capture operation, and obtain multiple posture joint point features of each preset action template data;

[0225] According to the plurality of posture joint point features of the respective preset action template data, a second posture joint point feature sequence for representing the respective preset action template data is obtained.

[0226] Wherein, the calculation unit 63 includes:

[0227] The distance calculation unit 631 is configured to perform, for each preset action template data in the preset action template database: based on the first posture joint point feature sequence and the second posture joint point feature sequence, calculate the distance between the preset action template data and the object action data;

[0228] The determining unit 632 is configured to use the distance between the preset action template data and the object action data as the similarity between the preset action template data and the object action data.

[0229] Wherein, the video data further includes object types, and the preset action template database includes preset action template data of different object types;

[0230] Before the calculation unit 63, it also includes:

[0231] A screening unit 65 is configured to perform a screening operation on the preset action template database according to the object type in the video data to obtain a preset action template data set that matches the object type;

[0232] Then the calculation unit 63 includes:

[0233] The distance calculating unit 631 calculates the distance between the preset action template data and the object action data based on the first posture joint point feature sequence and the second posture joint point feature sequence for each preset action template data in the preset action template data set;

[0234] The determining unit 632 is configured to use the distance between the preset action template data and the object action data as the similarity between the preset action template data and the object action data. The output unit 64 includes:

[0235] A sorting unit 641 is configured to perform a sorting operation on the distances between the preset action template data and the object action data;

[0236] The target determination unit 642 is configured to determine the preset action template data corresponding to the minimum distance as the target preset action template data with the greatest similarity in the preset action template database;

[0237] The target output unit 643 is configured to execute output of the target preset action template data.

[0238] Wherein, the first posture joint feature sequence of the object motion data adopts A={a1, a2, ...a i ...a k} represents; k represents that the video data contains k frames; a i Represents the posture joint point features of the i-th frame in the video data;

[0239] The second posture joint feature sequence of the preset action template data adopts B={b1, b2, ...b i ...b l} represents; k≤l, l represents that the preset action template data includes l frames; b i Represents the posture joint point features of the i-th frame in the preset action template data;

[0240] Among them, a i =(x root , v root , p0, p1...., p n ), x root Indicates the position of the root node of the object in three-dimensional space, v root represents the linear velocity vector and angular velocity vector of the root node of the object in the three-dimensional space, p0 represents the rotation vector of the root node of the object in the three-dimensional space, p1...., p n Represents the rotation vectors of the other n joint points of the object in three-dimensional space;

[0241] Where bi = (x′ root , v′ root ,p′0,p′1....,p′ n ), x′ root Indicates the position of the root node of the preset action template data in the three-dimensional space, v′ root represents the linear velocity and angular velocity vector of the root node of the preset action template data in the three-dimensional space, p′0 represents the rotation vector of the root node of the preset action template data in the three-dimensional space, p′1...., p′ n Represents the rotation vectors of the other n joint points of the preset action template data in the three-dimensional space;

[0242] The root node is the root node in the object joint point structure represented by the tree structure in the motion capture model, and n is the total number of joint points remaining after excluding the root node among all joint points pre-set in the motion capture model.

[0243] The distance calculation unit comprises:

[0244] Based on the first posture joint feature sequence A={a1,a2,...a i ...a k} and the second posture joint feature sequence B = {b1, b2, ... b i ...b l}, the distance d(A, B) between the preset action template data and the object action data is calculated using the following formula;

[0245]

[0246]

[0247] Among them, E(a, b) represents the distance between the posture joint point feature a in A and the posture joint point feature b in B; t represents the starting sequence number of k frames intercepted from B.

[0248] After the output unit 64, it also includes:

[0249] The receiving unit 66 is configured to receive and execute a distribution instruction for distributing the target preset action template data to the virtual character;

[0250] An adaptation unit 67 is configured to adapt the target preset action template data to the virtual character to obtain a target action video;

[0251] The display unit 68 is configured to display the target action video.

[0252] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been detailed in the embodiment of the method, and will not be elaborated here.

[0253] Figure 8 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an exemplary embodiment. Figure 8 , the electronic device includes a memory 11 and a processor 12.

[0254] The memory 11 is used to store the processor executable instructions, and the processor is configured to execute the instructions to implement any one of the action search methods shown in Embodiments 1 to 6.

[0255] The present disclosure also provides a storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute any one of the action search methods shown in Embodiments 1 to 6.

[0256] Alternatively, the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0257] The action search method in the above embodiment has been characterized in detail in the embodiment of the method and will not be elaborated here.

[0258] The present invention discloses a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device can execute any one of the action search methods shown in Embodiments 1 to 6.

[0259] The action search method in the above embodiment has been characterized in detail in the embodiment of the method and will not be elaborated here.

[0260] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0261] It should be understood that the present disclosure is not limited to the precise structures that have been characterized above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An action search method, characterized in that: include: receiving video data captured during the subject's performance of an action; extracting object motion data from the video data; Calculate the similarity between the object action data and the preset action template data in the preset action template database; Outputting the target preset action template data with the greatest similarity in the preset action template database; The step of extracting object motion data from the video data comprises: Inputting a plurality of continuous video frames in the video data into a motion capture model to perform motion capture operations, and obtaining a plurality of posture joint point features corresponding to the plurality of continuous video frames one by one; According to the plurality of posture joint point features, obtaining a first posture joint point feature sequence for representing the object motion data; Before receiving the video data captured by the object during the action, the method further includes: Inputting each preset action template data in the preset action template database into the motion capture model to perform a motion capture operation, and obtaining multiple posture joint point features of each preset action template data; According to the plurality of posture joint point features of the respective preset action template data, a second posture joint point feature sequence for representing the respective preset action template data is obtained; The first posture joint feature sequence of the object motion data is A={a1, a2, ... a i …a k } represents; k represents that the video data contains k frames; a i Represents the posture joint point features of the i-th frame in the video data; The second posture joint point feature sequence of the preset action template data adopts B={b1, b2, ...b i …b l } represents; k≤l, l represents that the preset action template data includes l frames; b i Represents the posture joint point features of the i-th frame in the preset action template data; Among them, a i =(x root ,v root ,p0,p1....,p n ), x root Indicates the position of the root node of the object in three-dimensional space, v root represents the linear velocity vector and angular velocity vector of the root node of the object in the three-dimensional space, p0 represents the rotation vector of the root node of the object in the three-dimensional space, p1…,p n Represents the rotation vectors of the other n joint points of the object in three-dimensional space; Among them, b i =(x′ root ,v′ root ,p′0,p′1....,p′ n ), x′ root Indicates the position of the root node of the preset action template data in the three-dimensional space, v′ root represents the linear velocity and angular velocity vector of the root node of the preset action template data in the three-dimensional space, p′0 represents the rotation vector of the root node of the preset action template data in the three-dimensional space, p′1…,p′ n Represents the rotation vectors of the other n joint points of the preset action template data in the three-dimensional space; The root node is the root node in the object joint point structure represented by the tree structure in the motion capture model, and n is the total number of joint points remaining after excluding the root node among all joint points pre-set in the motion capture model.

2. The motion search method according to claim 1, characterized in that: The calculating of the similarity between the object action data and the preset action template data in the preset action template database comprises: For each preset action template data in the preset action template database: based on the first posture joint point feature sequence and the second posture joint point feature sequence, calculating the distance between the preset action template data and the object action data; The distance between the preset action template data and the object action data is used as the similarity between the preset action template data and the object action data.

3. The motion search method according to claim 1, characterized in that: The video data also includes object types, and the preset action template database includes preset action template data of different object types; Before calculating the similarity between the object action data and the preset action template data in the preset action template database, the method further includes: According to the object type in the video data, a screening operation is performed on the preset action template database to obtain a preset action template data set that conforms to the object type; Then, calculating the similarity between the object action data and the preset action template data in the preset action template database includes: For each preset action template data in the preset action template data set: based on the first posture joint point feature sequence and the second posture joint point feature sequence, calculating the distance between the preset action template data and the object action data; The distance between the preset action template data and the object action data is used as the similarity between the preset action template data and the object action data.

4. The motion search method according to claim 2 or 3, characterized in that: The outputting of the target preset action template data with the greatest similarity in the preset action template database comprises: Performing a sorting operation on the distances between the preset action template data and the object action data; Determine the preset action template data corresponding to the minimum distance as the target preset action template data with the greatest similarity in the preset action template database; Output the target preset action template data.

5. The motion search method according to claim 2, characterized in that: The calculating the distance between the preset action template data and the object action data based on the first posture joint point feature sequence and the second posture joint point feature sequence comprises: Based on the first posture joint feature sequence A={a1, a2, …a i …a k } and the second posture joint feature sequence B = {b1, b2, ... b i …b l }, the distance d(A, B) between the preset action template data and the object action data is calculated using the following formula; Among them, E(a,b) represents the distance between the posture joint point feature a in A and the posture joint point feature b in B; t represents the starting sequence number of k frames intercepted from B.

6. The motion search method according to claim 1, characterized in that: After outputting the target preset action template data with the greatest similarity in the preset action template database, the method further includes: receiving a distribution instruction for distributing the target preset action template data to the virtual character; Adapting the target preset action template data to the virtual character to obtain a target action video; Display the target action video.

7. An action search device, characterized in that: include: A receiving unit, configured to receive video data captured during the object performing an action; An extraction unit, configured to extract object motion data from the video data; A calculation unit, configured to calculate the similarity between the object action data and the preset action template data in the preset action template database; An output unit is configured to execute outputting the target preset action template data with the greatest similarity in the preset action template database; Wherein, the extraction unit comprises: A capture unit is configured to input a plurality of continuous video frames in the video data into a motion capture model to perform a motion capture operation, and obtain a plurality of posture joint point features corresponding to the plurality of continuous video frames one by one; A combining unit is configured to obtain a first posture joint point feature sequence for representing the object motion data according to the plurality of posture joint point features; Before the receiving unit, it also includes: The capture unit is further configured to input each preset action template data in the preset action template database into the action capture model to perform a motion capture operation, and obtain multiple posture joint point features of each preset action template data; According to the plurality of posture joint point features of the respective preset action template data, a second posture joint point feature sequence for representing the respective preset action template data is obtained; The first posture joint feature sequence of the object motion data is A={a1, a2, ... a i …a k } represents; k represents that the video data contains k frames; a i Represents the posture joint point features of the i-th frame in the video data; The second posture joint point feature sequence of the preset action template data adopts B={b1, b2, ...b i …b l } represents; k≤l, l represents that the preset action template data includes l frames; b i Represents the posture joint point features of the i-th frame in the preset action template data; Among them, a i =(x root ,v root ,p0,p1....,p n ), x root Indicates the position of the root node of the object in three-dimensional space, v root represents the linear velocity vector and angular velocity vector of the root node of the object in the three-dimensional space, p0 represents the rotation vector of the root node of the object in the three-dimensional space, p1...., pn represent the rotation vectors of the other n joint points of the object in the three-dimensional space; Among them, b i =(x′ root ,v′ root ,p′0,p′1....,p′ n ), x′ root Indicates the position of the root node of the preset action template data in the three-dimensional space, v′ root represents the linear velocity and angular velocity vector of the root node of the preset action template data in the three-dimensional space, p′0 represents the rotation vector of the root node of the preset action template data in the three-dimensional space, p′1…,p′ n Represents the rotation vectors of the other n joint points of the preset action template data in the three-dimensional space; The root node is the root node in the object joint point structure represented by the tree structure in the motion capture model, and n is the total number of joint points remaining after excluding the root node among all joint points pre-set in the motion capture model.

8. The motion search device according to claim 7, characterized in that: The computing unit comprises: A distance calculation unit is configured to perform, for each preset action template data in the preset action template database: calculating the distance between the preset action template data and the object action data based on the first posture joint point feature sequence and the second posture joint point feature sequence; The distance between the preset action template data and the object action data is used as the similarity between the preset action template data and the object action data.

9. The motion search device according to claim 7, characterized in that: The video data also includes object types, and the preset action template database includes preset action template data of different object types; Before the computing unit, it also includes: A screening unit is configured to perform a screening operation on the preset action template database according to the object type in the video data to obtain a preset action template data set that conforms to the object type; Then the calculation unit comprises: A distance calculation unit, for each preset action template data in the preset action template data set: based on the first posture joint point feature sequence and the second posture joint point feature sequence, calculates the distance between the preset action template data and the object action data; The distance between the preset action template data and the object action data is used as the similarity between the preset action template data and the object action data.

10. The motion search device according to claim 8 or 9, characterized in that: The output unit comprises: A sorting unit, configured to perform a sorting operation on the distances between the respective preset action template data and the object action data; A target determination unit is configured to determine the preset action template data corresponding to the minimum distance as the target preset action template data with the greatest similarity in the preset action template database; The target output unit is configured to execute and output the target preset action template data.

11. The motion search device according to claim 8, characterized in that: The distance calculation unit comprises: Based on the first posture joint feature sequence A={a1, a2, …a i …a k } and the second posture joint feature sequence B = {b1, b2, ... b i …b l }, the distance d(A, B) between the preset action template data and the object action data is calculated using the following formula; Among them, E(a,b) represents the distance between the posture joint point feature a in A and the posture joint point feature b in B; t represents the starting sequence number of k frames intercepted from B.

12. The motion search device according to claim 9, characterized in that: After the output unit, it also includes: A receiving unit, configured to execute and receive a dispatching instruction for dispatching the target preset action template data to the virtual character; An adapting unit is configured to adapt the target preset action template data to the virtual character to obtain a target action video; The display unit is configured to display the target action video.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the action search method as described in any one of claims 1 to 6. 14 . A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the motion search method according to claim 1 .

Images