A motion capture data retrieval method and apparatus

Abstract

The application discloses a motion capture data retrieval method and device. The method comprises: learning an encoded posture dictionary using an unsupervised adversarial autoencoder; representing each motion capture sequence according to the learned posture dictionary to obtain a database; representing a query sequence according to the learned posture dictionary to obtain a symbol sequence; aligning the symbol sequence with each sub-sequence in the database through cross-step time alignment, and retrieving the most matched sub-sequence from the database. For a motion capture frame, the posture dictionary item most similar to the posture in the frame is found in the posture dictionary, and the corresponding dictionary index is used to represent the frame. The scheme has flexibility, and can flexibly adjust the granularity of similarity search, the tolerance for abnormal frames, and the balance between retrieval accuracy and running efficiency. In addition, since data labels are not required, the expansion of the database is relatively easy.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for retrieving motion capture data. Background Technology

[0002] Motion capture is widely used in filmmaking, video game development, and computer-aided medical diagnosis. This widespread application generates a massive amount of motion capture data, and there is often a need to retrieve specific sequences or subsequences from a large database of motion capture sequences. While it's possible to search for motion capture sequences in a database using text tags, annotating all sequences is tedious, and text tags may not be sufficiently descriptive. In contrast, content-based retrieval is often more desirable. This approach finds sequences or subsequences in the database that are similar to the query sample. Consequently, content-based motion capture data retrieval has become a research hotspot in recent years.

[0003] Given a query example, some algorithms retrieve the entire motion capture sequence similar to the example, while others retrieve sub-motion capture sequences (or simply sub-sequences) similar to the example. Typically, the former performs matching at a coarser granularity without temporally aligning sequences or sub-sequences in the database with the query example, thus achieving higher efficiency; the latter performs similarity searches at a finer granularity, resulting in lower efficiency. Existing algorithms can also be categorized into supervised and unsupervised algorithms. While the former offers higher performance, it usually requires extensive data labeling, is sensitive to label ambiguity, has limited generality due to the range of labels, and is not well-suited for sub-sequence level similarity retrieval.

[0004] There is currently no effective solution to the technical problems of motion capture data retrieval in the existing technologies mentioned above. Summary of the Invention

[0005] The embodiments of this disclosure provide a motion capture data retrieval method and apparatus to at least solve the technical problems existing in the prior art.

[0006] To achieve fine-grained similarity retrieval and algorithm versatility, this application focuses on subsequence retrieval and employs an unsupervised approach. Specifically, this application proposes a flexible scheme for content-based motion capture data retrieval. The core of this scheme is to use an unsupervised neural network-based adversarial autoencoder (AAE) technique to learn an encoded pose dictionary. Each motion capture sequence is represented based on this learned dictionary, and given query sample times are aligned to sequences in the database to find similar subsequences. This scheme is flexible, allowing for adjustments to the granularity of similarity search, tolerance for singular frames, and a balance between retrieval accuracy and operational efficiency.

[0007] According to one aspect of the present disclosure, a motion capture data retrieval method is provided, comprising:

[0008] Use an unsupervised adversarial autoencoder to learn the encoded pose dictionary;

[0009] The database is obtained by representing each motion capture sequence using the learned pose dictionary;

[0010] Based on the learned pose dictionary representation of the query sequence, a symbol sequence is obtained;

[0011] The symbol sequence is aligned with each subsequence in the database by step-time alignment, and the most matching subsequence is retrieved from the database.

[0012] According to another aspect of the present disclosure, a storage medium is also provided, the storage medium including a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0013] According to another aspect of the present disclosure, a motion capture data retrieval device is also provided, comprising:

[0014] The AAE module is used to learn the encoded pose dictionary using an unsupervised adversarial autoencoder.

[0015] The motion capture sequence module is used to represent each motion capture sequence according to the learned pose dictionary to obtain a database;

[0016] The query sequence module is used to obtain a symbol sequence based on the learned pose dictionary representation of the query sequence;

[0017] The step-time alignment and retrieval module is used to align the symbol sequence with each subsequence in the database through step-time alignment, and retrieve the most matching subsequence from the database.

[0018] According to another aspect of the present disclosure, a motion capture data retrieval device is also provided, comprising:

[0019] First processor; and

[0020] A first memory, connected to the first processor, is used to provide the first processor with instructions to perform the following processing steps:

[0021] Use an unsupervised adversarial autoencoder to learn the encoded pose dictionary;

[0022] The database is obtained by representing each motion capture sequence based on the learned pose dictionary;

[0023] Based on the learned pose dictionary representation of the query sequence, a symbol sequence is obtained;

[0024] The symbol sequence is aligned with each subsequence in the database by step-time alignment, and the most matching subsequence is retrieved from the database.

[0025] The most significant features of the proposed solution in this application are as follows, as described in the embodiments disclosed herein.

[0026] 1. Fine-grained similarity search. This scheme utilizes AAE-based pose dictionary learning to convert each original motion capture sequence into a compact symbol string, and finds matches with query samples from the database at the subsequence level.

[0027] 2. Robustness to singular frames. Unlike existing subsequence retrieval methods, the strided temporal alignment (STA) method proposed in this application relaxes the continuity constraint and has strong tolerance for singular frames that appear intermittently in the sequence.

[0028] 3. Flexible Algorithm Framework. This application is flexible in several aspects. It can be used for both subsequence retrieval and whole sequence retrieval. By adjusting the stride size, different levels of singular frame tolerance can be achieved. By adjusting the size of the pose dictionary, a flexible balance can be achieved between retrieval accuracy and speed. Furthermore, since no data labels are required, database expansion is relatively easy. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this application, illustrate exemplary embodiments of this disclosure and are used to explain this disclosure, but do not constitute an undue limitation of this disclosure. In the drawings:

[0030] Figure 1 This is a hardware structure block diagram of a computing device for implementing the method described in Embodiment 1 of this disclosure;

[0031] Figure 2 This is a schematic diagram of the motion capture data retrieval system according to Embodiment 1 of this disclosure;

[0032] Figure 3 This is a flowchart illustrating the motion capture data retrieval method according to the first aspect of Embodiment 1 of this disclosure;

[0033] Figure 4 This is a schematic diagram of the motion capture sequence symbolization process according to Embodiment 1 of this disclosure;

[0034] Figure 5 This is an example diagram of a 341-frame cartwheel motion capture sequence simplified into an 8-symbol sequence;

[0035] Figure 6 This is a schematic diagram illustrating time alignment based on STA and DTW. The symbolized query sequence and database sequence are shown on the left and bottom, respectively. Path 3 and Path 1 represent the alignment between the two sub-database sequences and the query sequence obtained using STA, respectively, while Path 2 represents the alignment between the entire database sequence and the query sequence obtained using DTW.

[0036] Figure 7 This is a statistical diagram illustrating the P@n values ​​of the SOM, WG, MS, and PESTA algorithms for full sequence retrieval on the first dataset.

[0037] Figure 8 This is a schematic diagram of the PR curves of several algorithms, including SOM, WG, MS, and PESTA, for full sequence retrieval on the first dataset.

[0038] Figure 9 This is a schematic diagram of the confusion matrix of the SOM, WG, MS and PESTA algorithms for full sequence retrieval on the first dataset;

[0039] Figure 10 This is a schematic diagram of the P@n statistics, PR curve, and confusion matrix of the SOM algorithm for subsequence retrieval on the second dataset;

[0040] Figure 11 This is a schematic diagram of the P@n statistics, PR curve, and confusion matrix of the PESTA algorithm for subsequence retrieval on the second dataset;

[0041] Figure 12 This is a schematic diagram of a motion capture data retrieval device according to the first aspect of Embodiment 2 of this disclosure;

[0042] Figure 13 This is a schematic diagram of a motion capture data retrieval device according to the second aspect of Embodiment 2 of this disclosure. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0044] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] Example 1

[0046] According to this embodiment, a method embodiment for motion capture data retrieval is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] The method embodiments provided in this example can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Figure 1 A hardware block diagram of a computing device for implementing a motion capture data retrieval method is shown. Figure 1 As shown, a computing device may include one or more processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), memory for storing data, transmission devices for communication functions, and input / output interfaces. The memory, transmission devices, and input / output interfaces are connected to the processor via a bus. In addition, it may also include a display, keyboard, and cursor control device connected to the input / output interfaces. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, a computing device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0048] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to as "data processing circuits" in this application. The data processing circuit can be embodied, in whole or in part, as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuit can be a single, independent processing module, or it can be integrated, in whole or in part, into any other element in a computing device. As involved in the embodiments of this disclosure, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0049] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the motion capture data retrieval method in this embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the motion capture data retrieval method of the aforementioned application. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the computing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0050] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the computing device's communications provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0051] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of the computing device.

[0052] It should be noted here that, in some optional embodiments, the above... Figure 1The computing device shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computing devices.

[0053] This application proposes a content-based motion capture data retrieval method, which is crucial for the effective reuse of motion capture data resources. This method can discover similarities at the sub-sequence level, is robust to singular frames appearing in the sequence, and achieves a balance between retrieval accuracy and time efficiency. The method first uses an unsupervised adversarial autoencoder to learn an encoded pose dictionary, and then uses this dictionary to create a concise symbolic representation of a motion capture sequence. Next, it aligns the query sequence with each sequence in the database through strided temporal alignment, and retrieves the most matching sub-sequence from the database. Experiments were conducted on both publicly available motion capture datasets and our own captured motion capture dataset, and the experimental results demonstrate the excellent performance of the proposed method.

[0054] Regarding the fundamental problem of sub-motion capture sequence retrieval, this application formally defines it as follows. This application denotes the query sequence as Q = [f1, f2, ..., f...]. q ], recording the motion capture database as here This represents the i-th frame in the query sequence (represented by an h-dimensional vector or an h×1 matrix). This represents the j-th motion capture sequence in the database, which contains nj frames. Sub-motion capture sequence retrieval involves retrieving the sequence from... Retrieve a maximum subsequence set from the data. Make And d(Q, R) i ) < U d .here, Representing the subsequence relation, d(.) is a distance function and U d It is a distance threshold. For ease of description, this application uses a matrix. This is used to record all the original motion capture sequence data in the database, and the column matrix may be treated as a vector or vice versa in the following text.

[0055] Figure 2 This is a schematic diagram of the motion capture data retrieval system according to this embodiment. (Refer to...) Figure 2 As shown, the system includes: a front-end portable electronic terminal 100 (e.g., a laptop computer), a computing device 200 for a motion capture data retrieval method, and a cloud server 300. It should be noted that the computing device 200 for the motion capture data retrieval method in the system can utilize the hardware structure described above.

[0056] Under the aforementioned operating environment, according to the first aspect of this embodiment, a motion capture data retrieval method is provided, the method comprising: Figure 2 The computing device 200 implements the motion capture data retrieval method shown in the figure. Figure 3 A flowchart illustrating the method is shown below. (Refer to...) Figure 3 As shown, the solution to the sub-motion capture sequence retrieval problem proposed in this application consists of two stages: preprocessing and retrieval. The preprocessing stage symbolizes all motion capture sequences in the database. This method first uses unsupervised adversarial autoencoders to learn a (encoded) pose dictionary. Then, based on this dictionary, it represents each motion sequence in the database with a concise symbol sequence. In the retrieval stage, the query sequence is first converted into a concise symbol sequence according to the pre-learned pose dictionary, and then aligned to the matching sequences in the database to retrieve similar symbol sub-sequences and the corresponding original motion capture sub-sequences. The following describes the method flow and technical details of this application in detail:

[0057] 1. Symbolization of motion capture sequences

[0058] 1.1 Overall Strategy

[0059] Raw motion capture data is typically high-dimensional, containing significant spatial and temporal redundancy. This application aims to reduce this redundancy to achieve concise and discriminative motion descriptions. Existing motion capture data retrieval algorithms employ various methods to reduce data redundancy. One relatively effective approach is to first generate a dictionary of poses or motion segments, and then transform each raw motion capture sequence into a sequence of symbols (dictionary indices), thereby reducing spatial redundancy. However, current methods operate on the data in the raw pose or motion space rather than in other, more discriminative feature spaces. Furthermore, SOM (self-organizing feature map) is generally better suited for visualization than data clustering.

[0060] This application also employs a dictionary-based motion capture sequence symbolization method, but more effectively reduces spatial and temporal redundancy. Regarding spatial redundancy reduction, the algorithm uses AAE (Action-Encoding-Based Image) technology to deeply mine discriminative pose features, thereby automatically constructing a compact and highly descriptive (encoded) pose description dictionary. To reduce temporal redundancy, this application uses a repetitive symbol pattern detection technique to reduce the two or more consecutive occurrences of the same symbol pattern to one.

[0061] 1.2 Space Redundancy Reduction

[0062] A raw motion capture frame contains rotation information for each joint relative to its parent joint. Rotation can be represented in various forms, with Euler angles being one of the commonly used. For the same rotation, different rotation axes can be chosen and ordered, and rotation angles 2π apart on the same axis correspond to the same rotation. Therefore, Euler angle representation is not unique for the same rotation. Using quaternions, a rotation can be represented using either of two unit quaternions with opposite signs. If this application always chooses the quaternion whose first element is non-negative, it achieves a unique representation of the rotation. This is the rotation representation adopted in this application. By concatenating the quaternions of all J nodes in the skeleton, this application obtains a long vector. h=4J is used to represent a raw motion capture frame.

[0063] To reduce spatial redundancy in motion capture frames, this application learns a concise (encoded) pose dictionary in an unsupervised manner. For a motion capture frame, this application finds the dictionary entry that most closely resembles the pose in that frame and represents the frame using the corresponding dictionary index. Essentially, unsupervised dictionary learning is the process of clustering unlabeled training samples and computing a representative (or head) for each cluster. While various techniques (e.g., K-means clustering and SOM) have been used to learn dictionaries of poses or motion segments, this application chooses to use the unsupervised AAE technique.

[0064] The extended AAE network excels in separating category and style information and has been successfully applied to unsupervised clustering. This application selects the extended AAE network for pose clustering because it automatically learns a set of optimized cluster heads that are highly representative and well-spaced, while also reducing the dimensionality of the data representation.

[0065] Extended AAE architecture for unsupervised clustering, such as Figure 4As shown in (a), assuming the dataset contains m classes of samples, this application reduces the dimensionality of each sample to n. For input x, the autoencoder predicts an m-dimensional discrete class variable y and an n-dimensional continuous latent variable z based on q(y,z|x) implemented by the encoder. Two adversarial networks apply a categorical distribution Cat(y) and a Gaussian distribution to the two latent variables, respectively. The autoencoder and two adversarial networks were jointly trained using stochastic gradient descent (SGD). Training consisted of two phases: reconstruction and regularization. Figure 4 In (b), the top and bottom views show two-dimensional projections of the cluster head and the encoded training pose samples, respectively. This application... Figure 4 In (b), it was observed that the encoded poses showed a clear clustering tendency, with cluster heads being far apart from each other.

[0066] Specifically, this application reduces spatial redundancy in the following way. First, this application uses a training pose set T = {t1, t2, ..., t...} u The unsupervised clustering AAE model is trained. In this application, T is set as the set of all motion capture sequences in the database that contain poses. During training, m cluster heads are automatically learned in the dimensionality-reduced space. These m cluster heads form the encoded pose dictionary U = {u1, u2, ..., u...} of this application. m For each pair of items in the dictionary, up and uq, this application computes their Euclidean distance d. p，q =|u p -u q |2, to obtain the pose distance matrix D=[d p，q ] m×m This matrix will be used in the time alignment of the symbolized motion capture sequences. After training the AAE model, this application first symbolizes all motion capture sequences in the database offline. Then, for any given query sequence, this application also symbolizes it online using this AAE model. Specifically, let a motion capture sequence be denoted as... For x i 1≤c≤C, the trained AAE model measures its categorical variable, i.e., the dictionary index y. c Therefore, x is converted into the symbol sequence Y = [y1, y2, ..., y]. C ].

[0067] 1.3 Time Redundancy Reduction

[0068] In the symbol sequence after spatial redundancy reduction, this application frequently observes segments with consecutive occurrences of the same symbol, such as ①①①. This is largely due to the dense temporal sampling of the motion capture system. To eliminate temporal redundancy, this application retains only one symbol for each such segment. After this simplification, this application continues to search for consecutively recurring symbol pattern segments, such as ①②③①②③. Such segments generally correspond to some kind of cyclical motion, where the signal pattern corresponds to a semantic unit (e.g., a step cycle in walking). For each segment of this type, this application retains only one instance of the symbol pattern to reduce temporal redundancy. For consecutively occurring segments of the same symbol, they can be easily detected by performing a single traversal of the symbol sequence. For recurring symbol pattern segments, this application uses a suffix array technique for detection.

[0069] exist Figure 4 In (c), this application provides a simple example to demonstrate the effects of reducing spatial redundancy and temporal redundancy. Figure 5 In this application, a real-world example of motion capture sequence symbolization is provided. In this example, the algorithm of this application significantly simplifies a 341-frame original cartwheel motion capture sequence into a sequence of 8 symbols. Figure 5 In the diagram, the first line is a sample of frames from the original motion capture data, with each frame labeled with a corresponding dictionary index; the second line is the final symbolic sequence, represented by the dictionary index and the pose reconstructed from the corresponding dictionary entry; the third line indicates that in the final symbolic sequence, a symbol typically corresponds to a longer segment of the original motion capture sequence (43 frames in this example).

[0070] 2. Step time alignment

[0071] All motion capture sequences in the database are concatenated into a long sequence. Will Symbolization And all G i By concatenating 1≤i≤b, this application obtains a symbolic representation of the entire database. Furthermore, this application also stores the index position correspondence between the two sequences S and M to facilitate fast cross-location of corresponding subsequences between them. Given a query sequence, Q = [f1, f2, ..., f... q It was first symbolized as This application proposes a step-time alignment method to align E to S, thereby extracting the subsequence in S that is most similar to E. Accordingly, this application obtains the subsequence in M ​​that is most similar to Q.

[0072] Specifically, this application uses dynamic programming to compute the weight matrix. The formula is as follows:

[0073]

[0074] in,

[0075]

[0076] w represents the step size, and a local search window of (w+1)×(w+1) is defined, with p = max(xm, yn) as the penalty factor. During the iterative calculation above, for each W(x, y), x∈(1, X], its corresponding optimal (m, n) value (called the backward pointer) is also recorded. Thus, each element W(X, y) in the last row of matrix W represents the distance between E and the best matching subsequence of E (denoted as V) ending at the y-th symbol in S. Following the backward pointers of the matrix elements, this application obtains an optimal alignment path traversing all rows of matrix W. The column numbers of the matrix traversed by this path are given by V. The motion capture subsequence in M ​​corresponding to V is denoted as T, and this application defines d(Q, T) = W(X, y).

[0077] To retrieve subsequences in S similar to E, this application first locates all elements in the last row of W with weights below a threshold Ud. This application traverses these elements in ascending order of weight, and starting from each element, backtracks to the first row of W to obtain a subsequence similar to E. During the search, if a similar subsequence overlaps with any valid similar subsequence already obtained, it is set to invalid. If an identified similar subsequence crosses the boundary between two adjacent action symbol sequences, it can be retained or discarded according to user needs. Once similar subsequences in S are located, the corresponding similar subsequences in M ​​can be easily found based on the pre-stored correspondence between the index positions of S and M.

[0078] DTW and its variants have been used by other motion capture data retrieval algorithms. When calculating W(x,y), they compare at most the values ​​of the three nearest neighbors already calculated in W. In motion capture sequences, computational errors or small local motions mixed in with the main motion often cause the appearance of singular frames. Sometimes, these singular frame fragments can overturn the matching between sequences that were originally quite similar. The STA method proposed in this application performs an optimal search within a local (w+1)×(w+1) window when calculating W(x,y), x∈(1,X], as shown in Equation 1. In this way, it can bypass frames with poor local matching (up to w-1 frames), thus being robust to singular frames that appear intermittently in globally similar sequences or subsequences.

[0079] Figure 6A simple example of time alignment based on STA and DTW is shown. A query sequence after spatial and temporal redundancy removal and a sequence from a database are displayed on the left and bottom, respectively. The poses marked by boxes correspond to singular frames in the sequence. Figure 6 For ease of illustration, this application presents pairwise pose distance matrices instead of weight matrices. Lighter shading indicates smaller distances. Paths 1 and 3 show the alignment results between the query sequence obtained using STA and the two sub-database sequences. Path 2 shows the alignment results between the query sequence obtained using DTW and the entire database sequence. For this example, this application uses w=3. From Figure 6 In this application, we observe that STA successfully traverses singular frames in both the query and database sequences, while DTW does not. Therefore, the total weight of path 2 is significantly higher than the other two paths. In other words, STA-based alignment more effectively captures the partial similarity between the database and query sequences.

[0080] 3. Experiment

[0081] 3.1 Dataset

[0082] All motion capture sequences in the first dataset came from the HDM05 database

[36] . Specifically, it consisted of motion capture sequences for ten types of human movements, including cartwheel, elbow-to-knee, hop, jog, jumping jack, kick, shuffle, squat, throw-sitting, and walk. Each movement type contained 20 sequences. The dataset contained 200 motion capture sequences and 58,781 frames. The movements were performed by 5 actors, and each movement differed slightly in terms of the number of loops, starting hand / foot position, and ending position.

[0083] The second dataset consists of data captured using the Vicon optical motion capture system in this application. It contains 45 long motion capture sequences across 12 motion types, filmed by 9 actors. Each long motion capture sequence is a mixed sequence of an indefinite number of motion types, with one motion type potentially appearing multiple times for varying durations. When performing a motion sequence, the actors performed a randomly selected subset of motion types in a randomly determined order. Table 1 shows the 12 motion types and their total frequency across the 45 motion capture sequences. On average, each sequence contains 8 motion types. This dataset contains a total of 103,849 frames.

[0084] Table 1 shows the types of motions and their frequency in the second dataset.

[0085]

[0086] 3.2 Evaluation Indicators

[0087] The algorithm performance evaluation metrics used in this application include top n precision (P@n), precision-recall curve (PR curve), and confusion matrix. For each query, the proportion of relevant samples in the result set gives the precision, and the proportion of relevant samples in the result set to all relevant samples in the database gives the recall. If the size of the result set is n, the corresponding precision is top n precision. By changing n, this application obtains the precision-recall curve for this query. When n = N (N is the total number of motion capture sequences in the database), the average precision (AP) of this query is calculated as follows:

[0088]

[0089] Where R is the total number of relevant samples in the database; if the j-th sequence in the result set is relevant, then Ij = 1, otherwise Ij = 0; Rj is the number of relevant samples in the first j samples after sorting the result set.

[0090] By averaging the statistics of all queries for a given sport type, this application obtains the P@n, PR, and MAP statistics for that type. Furthermore, this application calculates a confusion matrix, a square matrix where each element represents the frequency with which a sample from one sport category is mistakenly identified as related to a query sample from another sport category during retrieval.

[0091] 3.3 Algorithm Performance Evaluation

[0092] This application will evaluate the proposed algorithm and other related unsupervised learning-based motion capture retrieval algorithms on two datasets. To this end, two experiments were conducted, details of which are as follows.

[0093] Whole-sequence retrieval. This experiment was conducted on the first dataset. This application compares its pose encoding and step-time alignment-based method (PESTA) with SOM-based, weighted graph-based, and motion signature-based methods (MS). When testing an algorithm, this application uses each motion capture sequence in the dataset as a query to retrieve similar sequences from the database. This application uses performance statistics collected from all searches to obtain P@n (n = 5, 10, 15, 20) statistics and precision-recall curves, and uses the top 20 search results from each search to obtain a confusion matrix. Figure 7 , Figure 8 and Figure 9 The P@n statistics, PR curves, and confusion matrices for several algorithms are shown, demonstrating that PESTA significantly outperforms the other algorithms. In this experiment, w=3 was used in this application.

[0094] Sub-sequence retrieval. This experiment was conducted on a second dataset. This application compares PESTA with SOM

[31] , which is most relevant to the sub-sequence retrieval method of this application, see Section 2.2 for details. For each of the 12 motion types, this application manually extracted 10 examples from long motion capture sequences, which together formed the query set. Each query was used for retrieval once. A returned sub-sequence may contain frames from multiple motion types, but is classified into the type with the most frames. A query hit is counted if and only if the returned sub-sequence and the query have the same motion type. Based on the performance statistics collected from all retrievals, this application obtained P@n (n=2, 5, 10) statistics and PR curves, and obtained a confusion matrix using the top 10 retrieval results in each retrieval. Figure 10 and Figure 11 The P@n statistics, PR curves, and confusion matrices of SOM and PESTA on the query set are shown, indicating that PESTA is significantly better than SOM in terms of performance.

[0095] This application proposes a content-based motion capture data retrieval scheme. It learns a pose dictionary and converts each motion capture sequence into a symbol sequence, i.e., a string of dictionary indices. Then, it performs temporal alignment between the query and database sequences to retrieve similar subsequences. For pose dictionary learning, this application employs an adversarial autoencoder technique, which effectively separates motion category and style information. For temporal alignment, this application proposes a step-search approach that effectively bypasses singular frames in the motion capture sequence. Experimental results demonstrate that the scheme has excellent performance.

[0096] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0098] Example 2

[0099] Figure 12 A motion capture data retrieval device 500 according to a first aspect of this embodiment is shown, which corresponds to the method described according to the first aspect of Embodiment 1. Reference Figure 12 As shown, the device 500 includes:

[0100] AAE module 510 is used to learn the encoded pose dictionary using an unsupervised adversarial autoencoder.

[0101] The motion capture sequence module 520 is used to represent each motion capture sequence according to the learned pose dictionary to obtain a database;

[0102] The query sequence module 530 is used to obtain a symbol sequence based on the learned pose dictionary representation query sequence;

[0103] The step-time alignment and retrieval module 540 is used to align the symbol sequence with each subsequence in the database through step-time alignment, and retrieve the most matching subsequence from the database.

[0104] According to another aspect of the embodiments of this disclosure, such as Figure 13 As shown, a motion capture data retrieval device 700 is also provided, comprising:

[0105] First processor 710; and

[0106] A first memory 720, connected to the first processor, is used to provide the first processor with instructions to perform the following processing steps:

[0107] Use an unsupervised adversarial autoencoder to learn the encoded pose dictionary;

[0108] The database is obtained by representing each motion capture sequence based on the learned pose dictionary;

[0109] Based on the learned pose dictionary representation of the query sequence, a symbol sequence is obtained;

[0110] The symbol sequence is aligned with each subsequence in the database by step-time alignment, and the most matching subsequence is retrieved from the database.

[0111] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0112] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0117] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A motion capture data retrieval method, characterized by, comprising: learning an encoded pose dictionary using unsupervised adversarial autoencoder; representing each motion capture sequence according to the learned pose dictionary, obtaining a database; representing a query sequence according to the learned pose dictionary, obtaining a symbol sequence; aligning the symbol sequence with each sub-sequence in the database by cross-step time alignment, and retrieving the most matching sub-sequence from the database.

2. The method of claim 1, wherein the representing each motion capture sequence according to the learned pose dictionary comprises: for a motion capture frame, finding the dictionary item in the pose dictionary that is most similar to the pose in this frame, and representing this frame using the corresponding dictionary index.

3. The method of claim 2, wherein the unsupervised adversarial autoencoder predicts an m-dimensional discrete class variable y and an n-dimensional continuous hidden variable z according to q(y, z|x) implemented by the autoencoder, two adversarial networks impose class distribution and Gaussian distribution on the two hidden variables respectively, and the autoencoder and the two adversarial networks are jointly trained using stochastic gradient descent method, and the training is performed in two stages of reconstruction and regularization; wherein x is the input, m is the number of sample categories included in the data set, and n is the dimension of each sample.

4. The method of claim 3, wherein the learning an encoded pose dictionary using unsupervised adversarial autoencoder comprises: training the unsupervised clustering AAE model using a training pose set, and setting the training pose set as the set of poses contained in all motion capture sequences in the database; the training process automatically learns m cluster heads in the reduced space, the m cluster heads form the encoded pose dictionary, and the Euclidean distance between each two items in the pose dictionary is calculated to obtain a pose distance matrix.

5. The method of claim 4, wherein the representing each motion capture sequence according to the learned pose dictionary, obtaining a database comprises: symbolizing all motion capture sequences offline to obtain the database.

6. The method of claim 5, wherein the representing a query sequence according to the learned pose dictionary, obtaining a symbol sequence comprises: for any given query sequence, symbolizing online according to the AAE model to obtain a symbol sequence.

7. The method of claim 5 or 6, wherein after the representing a query sequence according to the learned pose dictionary, obtaining a symbol sequence, further comprising: in the symbol sequence, only keeping one symbol for a segment in which the same symbol appears continuously; and continuing to search for a continuously repeated symbol pattern segment, and only keeping one instance of the symbol pattern.

8. A storage medium, characterized by The storage medium includes a stored program, wherein the program is executed by a processor when the program is run to perform the method of any one of claims 1 to 7.

9. An action capture data retrieval apparatus, characterized by comprising: an AAE module configured to learn an encoded pose dictionary using an unsupervised adversarial autoencoder; a motion capture sequence module configured to represent each motion capture sequence according to the learned pose dictionary, obtaining a database; a query sequence module configured to query a sequence according to the learned pose dictionary representation, to obtain a symbol sequence; a cross-step time alignment and retrieval module configured to align the symbol sequence with each sub-sequence in a database by cross-step time alignment, and retrieve the most matching sub-sequence from the database.

10. An action capture data retrieval apparatus, characterized by comprise: a first processor; and a first memory connected with the first processor, configured to provide the first processor with instructions for processing the following processing steps: learning an encoded pose dictionary using unsupervised adversarial autoencoder; representing each motion capture sequence according to the learned pose dictionary representation, to obtain a database; querying a sequence according to the learned pose dictionary representation, to obtain a symbol sequence; aligning the symbol sequence with each sub-sequence in a database by cross-step time alignment, and retrieving the most matching sub-sequence from the database.

Images