Sequence generation methods and apparatus, storage media, and computer program products
By generating differentiated transition sequences and performing similarity deduplication in the digital human video generation system, the problems of abrupt pose changes and mechanical repetition during action segment splicing are solved, resulting in more natural and realistic video performance.
Patent Information
- Application Number
- CN202610976144.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-01
- Publication Date
- 2026-11-17
AI Technical Summary
Existing digital human video generation systems are prone to sudden changes in posture or emotion when splicing action segments, resulting in poor action continuity and mechanical repetition, which affects the naturalness and realism of the video.
By acquiring adjacent action segments in the temporal domain, a transition sequence is generated. Perturbations are applied during the generation process to produce differentiated transition sequences. At the same time, deduplication is performed based on similarity metrics to ensure the coherence and naturalness between action segments.
It improves the continuity between action segments, avoids mechanical repetition, enhances the visual presentation of the video, and makes it more natural and realistic.
Smart Images

Figure CN123123659A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a sequence generation method and apparatus, storage medium, and computer program product. Background Technology
[0002] Current digital human video generation systems widely employ motion assets libraries to drive the performance of digital human characters. These assets typically originate from videos recorded by real people or motion capture data, and are used in conjunction with speech recognition and semantic analysis for matching. Motion asset libraries are usually stored in segmented form, and complete video motion sequences are assembled through simple splicing. To improve matching accuracy, some systems also incorporate text semantic vector and motion tag matching algorithms.
[0003] Currently, when splicing multiple action clips into a complete video, there may be issues such as sudden changes in the digital human's posture or emotions, and overly rigid transitions between action clips. This may result in a lack of continuity in real-life actions, as well as mechanical repetition in the same or similar action clips, leading to an unnatural and unrealistic visual presentation of the entire video. Summary of the Invention
[0004] This application provides a sequence generation method and apparatus, a storage medium, and a computer program product.
[0005] This application provides a sequence generation method, including: Obtain the first and second action segments that are adjacent to each other in the temporal domain; At the junction between the first action segment and the second action segment, a transition sequence is generated, and one or more of the following are performed: A perturbation is applied during the generation of the transition sequence to generate a transition sequence at the junction that differs from other transition sequences. Based on similarity metrics, duplicates are removed from the transition sequences already generated at the connection points and from other transition sequences.
[0006] This application provides a sequence generation apparatus, including: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute one or more computer programs stored in the memory to implement the sequence generation method.
[0007] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a sequence generation method.
[0008] This application provides a computer program product, including a computer program that, when executed by a processor, implements a sequence generation method.
[0009] This application provides a sequence generation method and apparatus, storage medium, and computer program product. The method includes: acquiring a first action segment and a second action segment that are sequentially adjacent in the temporal domain; generating a transition sequence at the connection point between the first action segment and the second action segment; and performing one or more of the following: applying a perturbation during the generation of the transition sequence to produce a transition sequence at the connection point that differs from other transition sequences; and deduplicating the already generated transition sequence at the connection point and other transition sequences based on a similarity metric. The technical solution provided by this application generates a transition sequence at the connection point between action segments. On the one hand, it allows for selective perturbation during the generation of the transition sequence; on the other hand, it allows for deduplication of the transition sequence based on a similarity metric. This ensures the continuity of the action segments while avoiding mechanical repetition, improving the visual presentation of the generated video and making it more natural and realistic. Attached Figure Description
[0010] Figure 1 A flowchart illustrating a sequence generation method provided in an embodiment of this application; Figure 2 A schematic diagram of an exemplary framework for generating a transition sequence is provided for embodiments of this application; Figure 3 A schematic diagram of an exemplary action segment provided for an embodiment of this application. Figure 1 ; Figure 4 A schematic diagram of an exemplary action segment provided for an embodiment of this application. Figure 2 ; Figure 5 A schematic diagram of an exemplary transition sequence provided for an embodiment of this application; Figure 6 A schematic diagram of the structure of a sequence generation device provided in this application embodiment. Figure 1 ; Figure 7 A schematic diagram of the structure of a sequence generation device provided in this application embodiment. Figure 2 . Detailed Implementation
[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0012] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below through embodiments and in conjunction with the accompanying drawings. The embodiments below can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0013] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.
[0014] This application addresses the problem in existing technologies where sudden changes in posture or emotion occur during the splicing of multiple action segments, making it difficult to guarantee the naturalness and coherence of digital human performance. It provides a sequence generation method that analyzes the user-input text content to obtain multiple semantic vectors. Based on these semantic vectors, action segments are matched against a digital human action library. When the boundary frame action connection score of two action segments A and B is less than a threshold, a transition sequence C is generated. A, C, and B are then merged to output a complete action sequence, ensuring the naturalness and coherence of the digital human's performance. The key point is: 1. Using a multi-layer Transformer encoder, a time series H is captured based on the high-dimensional features of boundary frames in the action segment and positional encoding. This time series represents the temporal context of the action segment. A contextual representation of conditional control is obtained based on the time series H and the overall semantic vector. ,based on Decode to obtain the transition sequence.
[0015] 2. When multiple transition sequences are generated corresponding to multiple preceding and following action sequences, affine normalization and encoding are performed on each transition sequence to obtain a sequence embedding. The sequence embedding and the semantic vectors of the preceding and following action segments are concatenated to obtain the corresponding transition embedding.
[0016] Before generating a new transition sequence, the nearest K transition sequences are found, and the avoidance direction is calculated based on the corresponding K transition embeddings. Orthogonal noise is obtained based on the avoidance direction and added to the overall semantic vector. The context representation H' of condition control is obtained based on the time series H and the new overall semantic vector. The transition sequence is obtained by decoding based on H', thus realizing the differentiation of the generated transition sequence.
[0017] 3. After generating multiple transition sequences corresponding to multiple preceding and following action sequences, the similarity between the transition sequences is calculated based on the transition embedding of the transition sequences. The multiple transition sequences are sorted by quality score, and similarity is calculated from high to low based on the sorting. Sequences with lower quality are discarded when the similarity exceeds a threshold. A temporal misalignment penalty is added to the similarity. Based on the new similarity and quality score, a subset selection process (DPP) is used to select back some of the discarded sequences, resulting in the final retained transition sequences, which are of good quality and exhibit differentiation.
[0018] Based on the above, the sequence generation method of this application is described in detail below.
[0019] The sequence generation method in this application is implemented using a sequence generation device. The sequence generation device can be any electronic device, and this application does not limit its use. Figure 1 This is a flowchart illustrating a sequence generation method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the sequence generation method mainly includes the following steps: S101. Obtain the first and second action segments that are adjacent in the time domain.
[0020] In the embodiments of this application, the sequence generation device may first acquire a first action segment and a second action segment that are adjacent in the time domain.
[0021] It should be noted that, in the embodiments of this application, the first action segment and the second action segment can be two action segments whose actions are not continuous and require a transition sequence to be inserted between them (see the aforementioned action connection scoring mechanism). The first action segment and the second action segment are adjacent in the temporal domain. Specifically, the first action segment and the second action segment can be digital human action video segments, or they can be two real human action video segments captured. This application embodiment does not limit the specific actions.
[0022] It should be noted that, in the embodiments of this application, the connection point between the first action segment and the second action segment can be the boundary position of the first action segment and the second action segment in terms of timing, or it can be the position where transition filling is required between the two. The embodiments of this application do not limit this.
[0023] S102. Generate a transition sequence at the connection point between the first action segment and the second action segment, and perform one or more of the following: A perturbation is applied during the generation of the transition sequence to generate a transition sequence at the junction that differs from other transition sequences. Based on similarity metrics, duplicates are removed from the already generated transition sequences and other transition sequences at the connection points.
[0024] In the embodiments of this application, after obtaining the connection point between the first action segment and the second action segment, the sequence generation device can generate a transition sequence at the connection point. The generation method of the transition sequence will be explained below.
[0025] Combination Figure 2 The complete workflow framework for generating the transition sequence in this application is as follows: 1. The user enters text content; 2. Semantic analysis: Semantic analysis is performed on the text content to obtain multiple corresponding semantic vectors. Based on each semantic vector, action segments are matched in the digital human action database. The action segments are ordered based on the position of the semantic vectors before and after the text content. 3. Obtain two action segments A and B (corresponding to the aforementioned step S101, where action segment A corresponds to the first action segment and action segment B corresponds to the second action segment). 4. Extract the boundary frames of each (N frames at the end of A, N frames at the beginning of B). 5. Calculate the score for the connection between movements; 6. If the action connection score is less than the threshold, then the generation of a transition sequence is triggered; 7. Use an action generation network (such as a conditional Transformer) to generate the transition sequence C; 8. Combine the outputs of A+C+B.
[0026] Taking the text content as: "...supermarket price X yuan per bottle"..., and matching the product display action as an example, the boundary frame of the preceding action segment is as follows: Figure 3 As shown, the boundary frames of subsequent action segments are as follows: Figure 4 As shown, by generating Figure 5 The transition sequence shown can be spliced together. Figure 3 and Figure 4 Between these elements, a coherent and complete video is generated.
[0027] The relevant explanations are as follows: Note 1: The motion posture scoring function is defined as follows: Let the last N frames of the preceding action segment A be the boundary frames. The initial N frames of the subsequent action segment B are boundary frames. N is a positive integer, and its value is not limited. The key joint vector for each frame's motion: , Where X and Y represent the two-dimensional coordinates (x, y) of each joint in the human skeleton, the motion connection score is represented by the average angle difference between frames. The average angle difference between frames can be defined as: ,in Let represent the orientation vector of the j-th joint in the i-th frame of the last N frames of the preceding action segment A. This represents the orientation vector of the j-th joint in the i-th frame of the last N frames of the preceding action segment B.
[0028] The orientation vector is obtained as follows: Get the joint node coordinates (x, y) Select parent-child joint pairs and calculate the connection direction vector.
[0029] For example, the direction of the upper arm = shoulder joint coordinates – elbow joint coordinates; Forearm direction = elbow joint coordinates – wrist coordinates.
[0030] Finally, the direction vector is normalized to obtain the unit orientation vector.
[0031] Note 2: The action generation network structure is as follows: Input features: spliced boundary frame sequence With semantic vectors The semantic vector can be used or not.
[0032] Feature Encoder: Uses positional encoding to encode temporal order, mapping the pose vector of each frame (i.e., the set of two-dimensional coordinates of all joints in a frame) into high-dimensional features through a fully connected network. .
[0033] Encoder-decoder structure: A multi-layer Transformer encoder is used to extract temporal dependencies and obtain the time series H. (1) The time series H is the temporal dependency representation obtained after passing through the Transformer encoder. It captures the contextual relationships of action segments in the time dimension, such as the motion trend between consecutive frames. In the decoder, H serves as the input context, guiding the generation of transition sequences that conform to dynamic patterns.
[0034] PE (Positional Encoding): Positional encoding is used to introduce temporal order information into the Transformer (because self-attention itself is not sensitive to sequence position). It typically uses sine / cosine functions or learnable positional embeddings.
[0035] Conditional control layer: This layer integrates semantic vectors into the time series data. It can be implemented using a self-attention interaction layer or a gating module. (2) Semantic vectors originate not from a single frame, but from the overall semantic encoding of the user-input text, representing the semantic intent of the action (such as "waving" or "nodding"). Their purpose is to provide additional semantic consistency constraints for transition sequences.
[0036] The reason for fusing the time series H is that the time series H only contains motion information of the boundary frames and lacks "intention". By fusing the semantic vector with H, the generated transition actions can not only be smooth and natural, but also conform to the target semantics (for example, the transition frames of a waving action should maintain the upward trend of the hand).
[0037] Action Decoder: The decoder decodes the context representation into a continuous sequence of actions. (3) Output layer: Generates intermediate action sequences Each frame is a two-dimensional coordinate vector of a key point.
[0038] Explanation 3: Action composition function ,in: A: Action segment A; C: Generated transition sequence (length adaptive); B: Action segment B.
[0039] Considering that a complete video requires splicing multiple actions, it's inevitable that similar actions will result in repetitive transitions, leading to mechanical characters and a lack of emotional expression. For example, a transition sequence C is generated between action segments A and B, and a transition sequence F is generated between action segments D and E. If the last frame of A is similar to the last frame of D, and the first frame of B is similar to the first frame of E, then the generated C and F may be similar. To avoid this, a transition frame exclusion logic is adopted to prevent highly consistent generated transition frames. The approach is a two-layer "pre- and post-"similar" removal: parallel similarity avoidance before generation and global rearrangement after generation, combined with a global memory and similarity metric to achieve exclusive control across transition points.
[0040] Based on this, in the embodiments of this application, the sequence generation device generates a transition sequence at the junction between the first action segment and the second action segment, and performs one or more of the following two sequence processing schemes: the first sequence processing scheme is parallel avoidance before generation, and the second sequence processing scheme is global rearrangement after generation. The two sequence processing schemes at the junction are explained below.
[0041] In the embodiments of this application, the first sequence processing scheme is parallel avoidance of homogeneity before generation, which aims to distinguish "potentially homogeneous" junction points before sampling. Specifically, it includes: applying a perturbation when performing transition sequence generation to generate a transition sequence at the junction point that differs from other transition sequences.
[0042] In embodiments of this application, the sequence generation device applies a perturbation when performing transition sequence generation, causing a transition sequence at the junction point that differs from other transition sequences. This includes: extracting the boundary frames of the first action segment and the second action segment at the junction point; querying K transition sequences from a transition sequence library that are similar to the sequence boundary and the boundary frames; where K is a positive integer; and the transition sequence library includes transition sequences between different action segments; determining the avoidance direction based on the K transition sequences, and projecting the noise set for the junction point onto the orthogonal complement space of the avoidance direction to obtain orthogonal noise; fusing the initial semantic vectors corresponding to the first action segment and the second action segment with the orthogonal noise to obtain a target semantic vector; and generating a differentiated transition sequence at the junction point based on the target semantic vector and the boundary frames.
[0043] It should be noted that, in the embodiments of this application, the boundary frame at the junction of the first action segment and the second action segment may include the last N frames of the first action segment and the first N frames of the second action segment, where N is a positive integer and the value of N is not limited.
[0044] It should be noted that, in the embodiments of this application, the transition sequence library can be described as a global transition memory (GTM), which can store all historical and current batches of "passed" transition sequences and their embeddings, semantic tags, statistical features, etc. The sequence generation device can query K transition sequences in the transition sequence library that are similar to the boundary and boundary frame, thereby determining the avoidance direction.
[0045] In embodiments of this application, determining the avoidance direction based on K transition sequences includes: obtaining the transition embedding of each transition sequence in the K transition sequences; and determining the mean of the K transition embeddings corresponding to the K transition sequences as the avoidance direction.
[0046] In the embodiments of this application, the transition embedding of the K-segment transition sequence is determined in the following manner: for each transition sequence in the K-segment transition sequence, affine normalization is performed, and the first and second derivatives are calculated based on the normalization results; for each transition sequence in the K-segment transition sequence, the corresponding normalization results, first and second derivatives are encoded to obtain the corresponding sequence embedding; for each transition sequence in the K-segment transition sequence, the semantic vectors of the corresponding preceding and following action segments are concatenated with the sequence embedding to obtain the corresponding transition embedding.
[0047] It should be noted that, in the embodiments of this application, the sequence generation device can perform key representation and measurement on K transition sequences, as follows: Preprocessing and alignment: For each transition sequence Perform affine normalization (descale, translate, rotate, and align to the pelvic coordinate system) and calculate the first / second derivatives (velocity V, acceleration A): (4) (5) (6) Transitional embedding: The sequence embedding is obtained using a lightweight GCN / Transformer encoder f(.): (7) Then, by concatenating the semantic vectors v of the preceding and following sequences corresponding to the transition sequence, we obtain the transition embedding: (8) Therefore, the avoidance direction is determined based on the transition embedding of the K-segment transition sequence: (9) in, This represents the transition embedding of the nth transition sequence in the K-segment transition sequence.
[0048] It should be noted that, in the embodiments of this application, after determining the direction of avoidance, a small amount of orthogonal noise set for the connection point can be added at the condition control layer. ,in These are the initial semantic vectors corresponding to the first and second action segments. The target semantic vector after noise fusion: (10) It should be noted that, in the embodiments of this application, noise can be set for all connection points, and the noise set for similar connection points can be different. Specifically, connection points can be clustered, that is, embedding and similarity clustering can be performed on the "boundary frames" of all connection points. Thus, different random seeds can be set for connection points within the same cluster to determine different noise, thereby reducing homogeneous output.
[0049] It should be noted that, in the embodiments of this application, the sequence generation device generates a differentiated transition sequence at the junction between the first action segment and the second action segment based on the target semantic vector and the boundary frame. The specific steps for generating the transition sequence are similar to the processing procedures represented by the aforementioned formulas (1)-(3). The difference is that, in order to generate a differentiated transition sequence when generating a transition sequence between similar segments, parameter replacement is performed when the semantic vector is fused into the time series of the conditional control layer. The semantic vector used is the target semantic vector after noise fusion. : (11) Other related processing procedures are the same and will not be repeated here.
[0050] In the embodiments of this application, the second sequence processing scheme is global rearrangement after generation. Its purpose is to remove duplicates of similar transition sequences between the transition sequence generated for the connection point and other action segments, and finally retain only the parts of the similar transition sequences that are "well covered and highly different from each other". Specifically, it includes: removing duplicates of the transition sequence generated at the connection point and other transition sequences based on similarity measurement.
[0051] In embodiments of this application, the sequence generation device performs deduplication on the transition sequence already generated at the connection point and other transition sequences based on a similarity metric, including: calculating a quality score for each transition sequence among multiple transition sequences; wherein, the multiple transition sequences include the transition sequence already generated at the connection point and transition sequences between other action segments; performing sequence retention and discarding by traversing multiple transition sequences from high to low according to the quality score; if the similarity between the current transition sequence and the retained transition sequence is lower than a threshold, the current transition sequence is retained; if the similarity between the current transition sequence and the retained transition sequence is higher than the threshold, the current transition sequence is discarded.
[0052] In embodiments of this application, the similarity between the current transition sequence and the retained transition sequence is determined based on first information, or based on the first information and the time misalignment penalty of the current transition sequence. The first piece of information includes one or more of the following determinations: Cosine similarity of transition embedding; Dynamic temporal warping similarity of attitude trajectories; Velocity spectrum difference value.
[0053] It should be noted that, in the embodiments of this application, the multiple transition sequences include the transition sequences already generated at the aforementioned connection points. These transition sequences can be generated at the connection points between the first action segment and the second action segment according to the aforementioned first sequence processing scheme, i.e., by introducing noise reduction in the same direction. Alternatively, they can be generated at the connection points between the first action segment and the second action segment according to the aforementioned formulas (1)-(3), i.e., without introducing noise reduction in the same direction. This application embodiment does not impose any limitations on these aspects. Furthermore, the multiple transition sequences also include transition sequences generated at connection points between other action segments. The specific number of sequences and their content are not limited in this application embodiment.
[0054] It should be noted that, in the embodiments of this application, non-maximum suppression (NMS) can be used to score multiple transition sequences according to their quality. (Smoothness / scoring function / physical validity) sorting; if similarity exceeds a threshold Then, lower-quality transition sequences are suppressed (discarded). Indicates the retained transition sequence: (12) It should be noted that, in the embodiments of this application, for any two transition sequences The similarity can be calculated using the following formula: (13) in, These are the transition embeddings of the two transition sequences (the calculation method for transition embeddings is described in the aforementioned related content). The results are the affine normalization results for the two transition sequences (see the related content above for the affine normalization method). Cosine similarity representing transition embedding. Represents the dynamic time warping similarity of the attitude trajectory (normalized to [0,1]). Indicates the velocity spectrum difference value (e.g., KL divergence of power spectrum), weight These are debugging values that can be generated through learning.
[0055] It should be noted that, in the embodiments of this application, time-shifted similarity removal processing can also be performed, that is, the aforementioned similarity is replaced by adding a time misalignment penalty to the similarity, thereby avoiding the same shape from passing the test simply by translation: (14) in, Right now Time misalignment penalty This represents the penalty weighting coefficient. This indicates shifting sequence i. The result after the frame, express The minimum DTW distance between it and its time-shifted version.
[0056] It should be noted that in the embodiments of this application, formula (12) or formula (13) can be used to calculate sequence similarity according to actual needs and application scenarios. This application embodiment does not limit the use of formula (12) or formula (13).
[0057] In embodiments of this application, the sequence generation device can also perform diversity measurement on discarded transition sequences by employing a DPP subset selection mechanism to recall partially differentiated transition sequences from the discarded transition sequences. The kernel matrix for DPP subset selection is constructed as follows: (15) (16) (17) Where L represents the kernel matrix, Q represents the quality diagonal matrix, and S represents the similarity matrix. By maximizing Select a subset Y to maximize diversity while ensuring quality. The subsequence was selected back. This represents the determinant of L corresponding to subset Y.
[0058] In the embodiments of this application, the above-mentioned scheme for generating suitable transition sequences between action segments is based on the fact that after the user inputs text content, semantic analysis is performed on the text content to obtain multiple corresponding semantic vectors, and suitable action segments are matched in the digital human action library according to each semantic vector. However, in practical applications, during the semantic analysis stage, when matching action segments in the digital human action library according to the semantic vectors, the semantic vectors may not match the action segments in the digital human action library. To address this situation, this application also proposes a semantic missing trigger dynamic action synthesis mechanism, the specific process of which is as follows: 1. During the semantic analysis phase, if the matching score between the semantic vector and the action tag of the action segment in the digital human action library is insufficient (e.g., <threshold), it is judged as "semantic coverage failure"; 2. Trigger the semantic vector-based action generation network to construct a complete action fragment based on the contextual semantics without relying on existing action fragments; Let the semantic vector be This represents the semantic feature vector of the current statement or segment. The goal is to generate action fragments (action sequences). Each frame , representing the two-dimensional positions of k joints.
[0059] It should be noted that, in the embodiments of this application, the action sequence is generated using a diffusion-based model: Initialize from noise distribution:
[0060] Iterative denoising generation:
[0061] (18) in For a conditional denoising network, the inputs are the current frame, time step, and semantic vector. This indicates the fault tolerance range, which can be set according to requirements.
[0062] Finally, the action sequence is obtained. .
[0063] It should be noted that, in the embodiments of this application, the generated action segments can be used independently or, similar to transition sequences, can be connected between related action segments.
[0064] This application also provides a sequence generation apparatus. Figure 6 A schematic diagram of the structure of a sequence generation device provided in this application embodiment. Figure 1 .like Figure 6 As shown, in an embodiment of this application, the sequence generation device 1 includes: The acquisition module 11 is used to acquire the first action segment and the second action segment that are adjacent to each other in the time domain; Processing module 12 is configured to generate a transition sequence at the connection point between the first action segment and the second action segment, and perform one or more of the following: A perturbation is applied during the generation of the transition sequence to generate a transition sequence at the junction that differs from other transition sequences. Based on similarity metrics, duplicates are removed from the transition sequences already generated at the connection points and from other transition sequences.
[0065] In one embodiment of this application, the processing module 12 is configured to: extract the boundary frames of the first action segment and the second action segment at the connection point; query K transition sequences from the transition sequence library whose sequence boundaries are similar to the boundary frames; where K is a positive integer; determine the avoidance direction based on the K transition sequences; project the noise set for the connection point onto the orthogonal complement space of the avoidance direction to obtain orthogonal noise; fuse the initial semantic vectors corresponding to the first action segment and the second action segment with the orthogonal noise to obtain a target semantic vector; and generate a differentiated transition sequence at the connection point based on the target semantic vector and the boundary frames.
[0066] In one embodiment of this application, the processing module 12 is used to obtain the transition embedding of each transition sequence in the K transition sequences; and to determine the mean of the K transition embeddings corresponding to the K transition sequences as the avoidance direction.
[0067] In one embodiment of this application, the processing module 12 is used to perform affine normalization on each transition sequence in the K transition sequences, and calculate the first and second derivatives based on the normalization results; for each transition sequence in the K transition sequences, the corresponding normalization results, first and second derivatives are encoded to obtain the corresponding sequence embedding; for each transition sequence in the K transition sequences, the semantic vectors of the corresponding preceding and following action segments are concatenated with the sequence embedding to obtain the corresponding transition embedding.
[0068] In one embodiment of this application, the processing module 12 is used to calculate the quality score of each transition sequence in a plurality of transition sequences; wherein, the plurality of transition sequences include the transition sequence generated at the connection point, and the transition sequences between other action segments; the plurality of transition sequences are traversed in descending order of quality score to perform sequence retention and discarding; if the similarity between the current transition sequence and the retained transition sequence is lower than a threshold, the current transition sequence is retained; if the similarity between the current transition sequence and the retained transition sequence is higher than the threshold, the current transition sequence is discarded.
[0069] In one embodiment of this application, the similarity between the current transition sequence and the retained transition sequence is determined based on first information, or based on the first information and the time misalignment penalty of the current transition sequence. The first information includes one or more of the following determinations: Cosine similarity of transition embedding; Dynamic temporal warping similarity of attitude trajectories; Velocity spectrum difference.
[0070] In one embodiment of this application, the processing module 12 is further configured to perform a diversity measure on the discarded transition sequences and recall a portion of the transition sequences from the discarded transition sequences by employing a deterministic point process (DPP) subset selection mechanism.
[0071] Based on the same inventive concept Figure 7 A schematic diagram of the structure of a sequence generation device provided in this application embodiment. Figure 2 .like Figure 7 As shown in the embodiments of this application, the sequence generation device 1 includes: a processor 13, a memory 14, and a communication bus 15; The communication bus 15 is used to realize the communication connection between the processor 13 and the memory 14; The processor 13 is used to execute one or more computer programs stored in the memory 14 to implement the sequence generation method.
[0072] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a sequence generation method.
[0073] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a sequence generation method. The computer-readable storage medium can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it can be a device including one or any combination of the above-mentioned memories, such as a mobile phone, computer, tablet device, personal digital assistant, etc.
[0074] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0075] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0078] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A sequence generation method, characterized by, include: Obtain the first and second action segments that are adjacent to each other in the temporal domain; At the junction between the first action segment and the second action segment, a transition sequence is generated, and one or more of the following are performed: A perturbation is applied during the generation of the transition sequence to generate a transition sequence at the junction that differs from other transition sequences. Based on similarity metrics, duplicates are removed from the transition sequences already generated at the connection points and from other transition sequences.
2. The method of claim 1, wherein, Applying a perturbation during transition sequence generation to generate a transition sequence at the junction that differs from other transition sequences includes: Extract the boundary frames of the first action segment and the second action segment at the connection point; From the transition sequence library, query K transition sequences whose sequence boundaries are similar to the boundary frame; K is a positive integer; Based on the K-segment transition sequence, the avoidance direction is determined, and the noise set for the connection point is projected onto the orthogonal complement space of the avoidance direction to obtain orthogonal noise; The initial semantic vectors corresponding to the first action segment and the second action segment are fused with the orthogonal noise to obtain the target semantic vector; Based on the target semantic vector and the boundary frame, a differentiated transition sequence is generated at the junction.
3. The method of claim 2, wherein, The determination of the avoidance direction based on the K-segment transition sequence includes: Obtain the transition embedding of each transition sequence in the K transition sequences; The mean of the K transition embeddings corresponding to the K transition sequences is determined as the avoidance direction.
4. The method of claim 3, wherein, The method further includes: For each transition sequence in the K-segment transition sequence, affine normalization is performed, and the first and second derivatives are calculated based on the normalization results. For each transition sequence in the K transition sequences, the corresponding normalization result, first derivative, and second derivative are encoded to obtain the corresponding sequence embedding; For each transition sequence in the K transition sequences, the semantic vectors of the corresponding preceding and following action segments are concatenated with the sequence embedding to obtain the corresponding transition embedding.
5. The method of claim 1, wherein, The step of removing duplicates from the already generated transition sequence and other transition sequences at the connection point based on similarity measurement includes: Calculate the quality score for each of the multiple transition sequences; wherein, the multiple transition sequences include the transition sequence generated at the connection point, as well as the transition sequences between other action segments; The multiple transition sequences are traversed from high to low according to their quality scores, and the sequence is retained or discarded. If the similarity between the current transition sequence and the retained transition sequence is lower than the threshold, the current transition sequence is retained. If the similarity between the current transition sequence and the retained transition sequence is higher than the threshold, the current transition sequence is discarded.
6. The method of claim 5, wherein, The similarity between the current transition sequence and the retained transition sequence is determined based on the first information, or based on the first information and the time misalignment penalty of the current transition sequence. The first information includes one or more of the following determinations: Cosine similarity of transition embedding; Dynamic temporal warping similarity of attitude trajectories; Velocity spectrum difference.
7. The method of claim 5, wherein, The method further includes: A diversity measure is performed on the discarded transition sequences, and a subset of transition sequences are recalled from the discarded transition sequences by adopting a deterministic point process DPP subset selection mechanism.
8. A sequence generating apparatus characterized by comprising: include: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute one or more computer programs stored in the memory to implement the method according to any one of claims 1-7.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.
10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.
