Text-to-action retrieval method and device and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of information retrieval, and provides a text-to-action retrieval method and device and electronic equipment, and the method comprises the steps: obtaining a to-be-retrieved three-dimensional human body motion video, a motion position, a rotation invariant feature vector and a rotation feature vector of the three-dimensional human body motion video, and a description text for retrieval; and inputting the to-be-retrieved three-dimensional human body motion video, the motion position, the rotation invariant feature and the rotation feature vector of the three-dimensional human body motion video, and a description text for retrieval into a text-to-action retrieval semantic interaction neural network to obtain a text-to-motion sequence retrieval result. Therefore, by combining multiple level representation alignment of the text and the action sequence, the superimposed motion can be effectively aligned with the corresponding text, and the sensitivity to fine local motion is improved, so that accurate semantic alignment between the text and the motion sequence is realized, and the accuracy of text-to-motion sequence retrieval is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to a text-to-action retrieval method, device and electronic equipment. Background Art

[0002] Text-to-action retrieval is a challenging task in information retrieval, whose core goal is to enable queries from natural language descriptions to related motion sequences by building a common space where semantically similar texts and actions are closely aligned. However, in real-world scenarios, multiple actions often appear simultaneously and are superimposed on the same character model. In addition, most motion changes occur locally and are subtle, which poses a huge challenge to accurately aligning motion sequences and corresponding texts.

[0003] In the prior art, text and action sequences are represented as point embeddings for alignment. However, a single global embedding can only capture the rough semantics of the motion sequence, and it is difficult to accurately distinguish the complex semantics of complex motions. For example, in an action sequence, the human body can make multiple actions at the same time, such as squatting and holding the head, so there is a problem of co-occurrence of different actions, which is more complicated than a single action occurring at the same time, and increases the difficulty of retrieval from text to action sequences. Therefore, how to perform more sensitive and sophisticated cross-modal alignment to accurately match these superimposed motions and subtle motion changes in text-to-action retrieval has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a text-to-action retrieval method, device and electronic device to solve the defects of the prior art that only the rough semantics of motion sequences can be captured, the complex semantics of complex motions cannot be accurately distinguished, and precise retrieval cannot be achieved, so as to achieve the semantic alignment accuracy between text and action sequences and improve the retrieval accuracy.

[0005] The present invention provides a text-to-action retrieval method, comprising: Acquire a 3D human motion video to be retrieved, a motion position, a rotation-invariant feature vector and a rotation feature vector of the 3D human motion video, and a description text for retrieval; Inputting the three-dimensional human motion video to be retrieved, the motion position, the rotation invariant features and the rotation feature vector of the three-dimensional human motion video, and the description text for retrieval into a semantic interactive neural network of text-to-action retrieval to obtain a retrieval result of text-to-motion sequence; The semantic interaction neural network for text-to-action retrieval is trained by the following steps: Obtain a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video, and a first description text corresponding to the first three-dimensional human motion video in a training set; Extracting skeleton features and frame-level features of the first three-dimensional human motion video and text features of the first description text based on the first three-dimensional human motion video, the first motion position, the first rotation-invariant feature vector of the first three-dimensional human motion video, and the first rotation feature vector; Constructing word-level, instance-level, and package-level representations of the text according to the text features, and generating a text graph according to the first description text; Constructing frame-level, instance-level and packet-level representations of the action sequence according to the skeleton features and the frame-level features, and generating a motion sequence graph according to the first three-dimensional human motion video; Applying graph convolutional networks to the text graph and the motion sequence graph respectively, modeling node relationships and enhancing node representation by aggregating semantic relationships between nodes and capturing diverse semantic associations; Using similarity metrics at word level, instance level and bag level to perform cross-modal alignment of the text graph and the motion sequence graph and calculate corresponding alignment losses; The trained semantic interaction neural network for text-to-action retrieval is obtained by optimizing the total alignment loss at the word level, instance level and bag level.

[0006] In a possible implementation, the method further includes: A pre-trained frozen CLIP text encoder is used to perform feature extraction on the first description text to obtain text features of the first description text.

[0007] In a possible implementation, the method further includes: Segment the first description text into tokens or phrases, and embed each token or phrase to initialize a node; The connections between nodes are constructed through co-occurrence relations and cosine similarity to generate a complete text graph.

[0008] In a possible implementation, the method further includes: Segmenting the first three-dimensional human motion video into a plurality of motion segments by using a sliding window strategy; Perform average pooling on each motion segment to obtain the initial embedding of the instance node; Redundant motion segments are filtered by cosine similarity, and the filtered motion segments are embedded into instance nodes for initializing the graph to generate a motion sequence graph.

[0009] In a possible implementation, the method further includes: Applying a two-layer graph convolutional network to the text graph and the motion sequence graph respectively; The semantic features of neighbor nodes are aggregated through convolution operations and residual connections to capture the semantic relationships and semantic associations between neighbor nodes.

[0010] In a possible implementation, the method further includes: Performing word-level interaction of the text and frame-level interaction of the action sequence through a multi-layer perceptron; Calculating the similarity between the instance nodes by cosine similarity to perform instance-level interaction between the text and the action sequence; The package-level interaction between the text and the action sequence is performed through a root node representation; A multilayer perceptron with an activation function is used to calculate the similarity of the instance level, the bag level, the word level of the text and the frame level of the action sequence, and obtain the alignment loss of the instance level, the bag level, the word level of the text and the frame level of the action sequence.

[0011] In a possible implementation, the method further includes: Determine the total alignment loss of the word-unit level, instance level and bag-level hierarchy based on the fusion value of the alignment loss of the instance level, bag level and the word-unit level of the text and the frame level of the action sequence; The semantic interaction neural network for text-to-action retrieval is trained based on the total alignment loss to obtain a trained semantic interaction neural network for text-to-action retrieval.

[0012] The present invention also provides a text-to-action retrieval device, comprising the following modules: An acquisition module, used to acquire a 3D human motion video to be retrieved, a motion position, a rotation-invariant feature vector and a rotation feature vector of the 3D human motion video, and a description text for retrieval; A retrieval module, used for inputting the three-dimensional human motion video to be retrieved, the motion position, the rotation invariant features and the rotation feature vector of the three-dimensional human motion video, and the description text for retrieval into a semantic interactive neural network of text-to-action retrieval to obtain a retrieval result of text-to-motion sequence; A network training module is used to obtain a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video, and a first description text corresponding to the first three-dimensional human motion video in a training set; extract skeleton features and frame-level features of the first three-dimensional human motion video and extract text features of the first description text based on the first three-dimensional human motion video, the first motion position, the first rotation-invariant feature vector and the first rotation feature vector of the first three-dimensional human motion video; construct word-level, instance-level and package-level representations of the text according to the text features, and extract the skeleton features and frame-level features of the first three-dimensional human motion video and text features of the first description text according to the first description text ... The method generates a text graph; constructs frame-level, instance-level and package-level representations of the action sequence based on the skeleton features and frame-level features, and generates a motion sequence graph based on the first three-dimensional human motion video; applies a graph convolutional network to the text graph and the motion sequence graph, respectively, models node relationships and enhances node representations by aggregating semantic relationships between nodes and capturing diverse semantic associations; uses similarity metrics at the word level, instance level and package level to perform cross-modal alignment of the text graph and the motion sequence graph and calculates the corresponding alignment loss; and obtains the trained semantic interaction neural network for text-to-action retrieval by optimizing the total alignment loss at the word level, instance level and package level.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the text-to-action retrieval method described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the text-to-action retrieval method as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the text-to-action retrieval method as described above is implemented.

[0016] The text-to-action retrieval method, device and electronic device provided by the present invention obtain a three-dimensional human motion video to be retrieved, a motion position, a rotation-invariant feature vector and a rotation feature vector of the three-dimensional human motion video, and a description text for retrieval; the three-dimensional human motion video to be retrieved, the motion position, the rotation-invariant feature and the rotation feature vector of the three-dimensional human motion video, and the description text for retrieval are input into a semantic interactive neural network for text-to-action retrieval to obtain a retrieval result of a text-to-motion sequence; the semantic interactive neural network for text-to-action retrieval is trained by the following steps: obtaining a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video in a training set, and a first description text corresponding to the first three-dimensional human motion video; based on the first three-dimensional human motion video, the first motion position, the first three-dimensional human motion video The first rotation invariant feature vector and the first rotation feature vector of the video extract the skeleton features and frame-level features of the first three-dimensional human motion video and extract the text features of the first description text; construct the word-level, instance-level and package-level representations of the text according to the text features, and generate a text graph according to the first description text; construct the frame-level, instance-level and package-level representations of the action sequence according to the skeleton features and frame-level features, and generate a motion sequence graph according to the first three-dimensional human motion video; apply a graph convolutional network to the text graph and the motion sequence graph respectively, model the node relationship and enhance the node representation by aggregating the semantic relationship between nodes and capturing diverse semantic associations; use the similarity metrics at the word-level, instance-level and package-level levels to perform cross-modal alignment of the text graph and the motion sequence graph and calculate the corresponding alignment loss; and obtain the trained semantic interaction neural network for text-to-action retrieval by optimizing the total alignment loss at the word-level, instance-level and package-level levels. Compared with the defects of the prior art that can only capture the rough semantics of motion sequences, it is difficult to accurately distinguish the complex semantics of complex motions, and it is impossible to achieve precise retrieval, this solution combines package-level alignment, instance-level alignment, and word-level (frame) alignment to achieve precise semantic alignment between text and motion sequences. The multi-granularity alignment method encodes text and motion sequences into embeddings of different granularities, which can effectively capture the semantics of co-occurring motions. By aligning the text and action semantics, the co-occurring actions are accurately matched with the corresponding text descriptions. The multi-granularity alignment method encodes text and motion into representations of different granularities, and performs semantic interaction between these representations to achieve multi-granularity semantic alignment. Based on the above method, precise semantic alignment between text and motion sequences is achieved, which improves the accuracy of text-to-motion sequence retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the text-to-action retrieval method provided by the present invention.

[0019] Figure 2 It is one of the flow charts of the semantic interactive neural network training method for text-to-action retrieval provided by the present invention.

[0020] Figure 3 This is the second flow chart of the semantic interactive neural network training method for text-to-action retrieval provided by the present invention.

[0021] Figure 4 This is the third flow chart of the semantic interactive neural network training method for text-to-action retrieval provided by the present invention.

[0022] Figure 5 This is the fourth flow chart of the semantic interactive neural network training method for text-to-action retrieval provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the semantic interactive neural network training device for text-to-action retrieval provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the text-to-action retrieval device provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0028] Figure 1is a flowchart of the text-to-action retrieval method provided by the present invention, such as Figure 1 As shown, the method includes: S11, obtaining a three-dimensional human motion video to be retrieved, a motion position, a rotation-invariant feature vector and a rotation feature vector of the three-dimensional human motion video, and a description text for retrieval.

[0029] The text-to-motion retrieval method of the embodiment of the present invention is applied to a text-to-motion retrieval scenario, achieving precise semantic alignment between text and motion sequence, and improving the accuracy of text-to-motion sequence retrieval.

[0030] In the embodiment of the present invention, it is first necessary to train a semantic interaction neural network for text-to-action retrieval, combining package-level alignment, instance-level alignment, and word-level (frame) alignment to achieve precise semantic alignment between text and motion sequences. The multi-granularity-based alignment method encodes the text and motion sequences into embeddings of different granularities, which can effectively capture the semantics of co-occurring motions. By aligning the text and action semantics, the co-occurring actions are accurately matched with the corresponding text descriptions. The multi-granularity-based alignment method encodes the text and motion into representations of different granularities, and performs semantic interaction between these representations to achieve multi-granularity semantic alignment. Based on the above method, precise semantic alignment between text and motion sequences is achieved, thereby improving the accuracy of text-to-motion sequence retrieval. The training method of the semantic interaction neural network for text-to-action retrieval is Figure 2 The corresponding embodiments are described in detail.

[0031] In order to accurately and efficiently perform multi-granular semantic alignment of motion sequence and text, in an embodiment of the present invention, the motion sequence information to be retrieved and the description text for retrieval are first obtained, wherein the motion sequence information includes three-dimensional human motion video, motion position, rotation-invariant feature vector and rotation feature vector of the three-dimensional human motion video.

[0032] S12, inputting the three-dimensional human motion video to be retrieved, the motion position, the rotation-invariant features and the rotation feature vector of the three-dimensional human motion video, and the descriptive text used for retrieval into the semantic interactive neural network of text-to-action retrieval to obtain the retrieval result of text-to-motion sequence.

[0033] The three-dimensional human motion video to be retrieved, the motion position, the rotation-invariant features and rotation feature vectors of the three-dimensional human motion video, and the descriptive text used for retrieval are input into the semantic interactive neural network of text-to-action retrieval. Reasoning is performed through the semantic interactive neural network of text-to-action retrieval to capture the semantic associations between nodes and model the relationships between various semantic roles and action instances. The alignment operation achieves cross-modal semantic matching by calculating the similarities at the word (frame) level, instance level, and package level in the text and action sequence to ensure that the contribution of different similarities to the final retrieval results in the reasoning stage is consistent with the training process, and finally the retrieval results of the motion sequence and text are obtained.

[0034] The text-to-action retrieval method provided by the embodiment of the present invention obtains the three-dimensional human motion video to be retrieved, the motion position, the rotation-invariant feature vector and the rotation feature vector of the three-dimensional human motion video, and the description text used for retrieval; the three-dimensional human motion video to be retrieved, the motion position, the rotation-invariant feature and the rotation feature vector of the three-dimensional human motion video, and the description text used for retrieval are input into the semantic interaction neural network of text-to-action retrieval to obtain the retrieval result of text-to-motion sequence. Thus, by combining the alignment of multiple levels of representations of text and action sequence, the superimposed motion can be effectively aligned with the corresponding text, and the sensitivity to subtle local motion can be improved, thereby achieving accurate semantic alignment between text and motion sequence, and improving the accuracy of text-to-motion sequence retrieval.

[0035] Figure 2 is one of the flow charts of the semantic interactive neural network training method for text-to-action retrieval provided by the present invention, such as Figure 2 As shown, the method specifically includes: S21, obtaining a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video, and a first description text corresponding to the first three-dimensional human motion video in a training set.

[0036] Embodiments of the present invention are combined Figure 4 A detailed description is given. In order to accurately and efficiently perform multi-granular semantic alignment of motion sequences and texts, the embodiment of the present invention first obtains motion sequence information in a training set and a corresponding first description text, wherein the motion sequence information includes a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector of the first three-dimensional human motion video, and a first rotation feature vector.

[0037] S22. Extract skeleton features and frame-level features of the first three-dimensional human motion video and extract text features of the first description text based on the first three-dimensional human motion video, the first motion position, the first rotation-invariant feature vector and the first rotation feature vector of the first three-dimensional human motion video.

[0038] The text feature extractor extracts the text feature of the first description text; the motion feature extractor extracts the skeleton feature and frame-level feature of the motion sequence.

[0039] Specifically, in order to map the original data into a lower dimensional feature space, thereby reducing the complexity and redundancy of the data, and extracting a more compact, discriminative and representative feature representation, the embodiment of the present invention uses a pre-trained feature extractor to obtain the skeleton features of the motion sequence and its frame-level features and text features.

[0040] Perform skeleton features on the motion sequence M Extract, where Indicates the number of input frames. Indicates the number of skeleton nodes, Feature dimension representing the skeleton features.

[0041] After obtaining the skeleton features, the SMPL (Skinned Multi-Person Linear model) model is used to further extract frame-level representations. Convert to frame-level features ,in The dimension is , Indicates the frame number. Represents the dimension of frame-level features.

[0042] Furthermore, the frame features Downsampling is performed in the time domain. Specifically, the average pooling method is used to downsample the frame feature sequence into a fixed-length feature sequence. .

[0043] Furthermore, the pre-trained frozen CLIP (Contrastive Language–Image Pretraining) text encoder is used to represent the text , perform feature extraction, each text sequence contains words, among which , Represents the characteristic dimension of words.

[0044] S23. Construct word-level, instance-level, and package-level representations of the text according to the text features, and generate a text graph according to the first description text.

[0045] The first description text is segmented into words or phrases, and each word or phrase is embedded to initialize the nodes; the connection between the nodes is constructed through co-occurrence relationship and cosine similarity to generate a complete text graph.

[0046] In the embodiment of the present invention, the text and action representations of corresponding granularity are processed through operations such as sliding window and mean pooling to obtain corresponding word (frame) level, instance level and package level representations for subsequent multi-granularity semantic interaction operations. A text feature node is constructed based on the first description text.

[0047] The text features are constructed into package-level representation, instance-level representation and word-level representation. First, the word-level sequence T is average-pooled to initialize the root node feature. , extract noun phrases and verbs from T using an off-the-shelf semantic role parser and determine the semantic role of each noun phrase .

[0048] Furthermore, verb representations are used as instance nodes , and connected to the root node by a direct edge Connection, noun phrases are used as token nodes , and the corresponding instance node Connection, Edge The connections between are represented as semantic roles related to motion, such as Figure 3 As shown, considering that multiple atomic actions may occur simultaneously on the same word-meta node, the word-meta node is replicated for each semantic role and connected with the corresponding motion nodes.

[0049] For the possible co-occurrence and mutual coupling of atomic actions in a sequence, a co-occurrence action mining method is introduced to solve this problem by measuring the semantic consistency between atomic actions.

[0050] Given a motion sequence diagram , where the frame node Connect to the instance node , calculate the semantic consistency score between the two , the calculation formula is as follows:

[0051] in, and Respectively and By setting a threshold , if the semantic consistency score Greater than threshold , then the atomic action and Identify as co-occurring actions, otherwise they are considered as sequential actions. For these co-occurring actions, the node and are fused to obtain co-occurring action nodes, all belonging to and The frame nodes of the text graph are connected to the new instance nodes. The corresponding nodes in will also be merged to ensure that the structure is consistent with Consistent.

[0052] S24. Construct frame-level, instance-level, and packet-level representations of the action sequence according to the skeleton features and the frame-level features, and generate a motion sequence graph according to the first three-dimensional human motion video.

[0053] The first three-dimensional human motion video is divided into multiple motion segments through a sliding window strategy; average pooling is performed on each motion segment to obtain the initial embedding of the instance node; redundant motion segments are filtered through cosine similarity, and the filtered motion segments are embedded into the instance nodes for initializing the graph to generate a motion sequence graph.

[0054] Specifically, the skeleton features and frame-level features are constructed into package-level representations, instance-level representations, and frame-level representations, and initialized into motion sequence graphs according to the timing or dependency relationship. First, the average pooling is used to aggregate The semantics of , thereby initializing the root node The sequence level representation of .

[0055] For instance nodes, since actions occur sequentially, a simple and effective sliding window strategy is used to construct atomic action instances. Multiple sliding windows of different lengths are set and overlapped with a stride of 1. The window size is set to , for each sliding window , by performing mean pooling on the features within the window to obtain the segment features, the motion sequence can be divided into motion segments ,in And the window size is .

[0056] Furthermore, redundant segments are removed by using the pre-constructed probabilistic embedding space and calculating the cosine similarity between segment features and noun phrases. The formula is as follows:

[0057] in and They are extracted by the pre-trained model. phrase and fragments.

[0058] Furthermore, using the root node obtained from mean pooling The graph is initialized with the features of the instance nodes. The instance nodes are connected to the root node and the word node through edges. The representation of these edges is calculated by cosine similarity. The formula is as follows:

[0059] in, represents the cosine similarity, Represents the node index.

[0060] S25. Applying a graph convolutional network to the text graph and the motion sequence graph respectively, modeling node relationships and enhancing node representation by aggregating semantic relationships between nodes and capturing diverse semantic associations.

[0061] A two-layer graph convolutional network is applied to the text graph and motion sequence graph respectively. The semantic features of neighbor nodes are aggregated through convolution operations and residual connections to capture the semantic relations and semantic associations between neighbor nodes.

[0062] Specifically, the relationship graph convolutional network (R-GCN) is used to model the correlation of these nodes. Considering that there are two types of nodes in the text graph, a two-layer graph convolutional network with residual connections is used to capture the semantics of the nodes. , and related edges , node features are aggregated by the following formula:

[0063] in, Representation Node The semantic role of Representation Node The number of relationships, is the ReLU activation function, are learnable parameters.

[0064] Node-based A two-layer graph neural network with residual connections is used to aggregate semantic features from neighboring nodes:

[0065] in, are the learnable parameters of the motion graph, Representation and Node The number of corresponding relationships.

[0066] S26. Use similarity metrics at the word level, instance level, and package level to perform cross-modal alignment of the text graph and the motion sequence graph and calculate the corresponding alignment loss.

[0067] S27. Obtain the trained semantic interactive neural network for text-to-action retrieval by optimizing the total alignment loss at the word level, instance level, and package level.

[0068] Semantic interaction is performed on multi-granular representations of motion sequences and texts. Semantic interaction introduces three levels of similarity metrics for cross-modal alignment: word unit (frame) level, instance level, and bag level.

[0069] Specifically, word-level similarity is fine-grained alignment by adding position embedding and combining Transformer encoder; instance-level and bag-level similarity respectively calculate the cosine similarity between instance nodes and root nodes. Finally, the three similarity metrics are combined to achieve semantic interaction between text and action sequences.

[0070] In order to better perform alignment training, the overall loss in the embodiment of the present invention is composed of packet-level alignment loss, instance-level alignment loss, and word (frame) alignment loss. At the same time, in order to capture the relationship and semantic association between nodes, the embodiment of the present invention applies two-layer graph convolutional networks and residual connections to the text and motion sequence graphs respectively through the graph reasoning module to aggregate the semantic features of neighboring nodes.

[0071] The package-level interaction directly calculates the semantic interaction between the text and the root node representation of the motion graph:

[0072] Furthermore, instance-level interaction uses instance node representations from text and motion sequence graphs to perform instance-level semantic interaction. and , the instance-level similarity is calculated as:

[0073] Furthermore, in addition to instance-level and bag-level semantic interactions, word-unit (frame)-level interactions are introduced to provide fine-grained semantic alignment as a supplement. First, the position embedding and Add to word-level and frame-level features and and feed it into the encoder:

[0074] in, Represents a splicing operation, It is a two-layer Transformer encoder.

[0075] Furthermore, a multi-layer perceptron (MLP) with ReLU activation function is used to calculate the similarity:

[0076] Among them, the MLP consists of two linear layers with ReLU function.

[0077] In the proposed multi-granular multi-instance learning, the positive sample pair is defined as a motion sequence containing content related to the query text. The negative sample pair is a motion sequence that does not contain any relevant content. By adopting the InfoNCE loss function as the training objective function, which is widely used in retrieval-related tasks, in small batches The loss function is:

[0078] in, represents a similarity measure, such as cosine similarity. The bag-level alignment loss and instance-level alignment loss are calculated by the above formula.

[0079] The loss of word-level (frame) semantic interaction is defined as:

[0080] in, is the cross entropy loss function.

[0081] Furthermore, the total training loss function is:

[0082] in , and They are respectively word-level (frame) alignment loss, instance-level alignment loss, and bag-level alignment loss.

[0083] Furthermore, after the model converges, the query text and motion sequence The similarity between them is calculated by combining the similarity at the package level, instance level, and word (frame) level:

[0084] In this way, semantic information at different levels can be comprehensively considered, thereby improving retrieval performance.

[0085] like Figure 5 As shown, the semantic interactive neural network for text-to-action retrieval in the embodiment of the present invention includes the following modules: Motion sequence feature extraction module: used for the motion sequence to be aligned, to obtain the corresponding motion sequence skeleton features and frame-level features; Motion sequence graph construction module: The motion sequence graph construction module is used to construct motion sequence graph nodes to obtain package-level representation, instance-level representation and frame-level representation of the action sequence.

[0086] Text feature extraction module: used for the text description corresponding to the motion sequence to obtain the corresponding text features.

[0087] Text graph construction module: The text graph construction module is used to construct text feature nodes to obtain text feature package-level representation, instance-level representation and word-level representation.

[0088] Graph reasoning module: includes text and motion sequence graph reasoning modules, which apply two-layer graph convolutional networks to text graphs and motion sequence graphs respectively, aggregate the semantic features of neighbor nodes through residual connections, and thus capture the relationship and semantic association between nodes.

[0089] Semantic Interaction Module: It is implemented through instance-level, bag-level, and token-level similarity calculations. It combines the alignment of instance nodes, root nodes, and word- and frame-level features between text and motion sequences to capture multi-granular semantic relationships.

[0090] Token (frame) level semantic interaction: After adding position embedding to the sequence features of text and action respectively, the features are extracted through the Transformer encoder, and the similarity scores of the two are calculated using a multi-layer perceptron (MLP) to achieve fine-grained semantic alignment.

[0091] Instance-level semantic interaction: Extract instance node features from text and motion sequence graphs, calculate the cosine similarity between them, and obtain the instance-level semantic alignment score.

[0092] Package-level semantic interaction: Compare the representations of the root nodes in the text and motion sequence graph, and obtain the overall semantic alignment score by calculating the cosine similarity between the two.

[0093] Specifically, the semantic interactive neural network of text-to-action retrieval includes a motion sequence feature extraction module, a text feature extraction module, a motion sequence graph construction module, a text graph construction module, a motion sequence graph reasoning module, a text graph reasoning module, and a multi-granularity semantic interaction module; wherein the motion sequence feature extraction module is used to extract skeleton features and frame-level features of the motion sequence, and the text feature extraction module is used to extract text features of the corresponding description text; optionally, the motion sequence feature extraction module is a model based on skinned vertices, which can accurately represent various body shapes in natural human postures, and the parameters of the model are obtained through data learning, including the remaining posture templates, mixing weights, posture-related mixed shapes, identity-related mixed shapes, and regressors from vertices to joint positions; optionally, the text feature extraction module uses a pre-trained frozen CLIP text encoder, which is a model based on a bidirectional encoder representation architecture of Transformer, and text feature extraction is performed by pre-training and freezing parameters.

[0094] The motion sequence graph construction module and the text graph construction module are used to obtain package-level representations, instance-level representations, and word-level (frame)-level representations of the motion sequence features and the text features according to the motion sequence features and the text features; optionally, the text package-level representations, instance-level representations, and word-level representations are obtained through a pre-trained CLIP text encoder, a ready-made semantic role tagger, and an average pooling operation; optionally, the package-level representations, instance-level representations, and frame-level representations of the motion sequence are obtained through operations such as an SMPL model, downsampling, and sliding windows.

[0095] The method of the above embodiment maps motion sequences and texts into a lower-dimensional feature space through the feature extraction module of the semantic interactive neural network of text-to-action retrieval, thereby reducing the complexity and redundancy of the data and extracting a more compact and discriminative and representative feature representation; through the graph construction module, the motion sequence features and text features are encoded into package-level representations, instance-level representations, and word-level (frame) representations for subsequent multi-granularity semantic alignment.

[0096] In one example, the motion sequence graph construction module is used to represent the frame features, motion segment features, and global features of the entire sequence in the motion sequence as different nodes in the graph, so as to better capture and represent the complex semantic information of the motion sequence in the subsequent graph reasoning and semantic interaction process, wherein sliding windows of different lengths are set in the motion sequence. , there is overlap between windows. Perform mean pooling on the features in each sliding window to generate motion segment features , and use these fragments as instance nodes.

[0097] To remove motion fragments The redundant fragments in are filtered using the previously constructed probabilistic embedding space. Specifically, according to the fragment features retained in the following formula :

[0098] These features are used to initialize instance nodes in the motion sequence graph.

[0099] In one example, the text graph reasoning module is used to: The multi-granularity representation of the text obtained by the graph construction module is used to semantically aggregate the text graph nodes. Specifically, in the embodiment of the present invention, the semantic aggregation of the neighbor nodes of the text graph is calculated using the following formula:

[0100] in, Representation Node The semantic role of Representation Node The number of relationships, is the learnable parameter matrix.

[0101] In one example, the motion sequence graph reasoning module is used to perform semantic aggregation of motion sequence graph nodes. Specifically, in an embodiment of the present invention, the semantic aggregation of neighbor nodes of the motion sequence graph is calculated using the following formula:

[0102] Among them, among them, Representation Node relationship, Representation Node The number of relationships, is the learnable parameter matrix.

[0103] In one example, a multi-granularity semantic interaction module is used to align motion sequences with text representations of various granularities. The semantic interaction achieves cross-modal semantic matching by calculating the similarities between the text and the motion sequence at the word level, instance level, and package level.

[0104] Specifically, in order to provide fine-grained semantic alignment, we use word and frame level features and Add position embeds separately and , and then input them into a two-layer Transformer encoder for processing:

[0105] In one example, in the proposed multi-granular multi-instance learning, the positive sample pair is defined as a motion sequence containing content related to the query text, and the negative sample pair is a motion sequence that does not contain any relevant content. The InfoNCE loss function is used as the training objective function. Specifically, in the embodiment of the present invention, the Loss calculation method in the training stage is:

[0106] in, represents a similarity measure, such as cosine similarity, and Respectively represent the instance-level similarity and package-level similarity Applied to the above formula, the loss of word-level semantic interaction is defined as:

[0107] in, is the cross entropy loss function, so the total training loss function is:

[0108] After the model converges, query the text and motion sequence The similarity between them is calculated by a combination of instance-level, bag-level, and token (frame)-level similarities:

[0109] In this way, semantic information at different levels can be comprehensively considered, thereby improving retrieval performance.

[0110] In order to evaluate the accuracy of the text-action retrieval results in the embodiments of the present invention, the test results of the present invention are compared with the existing text-action retrieval methods. The multi-granular multi-instance semantic interaction neural network for text-action retrieval outperforms other methods and achieves the best performance. Tested on the test sets of HumanML3D and KITMotion-Language, 3D human motion datasets with text descriptions, our method achieved state-of-the-art results on the most stringent text-to-motion retrieval index R@1, indicating that the proposed method can achieve accurate text-to-3D human motion retrieval by achieving multi-granular semantic alignment. On the index Rsum that reports the overall retrieval performance, our method surpasses the current state-of-the-art work with scores of 201.27 and 243.64 on HumanML3D and KIT-ML, respectively. This proves the superiority of our multi-granular multi-instance registration technology in text-to-action retrieval.

[0111] The semantic interactive neural network training method for text-to-action retrieval provided in an embodiment of the present invention obtains a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video in a training set, and a first description text corresponding to the first three-dimensional human motion video; based on the first three-dimensional human motion video, the first motion position, the first rotation-invariant feature vector and the first rotation feature vector of the first three-dimensional human motion video, skeleton features and frame-level features of the first three-dimensional human motion video are extracted, and text features of the first description text are extracted; word-level, instance-level and package-level representations of the text are constructed according to the text features , and generate a text graph according to the first description text; construct frame-level, instance-level and package-level representations of the action sequence according to the skeleton features and frame-level features, and generate a motion sequence graph according to the first three-dimensional human motion video; apply graph convolutional networks to the text graph and motion sequence graph respectively, model node relationships and enhance node representations by aggregating semantic relationships between nodes and capturing diverse semantic associations; use similarity metrics at the word level, instance level and package level to perform cross-modal alignment of the text graph and motion sequence graph and calculate the corresponding alignment loss; and obtain the trained semantic interaction neural network for text-to-action retrieval by optimizing the total alignment loss at the word level, instance level and package level. With this method, package-level alignment, instance-level alignment and word (frame) level alignment are combined to achieve accurate semantic alignment between text and motion sequence. The multi-granularity alignment method encodes text and motion sequence into embeddings of different granularities, which can effectively capture the semantics of co-occurring motions, and accurately match co-occurring motions with corresponding text descriptions by aligning text and action semantics. The multi-granularity alignment method encodes text and motion into representations of different granularities, and performs semantic interactions between these representations to achieve multi-granularity semantic alignment. Based on the above method, accurate semantic alignment between text and motion sequences is achieved, improving the accuracy of text-to-motion sequence retrieval.

[0112] The semantic interactive neural network training device for text-to-action retrieval provided by the present invention is described below. The semantic interactive neural network training device for text-to-action retrieval described below and the semantic interactive neural network training method for text-to-action retrieval described above can be referenced to each other.

[0113] Figure 6 : is a schematic diagram of the structure of the semantic interactive neural network training device for text-to-action retrieval provided by the present invention, which specifically includes: The acquisition module 710 is used to acquire the first 3D human motion video, the first motion position, the first rotation-invariant feature vector and the first rotation feature vector of the first 3D human motion video in the training set, and the first description text corresponding to the first 3D human motion video; based on the first 3D human motion video, the first motion position, the first rotation-invariant feature vector and the first rotation feature vector of the first 3D human motion video, the skeleton features and frame-level features of the first 3D human motion video are extracted, and the text features of the first description text are extracted. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0114] The graph construction module 720 is used to construct a word-level, instance-level and package-level representation of the text according to the text features, and generate a text graph according to the first description text; construct a frame-level, instance-level and package-level representation of the action sequence according to the skeleton features and frame-level features, and generate a motion sequence graph according to the first three-dimensional human motion video. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0115] The graph reasoning module 730 is used to apply a graph convolutional network to the text graph and the motion sequence graph, respectively, to model node relationships and enhance node representation by aggregating semantic relationships between nodes and capturing diverse semantic associations. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0116] The multi-granularity semantic interaction module 740 is used to perform cross-modal alignment of the text graph and the motion sequence graph using similarity metrics at the word level, instance level, and package level and calculate the corresponding alignment loss; the trained semantic interaction neural network for text-to-action retrieval is obtained by optimizing the total alignment loss at the word level, instance level, and package level. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0117] The text-to-action retrieval device provided by the present invention is described below. The text-to-action retrieval device described below and the text-to-action retrieval method described above can be referred to each other.

[0118] Figure 7 : is a schematic diagram of the structure of the text-to-action retrieval device provided by the present invention, which specifically includes: The acquisition module 701 is used to acquire the 3D human motion video to be retrieved, the motion position, the rotation invariant feature vector and the rotation feature vector of the 3D human motion video, and the description text used for retrieval. Detailed descriptions refer to the relevant descriptions corresponding to the above method embodiments, which will not be repeated here.

[0119] The retrieval module 702 is used to input the three-dimensional human motion video to be retrieved, the motion position, the rotation invariant features and rotation feature vectors of the three-dimensional human motion video, and the description text for retrieval into the semantic interactive neural network of text-to-action retrieval to obtain the retrieval result of the text-to-motion sequence. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0120] The network training module 703 is used to obtain a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video, and a first description text corresponding to the first three-dimensional human motion video in a training set; extract skeleton features and frame-level features of the first three-dimensional human motion video and extract text features of the first description text based on the first three-dimensional human motion video, the first motion position, the first rotation-invariant feature vector and the first rotation feature vector of the first three-dimensional human motion video; construct word-level, instance-level and package-level representations of the text according to the text features, and extract the skeleton features and frame-level features of the first three-dimensional human motion video and the text features of the first description text according to the first description text ... The text generates a text graph; the frame-level, instance-level and package-level representations of the action sequence are constructed according to the skeleton features and frame-level features, and a motion sequence graph is generated according to the first three-dimensional human motion video; the graph convolutional network is applied to the text graph and the motion sequence graph respectively, and the node relationship is modeled and the node representation is enhanced by aggregating the semantic relationship between nodes and capturing diverse semantic associations; the similarity measurement at the word level, instance level and package level is used to perform cross-modal alignment of the text graph and the motion sequence graph and calculate the corresponding alignment loss; the semantic interactive neural network for text-to-action retrieval that has been trained is obtained by optimizing the total alignment loss at the word level, instance level and package level. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0121] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the text-to-action retrieval method, which includes: obtaining a three-dimensional human motion video to be retrieved, a motion position, a rotation-invariant feature vector and a rotation feature vector of the three-dimensional human motion video, and a description text for retrieval; inputting the three-dimensional human motion video to be retrieved, the motion position, the rotation-invariant feature and the rotation feature vector of the three-dimensional human motion video, and the description text for retrieval into a semantic interactive neural network for text-to-action retrieval to obtain a retrieval result from a text-to-motion sequence; the semantic interactive neural network for text-to-action retrieval is trained by the following steps: obtaining a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video in a training set, and a first description text corresponding to the first three-dimensional human motion video; based on the first three-dimensional human motion video, the first motion position, the rotation-invariant feature vector and the rotation feature vector of the first three-dimensional human motion video, The first rotation invariant feature vector and the first rotation feature vector of the first three-dimensional human motion video extract the skeleton features and frame-level features of the first three-dimensional human motion video and extract the text features of the first description text; construct the word-level, instance-level and package-level representations of the text according to the text features, and generate a text graph according to the first description text; construct the frame-level, instance-level and package-level representations of the action sequence according to the skeleton features and frame-level features, and generate a motion sequence graph according to the first three-dimensional human motion video; apply a graph convolutional network to the text graph and the motion sequence graph respectively, model the node relationship and enhance the node representation by aggregating the semantic relationships between nodes and capturing diverse semantic associations; use the similarity metrics at the word-level, instance-level and package-level levels to perform cross-modal alignment of the text graph and the motion sequence graph and calculate the corresponding alignment loss; and obtain the trained semantic interactive neural network for text-to-action retrieval by optimizing the total alignment loss at the word-level, instance-level and package-level levels.

[0122] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the text-to-action retrieval method provided by the above-mentioned methods, which includes: obtaining a three-dimensional human motion video to be retrieved, a motion position, a rotation-invariant feature vector and a rotation feature vector of the three-dimensional human motion video, and a description text for retrieval; inputting the three-dimensional human motion video to be retrieved, the motion position, the rotation-invariant features and the rotation feature vector of the three-dimensional human motion video, and the description text for retrieval into a semantic interactive neural network for text-to-action retrieval to obtain a retrieval result from a text-to-motion sequence; the semantic interactive neural network for text-to-action retrieval is trained by the following steps: obtaining a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video in a training set, and a description text for the first three-dimensional human motion video. The invention relates to a first description text corresponding to the first three-dimensional human motion video; based on the first three-dimensional human motion video, the first motion position, the first rotation invariant feature vector of the first three-dimensional human motion video and the first rotation feature vector, the skeleton features and frame-level features of the first three-dimensional human motion video are extracted, and the text features of the first description text are extracted; according to the text features, the word-level, instance-level and package-level representations of the text are constructed, and a text graph is generated according to the first description text; according to the skeleton features and frame-level features, the frame-level, instance-level and package-level representations of the action sequence are constructed, and a motion sequence graph is generated according to the first three-dimensional human motion video; a graph convolutional network is applied to the text graph and the motion sequence graph respectively, and the node relationship is modeled and the node representation is enhanced by aggregating the semantic relationships between nodes and capturing diverse semantic associations; the similarity metrics at the word-level, instance-level and package-level are used to perform cross-modal alignment of the text graph and the motion sequence graph and calculate the corresponding alignment loss; the semantic interaction neural network for text-to-action retrieval that has been trained is obtained by optimizing the total alignment loss at the word-level, instance-level and package-level.

[0124] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the text-to-action retrieval method provided by the above-mentioned methods, the method comprising: obtaining a three-dimensional human motion video to be retrieved, a motion position, a rotation-invariant feature vector and a rotation feature vector of the three-dimensional human motion video, and a description text for retrieval; inputting the three-dimensional human motion video to be retrieved, the motion position, the rotation-invariant features and the rotation feature vector of the three-dimensional human motion video, and the description text for retrieval into a semantic interactive neural network for text-to-action retrieval to obtain a retrieval result of a text-to-motion sequence; the semantic interactive neural network for text-to-action retrieval is trained by the following steps: obtaining a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video in a training set, and a first description text corresponding to the first three-dimensional human motion video; based on the first A three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector of the first three-dimensional human motion video, and a first rotation feature vector are used to extract skeleton features and frame-level features of the first three-dimensional human motion video and extract text features of the first description text; word-level, instance-level, and package-level representations of the text are constructed according to the text features, and a text graph is generated according to the first description text; frame-level, instance-level, and package-level representations of the action sequence are constructed according to the skeleton features and frame-level features, and a motion sequence graph is generated according to the first three-dimensional human motion video; a graph convolutional network is applied to the text graph and the motion sequence graph respectively, and node relationships are modeled and node representations are enhanced by aggregating semantic relationships between nodes and capturing diverse semantic associations; similarity metrics at the word-level, instance-level, and package-level are used to perform cross-modal alignment of the text graph and the motion sequence graph and calculate the corresponding alignment loss; the trained semantic interaction neural network for text-to-action retrieval is obtained by optimizing the total alignment loss at the word-level, instance-level, and package-level.

[0125] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text-to-action retrieval method, characterized in that: include: Acquire a 3D human motion video to be retrieved, a motion position, a rotation-invariant feature vector and a rotation feature vector of the 3D human motion video, and a description text for retrieval; Inputting the three-dimensional human motion video to be retrieved, the motion position, the rotation invariant features and the rotation feature vector of the three-dimensional human motion video, and the description text for retrieval into a semantic interactive neural network of text-to-action retrieval to obtain a retrieval result of the text-to-motion sequence; The semantic interaction neural network for text-to-action retrieval is trained by the following steps: Obtain a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video, and a first description text corresponding to the first three-dimensional human motion video in a training set; Extracting skeleton features and frame-level features of the first three-dimensional human motion video and text features of the first description text based on the first three-dimensional human motion video, the first motion position, the first rotation-invariant feature vector of the first three-dimensional human motion video, and the first rotation feature vector; Constructing word-level, instance-level, and package-level representations of the text according to the text features, and generating a text graph according to the first description text; Constructing frame-level, instance-level and packet-level representations of the action sequence according to the skeleton features and the frame-level features, and generating a motion sequence graph according to the first three-dimensional human motion video; Applying graph convolutional networks to the text graph and the motion sequence graph respectively, modeling node relationships and enhancing node representation by aggregating semantic relationships between nodes and capturing diverse semantic associations; Using similarity metrics at word level, instance level and bag level to perform cross-modal alignment of the text graph and the motion sequence graph and calculate corresponding alignment losses; The trained semantic interaction neural network for text-to-action retrieval is obtained by optimizing the total alignment loss at the word level, instance level and bag level.

2. The method according to claim 1, characterized in that The extracting of the text features of the first description text includes: A pre-trained frozen CLIP text encoder is used to perform feature extraction on the first description text to obtain text features of the first description text.

3. The method according to claim 2, characterized in that The step of generating a text graph according to the first description text comprises: Segment the first description text into tokens or phrases, and embed each token or phrase to initialize a node; The connections between nodes are constructed through co-occurrence relations and cosine similarity to generate a complete text graph.

4. The method according to claim 2, characterized in that: The step of generating a motion sequence graph according to the first three-dimensional human motion video comprises: Segmenting the first three-dimensional human motion video into a plurality of motion segments by using a sliding window strategy; Perform average pooling on each motion segment to obtain the initial embedding of the instance node; Redundant motion segments are filtered by cosine similarity, and the filtered motion segments are embedded into instance nodes for initializing the graph to generate a motion sequence graph.

5. The method according to claim 3 or 4, characterized in that: The graph convolutional network is applied to the text graph and the motion sequence graph respectively, and the node relationship is modeled and the node representation is enhanced by aggregating the semantic relationship between nodes and capturing diverse semantic associations, including: Applying a two-layer graph convolutional network to the text graph and the motion sequence graph respectively; The semantic features of neighbor nodes are aggregated through convolution operations and residual connections to capture the semantic relationships and semantic associations between neighbor nodes.

6. The method according to claim 5, characterized in that The cross-modal alignment of the text graph and the motion sequence graph using similarity metrics at the word level, instance level, and bag level and calculating the corresponding alignment loss includes: Performing word-level interaction of the text and frame-level interaction of the action sequence through a multi-layer perceptron; Calculating the similarity between the instance nodes by cosine similarity to perform instance-level interaction between the text and the action sequence; The package-level interaction between the text and the action sequence is performed through a root node representation; A multilayer perceptron with an activation function is used to calculate the similarity of the instance level, the bag level, the word level of the text and the frame level of the action sequence, and obtain the alignment loss of the instance level, the bag level, the word level of the text and the frame level of the action sequence.

7. The method according to claim 6, characterized in that The semantic interaction neural network for text-to-action retrieval obtained by optimizing the total alignment loss at the word level, instance level and bag level includes: Determine the total alignment loss of the word-unit level, instance level and bag-level hierarchy based on the fusion value of the alignment loss of the instance level, bag level and the word-unit level of the text and the frame level of the action sequence; The semantic interaction neural network for text-to-action retrieval is trained based on the total alignment loss to obtain a trained semantic interaction neural network for text-to-action retrieval.

8. A text-to-action retrieval device, characterized in that: include: An acquisition module, used to acquire a 3D human motion video to be retrieved, a motion position, a rotation-invariant feature vector and a rotation feature vector of the 3D human motion video, and a description text for retrieval; A retrieval module, used for inputting the three-dimensional human motion video to be retrieved, the motion position, the rotation invariant features and the rotation feature vector of the three-dimensional human motion video, and the description text for retrieval into a semantic interactive neural network of text-to-action retrieval to obtain a retrieval result of text-to-motion sequence; A network training module, used for obtaining a first three-dimensional human motion video, a first motion position, a first rotation-invariant feature vector and a first rotation feature vector of the first three-dimensional human motion video, and a first description text corresponding to the first three-dimensional human motion video in a training set; Extract skeleton features and frame-level features of the first three-dimensional human motion video and extract text features of the first description text based on the first three-dimensional human motion video, the first motion position, the first rotation-invariant feature vector of the first three-dimensional human motion video, and the text features of the first description text; construct word-level, instance-level, and package-level representations of the text according to the text features, and generate a text graph according to the first description text; construct frame-level, instance-level, and package-level representations of the action sequence according to the skeleton features and frame-level features, and generate a motion sequence graph according to the first three-dimensional human motion video; A graph convolutional network is applied to the text graph and motion sequence graph respectively, and the node relationship is modeled and the node representation is enhanced by aggregating the semantic relationship between nodes and capturing diverse semantic associations; the text graph and motion sequence graph are cross-modally aligned using similarity metrics at the word level, instance level, and package level, and the corresponding alignment loss is calculated; and the trained semantic interaction neural network for text-to-action retrieval is obtained by optimizing the total alignment loss at the word level, instance level, and package level.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the text-to-action retrieval method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text-to-action retrieval method according to any one of claims 1 to 7 is implemented.

Images