Three-dimensional motion retrieval method and device, electronic equipment and storage medium
Through the combination of pre-trained search neural network and semantic interaction layer alignment layer, the problem of inaccurate text search for three-dimensional motion in the prior art is solved, and accurate retrieval of complex motion is achieved.
Patent Information
- Application Number
- CN202411475017.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-22
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art is difficult to accurately search for three-dimensional motion through text, especially when multiple body parts coordinate movement and local subtle changes.
Through a pre-trained search neural network, the three-dimensional motion search text is input into the neural network, and the three-dimensional motion sequence is determined using the semantic interaction layer and the semantic alignment layer matching.
It realizes accurate search of three-dimensional motion based on text, and can effectively deal with complex motion coordination and local subtle changes.
Smart Images
Figure CN119474450A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of motion retrieval technology, and in particular to a three-dimensional motion retrieval method, device, electronic equipment and storage medium. Background Art
[0002] Text-to-3D motion retrieval is a challenging task in information retrieval, where the core goal is to map natural language descriptions to relevant motion sequences by building a common space where text and motion are closely aligned.
[0003] However, an action sequence usually involves coordinated movements of multiple body parts, and it may be difficult to accurately retrieve the target motion sequence using coarse-grained natural language text descriptions. In addition, most motion changes occur locally and are subtle, making it difficult to align the retrieved text with the motion sequence.
[0004] Therefore, how to accurately retrieve three-dimensional motion through text has become a technical problem that needs to be solved urgently in this field. Summary of the invention
[0005] The present invention provides a three-dimensional motion retrieval method, device, electronic device and storage medium to solve the defect in the prior art that three-dimensional motion cannot be accurately retrieved through text. In the technical solution of the present invention, a retrieval neural network is pre-trained, and a three-dimensional motion retrieval text is input into the retrieval neural network, and then the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text is determined by matching the retrieval neural network, so that three-dimensional motion can be accurately retrieved according to the text.
[0006] The present invention provides a three-dimensional motion retrieval method, comprising the following steps.
[0007] Get three-dimensional motion retrieval text; Inputting the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network; The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text.
[0008] According to a three-dimensional motion retrieval method provided by the present invention, the retrieval neural network is trained by the following steps: Acquire the three-dimensional motion sample data and the description text; Determine a positive sample alignment loss function and a negative sample alignment loss function based on the three-dimensional motion sample data and the description text; The retrieval neural network to be trained is iteratively trained, and the parameters of the retrieval neural network to be trained are optimized based on the positive sample alignment loss function and the negative sample alignment loss function, and the retrieval neural network is obtained when a preset iteration termination condition is met.
[0009] According to a three-dimensional motion retrieval method provided by the present invention, the three-dimensional motion sample data includes a three-dimensional motion sample video, and the motion position, rotation invariant features and rotation features of the skeleton points corresponding to the three-dimensional motion sample video; the retrieval neural network to be trained includes a semantic interaction layer and a semantic alignment layer; The determining of the positive sample alignment loss function and the negative sample alignment loss function based on the three-dimensional motion sample data and the description text includes: Determine a semantically enhanced motion feature based on the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature; Performing fine-grained feature extraction on the description text to determine fine-grained text features corresponding to the description text; Inputting the semantically enhanced motion feature and the fine-grained text feature into the semantic interaction layer to obtain a target motion feature; The target motion feature and the fine-grained text feature are input into the semantic alignment layer to determine the positive sample alignment loss function and the negative sample alignment loss function.
[0010] According to a three-dimensional motion retrieval method provided by the present invention, the retrieval neural network to be trained also includes a skeleton feature extraction layer and a frame-level feature extraction layer; The determining of the semantically enhanced motion feature based on the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature comprises: Inputting the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature into the skeleton feature extraction layer, performing skeleton feature extraction, and obtaining a skeleton feature motion sequence; Inputting the skeleton feature motion sequence into the frame-level feature extraction layer, performing image frame extraction processing, and obtaining an image frame motion sequence; The image frame motion sequence is subjected to semantic enhancement processing to determine semantic enhancement motion features.
[0011] According to a three-dimensional motion retrieval method provided by the present invention, the retrieval neural network to be trained further includes a temporal encoder and a spatial encoder; The performing semantic enhancement processing on the image frame motion sequence to determine the semantic enhancement motion features includes: Inputting the image frame motion sequence into the time encoder to obtain the motion sequence temporal features; Inputting the image frame motion sequence into the spatial encoder to obtain motion sequence spatial features; The motion sequence temporal features and the motion sequence spatial features are fused to obtain the semantically enhanced motion features.
[0012] According to a three-dimensional motion retrieval method provided by the present invention, the retrieval neural network to be trained also includes a large language model and a text feature extraction layer; The step of extracting fine-grained features from the description text to determine fine-grained text features corresponding to the description text includes: Inputting the description text and the preset prompt word into the large language model to obtain a fine-grained text description corresponding to the description text output by the large language model; The fine-grained text description is input into the text feature extraction layer, and feature extraction processing is performed to obtain fine-grained text features.
[0013] According to a three-dimensional motion retrieval method provided by the present invention, the semantic interaction layer is used for: Performing local description processing on the semantically enhanced motion features to obtain a local description set; Based on the fine-grained text features and the cross-modal attention mechanism, each local descriptor in the local description set is aggregated to determine the target motion features.
[0014] The present invention also provides a three-dimensional motion retrieval device, comprising the following modules: An acquisition module, used for acquiring three-dimensional motion retrieval text; An acquisition module, used for inputting the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network; The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text.
[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the three-dimensional motion retrieval methods described above is implemented.
[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the three-dimensional motion retrieval method as described above is implemented.
[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the three-dimensional motion retrieval method as described above is implemented.
[0018] The three-dimensional motion retrieval method, device, electronic device and storage medium provided by the present invention first obtain a three-dimensional motion retrieval text, and then input the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network. The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text. In the technical solution of the present invention, the retrieval neural network is pre-trained, and the three-dimensional motion retrieval text is input into the retrieval neural network, and then the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text is matched by the retrieval neural network, so that the three-dimensional motion can be accurately retrieved according to the text. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0020] Figure 1 It is a flow chart of the three-dimensional motion retrieval method provided by the present invention.
[0021] Figure 2 It is a structural schematic diagram of the three-dimensional motion retrieval device provided by the present invention.
[0022] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] In view of the above problems in the prior art, the present invention provides a three-dimensional motion retrieval method. Figure 1 is one of the flow charts of the three-dimensional motion retrieval method provided by the present invention, such as Figure 1 As shown, the three-dimensional motion retrieval method includes the following steps 110 and 120.
[0025] Step 110: Obtain three-dimensional motion retrieval text.
[0026] Specifically, the movement search text input by the user can be obtained. The movement search text can be, for example, "a person jogs and throws an object".
[0027] Step 120: inputting the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network; The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text.
[0028] Specifically, the 3D motion retrieval text can be input into the retrieval neural network to obtain the 3D motion sequence output by the retrieval neural network. It is easy to understand that the retrieval neural network to be trained can be trained in advance based on the 3D motion sample data and the description text corresponding to the 3D motion sample data to obtain the retrieval neural network. After receiving the 3D motion retrieval text, the retrieval neural network can match the video in the preset 3D motion database with the 3D motion retrieval text, and determine the 3D motion sequence corresponding to the 3D motion retrieval text.
[0029] The three-dimensional motion retrieval method provided by the present invention first obtains a three-dimensional motion retrieval text, and then inputs the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network. The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text. In the technical solution of the present invention, the retrieval neural network is pre-trained, and the three-dimensional motion retrieval text is input into the retrieval neural network, and then the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text is matched and determined by the retrieval neural network, so that the three-dimensional motion can be accurately retrieved according to the text.
[0030] In one embodiment, the retrieval neural network is trained by the following steps: Acquire the three-dimensional motion sample data and the description text; Determine a positive sample alignment loss function and a negative sample alignment loss function based on the three-dimensional motion sample data and the description text; The retrieval neural network to be trained is iteratively trained, and the parameters of the retrieval neural network to be trained are optimized based on the positive sample alignment loss function and the negative sample alignment loss function, and the retrieval neural network is obtained when a preset iteration termination condition is met.
[0031] Specifically, three-dimensional motion sample data and description text can be obtained, and then the positive sample alignment loss function and the negative sample alignment loss function can be determined based on the three-dimensional motion sample data and the description text. Furthermore, the retrieval neural network to be trained is iteratively trained. During the iterative training process, the parameters of the retrieval neural network to be trained can be optimized based on the positive sample alignment loss function and the negative sample alignment loss function. When the preset iteration termination condition is met, the trained retrieval neural network can be obtained. Among them, the preset iteration termination condition can be that the iterative training round reaches a preset number of iterations, or it can also be that the positive sample alignment loss function and the negative sample alignment loss function reach their respective corresponding preset thresholds.
[0032] In the above embodiment, the positive sample alignment loss function and the negative sample alignment loss function are determined by the three-dimensional motion sample data and the description text. The three-dimensional motion sample data and the description text are used as training sets to ensure that the retrieval neural network can accurately complete the matching of the three-dimensional motion and the text. Further, in the process of iterative training of the retrieval neural network to be trained, the parameters of the retrieval neural network to be trained are optimized based on the positive sample alignment loss function and the negative sample alignment loss function, and the retrieval neural network is obtained when the preset iteration termination condition is met, so that the retrieval neural network can converge quickly.
[0033] In one embodiment, the three-dimensional motion sample data includes a three-dimensional motion sample video, and the motion position, rotation invariant features and rotation features of the skeletal points corresponding to the three-dimensional motion sample video; the retrieval neural network to be trained includes a semantic interaction layer and a semantic alignment layer; The determining of the positive sample alignment loss function and the negative sample alignment loss function based on the three-dimensional motion sample data and the description text includes: Determine a semantically enhanced motion feature based on the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature; Performing fine-grained feature extraction on the description text to determine fine-grained text features corresponding to the description text; Inputting the semantically enhanced motion feature and the fine-grained text feature into the semantic interaction layer to obtain a target motion feature; The target motion feature and the fine-grained text feature are input into the semantic alignment layer to determine the positive sample alignment loss function and the negative sample alignment loss function.
[0034] Specifically, the three-dimensional motion sample data includes a three-dimensional motion sample video, and the motion positions of the skeleton points, rotation-invariant features, and rotation features corresponding to the three-dimensional motion sample video. That is, firstly, the three-dimensional motion sample video, and the motion positions of the skeleton points, rotation-invariant features, and rotation features corresponding to the three-dimensional motion sample video can be obtained, and further, the semantically enhanced motion features can be determined based on the three-dimensional motion sample video, and the motion positions of the skeleton points, rotation-invariant features, and rotation features corresponding to the three-dimensional motion sample video. And fine-grained feature extraction can be performed on the description text corresponding to the three-dimensional motion sample data to determine the fine-grained text features corresponding to the description text. It is easy to understand that fine-grained feature extraction is to make a more accurate description based on the meaning of the description text.
[0035] Furthermore, the semantically enhanced motion features and the fine-grained text features can be input into a semantic interaction layer in a retrieval neural network to be trained, and the semantic interaction layer can output target motion features.
[0036] Then, the target motion features and fine-grained text features can be input into the semantic alignment layer in the retrieval neural network to be trained. The semantic alignment layer can output the similarity result, and then the positive sample alignment loss function and the negative sample alignment loss function can be determined based on the similarity result. The above process can be expressed by the following formula:
[0037] in, Represents the similarity result, Indicates the target motion characteristics, represents fine-grained text features, express The number of Indicates indivual , express The number of Indicates indivual , Indicates the maximum value, represents the positive sample alignment loss function, represents the negative sample alignment loss function, Indicates the number of three-dimensional motion sample data, Indicates control The hyperparameters of represents a natural constant, and Indicates the corresponding and The negative sample with the highest similarity, express A larger value between 0 and 0.
[0038] In addition, the positive sample alignment loss function can ensure tighter clustering and finer semantic capture, and the negative sample alignment loss function can fill the semantic inequality.
[0039] In the above embodiment, the original three-dimensional motion sample video, as well as the skeletal point motion positions, rotation-invariant features and rotation features corresponding to the three-dimensional motion sample video, are converted into semantically enhanced motion features, which can better reflect the three-dimensional motion. And the original description text is converted into fine-grained text features, which can better reflect the semantics of the text. Further combined with the fine-grained text features, the semantically enhanced motion features are converted into target motion features that are more strongly associated with the text. Finally, based on the target motion features and the fine-grained text features, the positive sample alignment loss function and the negative sample alignment loss function can be accurately determined to help iterative optimization of the retrieval neural network.
[0040] In one embodiment, the retrieval neural network to be trained further includes a skeleton feature extraction layer and a frame-level feature extraction layer; The determining of the semantically enhanced motion feature based on the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature comprises: Inputting the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature into the skeleton feature extraction layer, performing skeleton feature extraction, and obtaining a skeleton feature motion sequence; Inputting the skeleton feature motion sequence into the frame-level feature extraction layer, performing image frame extraction processing, and obtaining an image frame motion sequence; The image frame motion sequence is subjected to semantic enhancement processing to determine semantic enhancement motion features.
[0041] Specifically, the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature can be input into the skeleton feature extraction layer in the retrieval neural network to be trained, and the skeleton feature extraction is performed to obtain the skeleton feature motion sequence. ,in, , represents the set of real numbers, Indicates the number of frames of the 3D motion sample video, Indicates the number of bone points. The feature dimension representing the motion sequence of skeleton features. In addition, the skeleton feature extraction layer can be constructed based on a model of skinned vertices that can accurately represent various body shapes in natural human poses.
[0042] Furthermore, the skeleton feature motion sequence can be input into the frame-level feature extraction layer in the retrieval neural network to be trained, and image frame extraction processing can be performed to obtain the image frame motion sequence. ,in, , Represents the feature dimension of the image frame motion sequence. After obtaining the image frame motion sequence, the image frame motion sequence can be semantically enhanced to determine the semantically enhanced motion feature. In addition, the frame-level feature extraction layer can be composed of two layers of convolution with a filter size of 4 and a step size of 2.
[0043] In the above embodiment, the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature are input into the skeleton feature extraction layer, and the skeleton feature extraction is performed to obtain the skeleton feature motion sequence. The skeleton feature motion sequence reduces the complexity and redundancy of the data and can more accurately represent the three-dimensional motion. The skeleton feature motion sequence is further input into the frame-level feature extraction layer, and the image frame extraction process is performed to obtain the image frame motion sequence. The image frame motion sequence is then semantically enhanced to determine the semantically enhanced motion feature, so that the semantically enhanced feature can more accurately express the characteristics of the three-dimensional motion under the condition of low data complexity.
[0044] In one embodiment, the retrieval neural network to be trained further includes a temporal encoder and a spatial encoder; The performing semantic enhancement processing on the image frame motion sequence to determine the semantic enhancement motion features includes: Inputting the image frame motion sequence into the time encoder to obtain the motion sequence temporal features; Inputting the image frame motion sequence into the spatial encoder to obtain motion sequence spatial features; The motion sequence temporal features and the motion sequence spatial features are fused to obtain the semantically enhanced motion features.
[0045] Specifically, the image frame motion sequence can be input into the time encoder in the retrieval neural network to be trained to obtain the temporal features of the motion sequence. This process can be expressed by the following formula:
[0046] in, Represents the temporal characteristics of the motion sequence, Represents a time encoder.
[0047] The image frame motion sequence can also be input into the spatial encoder in the retrieval neural network to be trained to obtain the motion sequence spatial features. This process can be expressed by the following formula:
[0048] in, represents the spatial characteristics of motion sequence, Represents the spatial encoder.
[0049] Furthermore, by fusing the temporal features of the motion sequence and the spatial features of the motion sequence, semantically enhanced motion features can be obtained. This process can be expressed by the following formula:
[0050] in, represents semantically enhanced motion features, and , Feature dimensions representing semantically enhanced motion features.
[0051] In the above embodiment, the image frame motion sequence is input into the time encoder to obtain the motion sequence temporal features, and the image frame motion sequence is input into the space encoder to obtain the motion sequence spatial features. The feature extraction of the image frame motion sequence in the time dimension and the space dimension is completed respectively, and then the semantic enhanced motion features determined after the fusion process fully capture the spatiotemporal features of the three-dimensional motion, thereby improving its semantic expression ability.
[0052] In one embodiment, the retrieval neural network to be trained further includes a large language model and a text feature extraction layer; The step of extracting fine-grained features from the description text to determine fine-grained text features corresponding to the description text includes: Inputting the description text and the preset prompt word into the large language model to obtain a fine-grained text description corresponding to the description text output by the large language model; The fine-grained text description is input into the text feature extraction layer, and feature extraction processing is performed to obtain fine-grained text features.
[0053] Specifically, the description text and preset prompt words can be input into the large language model in the retrieval neural network to be trained. The large language model can split and expand the description text, and then obtain a fine-grained text description corresponding to the description text output by the large language model. The large language model can be pre-trained and constructed.
[0054] For example, the description text is: "A person jogging and throwing an object". After the description text and the preset prompt words are input into the large language model, the fine-grained text description output by the large language model can be: "Step 1: Stand with your feet shoulder-width apart; Step 2: Run forward with a steady gait; Step 3: Hold the object; Step 4: Keep your left arm at both sides of the body".
[0055] Furthermore, the fine-grained text description can be input into the text feature extraction layer in the retrieval neural network to be trained, and feature extraction processing can be performed to obtain fine-grained text features. This process can be expressed by the following formula:
[0056] in, Represents a fine-grained text description, Represents the processing done by the DistillBERT model in the text feature extraction layer. The DistillBERT model is a pre-trained language representation model obtained through knowledge distillation technology. represents the average pooling operation, represents the fine-grained text description after the average pooling operation, represents a feed-forward neural network, represents normalization processing, Represents fine-grained text features.
[0057] In the above embodiment, the description text and preset prompt words are input into the large language model to obtain a fine-grained text description corresponding to the description text output by the large language model, and then the fine-grained text description is input into the text feature extraction layer to perform feature extraction processing to obtain fine-grained text features. The fine-grained text feature optimization reduces the amount of data and can characterize the semantic features of the description text in more detail and accurately.
[0058] In one embodiment, the semantic interaction layer is used to: Performing local description processing on the semantically enhanced motion features to obtain a local description set; Based on the fine-grained text features and the cross-modal attention mechanism, each local descriptor in the local description set is aggregated to determine the target motion features.
[0059] Specifically, after the semantically enhanced motion features and fine-grained text features are input into the semantic interaction layer, the semantic interaction layer can perform local description processing on the semantically enhanced motion features through k descriptors Get a local description set, which includes multiple local descriptors , where the local descriptor It can be expressed by the following formula:
[0060] in, represents the number of semantically enhanced motion features, Indicates the number of descriptors, Indicates descriptors, Indicates semantically enhanced motion features, represents the exponential function, and Both represent learnable bias terms, and represents the trainable weights, represents L2 regularization processing, Indicates transpose.
[0061] Further based on the fine-grained text features and cross-modal attention mechanism, the local descriptors in the local description set are aggregated to determine the target motion features. The above process can be expressed by the following formula:
[0062] in, Represent the query, key and value determined based on the fine-grained text features and the local description set of motion features, respectively. represents transpose, represents the cross-modal attention mechanism, represents the activation function, represents the corresponding feature dimension, represents the target motion characteristics, represents normalization processing, Represents a trainable weight matrix.
[0063] In the above embodiment, the semantic interaction layer performs local description processing on the semantically enhanced motion features to obtain a local description set, and then aggregates the local descriptors in the local description set based on fine-grained text features and cross-modal attention mechanism to determine the target motion features, laying the foundation for determining the positive sample alignment loss function and the negative sample alignment loss function.
[0064] The three-dimensional motion retrieval device provided by the present invention is described below. The three-dimensional motion retrieval device described below and the three-dimensional motion retrieval method described above can be referenced to each other.
[0065] Figure 2 is a schematic diagram of the structure of the three-dimensional motion retrieval device provided by the present invention, such as Figure 2 As shown, the three-dimensional motion retrieval device 200 includes the following modules: An acquisition module 210 is used to acquire a three-dimensional motion retrieval text; An acquisition module 220 is used to input the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network; The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text.
[0066] The three-dimensional motion retrieval device provided by the present invention first obtains a three-dimensional motion retrieval text, and then inputs the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network. The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text. In the technical solution of the present invention, the retrieval neural network is pre-trained, and the three-dimensional motion retrieval text is input into the retrieval neural network, and then the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text is matched and determined by the retrieval neural network, so that the three-dimensional motion can be accurately retrieved according to the text.
[0067] In one embodiment, the 3D motion retrieval apparatus 200 further includes a training module, which is used to train the retrieval neural network. The training module is used to: Acquire the three-dimensional motion sample data and the description text; Determine a positive sample alignment loss function and a negative sample alignment loss function based on the three-dimensional motion sample data and the description text; The retrieval neural network to be trained is iteratively trained, and the parameters of the retrieval neural network to be trained are optimized based on the positive sample alignment loss function and the negative sample alignment loss function, and the retrieval neural network is obtained when a preset iteration termination condition is met.
[0068] In one embodiment, the three-dimensional motion sample data includes a three-dimensional motion sample video, and the motion position, rotation invariant features and rotation features of the skeletal points corresponding to the three-dimensional motion sample video; the retrieval neural network to be trained includes a semantic interaction layer and a semantic alignment layer; the training module is specifically used for: Determine a semantically enhanced motion feature based on the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature; Performing fine-grained feature extraction on the description text to determine fine-grained text features corresponding to the description text; Inputting the semantically enhanced motion feature and the fine-grained text feature into the semantic interaction layer to obtain a target motion feature; The target motion feature and the fine-grained text feature are input into the semantic alignment layer to determine the positive sample alignment loss function and the negative sample alignment loss function.
[0069] In one embodiment, the retrieval neural network to be trained further includes a skeleton feature extraction layer and a frame-level feature extraction layer; the training module is specifically used for: Inputting the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature into the skeleton feature extraction layer, performing skeleton feature extraction, and obtaining a skeleton feature motion sequence; Inputting the skeleton feature motion sequence into the frame-level feature extraction layer, performing image frame extraction processing, and obtaining an image frame motion sequence; The image frame motion sequence is subjected to semantic enhancement processing to determine semantic enhancement motion features.
[0070] In one embodiment, the retrieval neural network to be trained further includes a temporal encoder and a spatial encoder; the training module is specifically used for: Inputting the image frame motion sequence into the time encoder to obtain the motion sequence temporal features; Inputting the image frame motion sequence into the spatial encoder to obtain motion sequence spatial features; The motion sequence temporal features and the motion sequence spatial features are fused to obtain the semantically enhanced motion features.
[0071] In one embodiment, the retrieval neural network to be trained further includes a large language model and a text feature extraction layer; the training module is specifically used for: Inputting the description text and the preset prompt word into the large language model to obtain a fine-grained text description corresponding to the description text output by the large language model; The fine-grained text description is input into the text feature extraction layer, and feature extraction processing is performed to obtain fine-grained text features.
[0072] In one embodiment, the semantic interaction layer is used to: Performing local description processing on the semantically enhanced motion features to obtain a local description set; Based on the fine-grained text features and the cross-modal attention mechanism, each local descriptor in the local description set is aggregated to determine the target motion features.
[0073] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the three-dimensional motion retrieval method, which includes: Get three-dimensional motion retrieval text; Inputting the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network; The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text.
[0074] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0075] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the three-dimensional motion retrieval method provided by the above methods, the method includes: Get three-dimensional motion retrieval text; Inputting the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network; The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text.
[0076] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the three-dimensional motion retrieval method provided by the above methods is implemented, and the method includes: Get three-dimensional motion retrieval text; Inputting the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network; The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text.
[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A three-dimensional motion retrieval method, characterized in that: include: Get three-dimensional motion retrieval text; Inputting the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network; The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text.
2. The three-dimensional motion retrieval method according to claim 1, characterized in that: The retrieval neural network is trained by the following steps: Acquire the three-dimensional motion sample data and the description text; Determine a positive sample alignment loss function and a negative sample alignment loss function based on the three-dimensional motion sample data and the description text; The retrieval neural network to be trained is iteratively trained, and the parameters of the retrieval neural network to be trained are optimized based on the positive sample alignment loss function and the negative sample alignment loss function, and the retrieval neural network is obtained when a preset iteration termination condition is met.
3. The three-dimensional motion retrieval method according to claim 2, characterized in that: The three-dimensional motion sample data includes a three-dimensional motion sample video, and the motion position, rotation invariant feature and rotation feature of the skeleton points corresponding to the three-dimensional motion sample video; The retrieval neural network to be trained includes a semantic interaction layer and a semantic alignment layer; The determining of the positive sample alignment loss function and the negative sample alignment loss function based on the three-dimensional motion sample data and the description text includes: Determine a semantically enhanced motion feature based on the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature; Performing fine-grained feature extraction on the description text to determine fine-grained text features corresponding to the description text; Inputting the semantically enhanced motion feature and the fine-grained text feature into the semantic interaction layer to obtain a target motion feature; The target motion feature and the fine-grained text feature are input into the semantic alignment layer to determine the positive sample alignment loss function and the negative sample alignment loss function.
4. The three-dimensional motion retrieval method according to claim 3, characterized in that: The retrieval neural network to be trained also includes a skeleton feature extraction layer and a frame-level feature extraction layer; The determining of the semantically enhanced motion feature based on the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature comprises: Inputting the three-dimensional motion sample video, the motion position of the skeleton point, the rotation invariant feature and the rotation feature into the skeleton feature extraction layer, performing skeleton feature extraction, and obtaining a skeleton feature motion sequence; Inputting the skeleton feature motion sequence into the frame-level feature extraction layer, performing image frame extraction processing, and obtaining an image frame motion sequence; The image frame motion sequence is subjected to semantic enhancement processing to determine semantic enhancement motion features.
5. The three-dimensional motion retrieval method according to claim 4, characterized in that: The retrieval neural network to be trained also includes a temporal encoder and a spatial encoder; The performing semantic enhancement processing on the image frame motion sequence to determine the semantic enhancement motion features includes: Inputting the image frame motion sequence into the time encoder to obtain the motion sequence temporal features; Inputting the image frame motion sequence into the spatial encoder to obtain motion sequence spatial features; The motion sequence temporal features and the motion sequence spatial features are fused to obtain the semantically enhanced motion features.
6. The three-dimensional motion retrieval method according to any one of claims 3 to 5, characterized in that: The retrieval neural network to be trained also includes a large language model and a text feature extraction layer; The step of extracting fine-grained features from the description text to determine fine-grained text features corresponding to the description text includes: Inputting the description text and the preset prompt word into the large language model to obtain a fine-grained text description corresponding to the description text output by the large language model; The fine-grained text description is input into the text feature extraction layer, and feature extraction processing is performed to obtain fine-grained text features.
7. The three-dimensional motion retrieval method according to claim 6, characterized in that: The semantic interaction layer is used to: Performing local description processing on the semantically enhanced motion features to obtain a local description set; Based on the fine-grained text features and the cross-modal attention mechanism, each local descriptor in the local description set is aggregated to determine the target motion features.
8. A three-dimensional motion retrieval device, characterized in that: include: An acquisition module, used for acquiring three-dimensional motion retrieval text; An acquisition module, used for inputting the three-dimensional motion retrieval text into a retrieval neural network to obtain a three-dimensional motion sequence output by the retrieval neural network; The retrieval neural network is used to match and determine the three-dimensional motion sequence corresponding to the three-dimensional motion retrieval text; the retrieval neural network is trained based on three-dimensional motion sample data and corresponding description text.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the three-dimensional motion retrieval method according to any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional motion retrieval method according to any one of claims 1 to 7 is implemented.