Target role action 3D sequence optimization method and electronic equipment

Abstract

The invention discloses a target role action 3D sequence optimization method and electronic equipment, and the method comprises the steps: carrying out the skeleton-level redirection of a source skeleton action sequence, and obtaining a target skeleton action sequence; driving the source role image through the source skeleton action sequence to obtain a source role action 3D sequence, and driving the target role image through the target skeleton action sequence to obtain a target role action 3D sequence; processing the source role action 3D sequence and the target role action 3D sequence based on a micro-renderable and visual language model to obtain source action semantic features and target action semantic features; and according to the source action semantic features and the target action semantic features, optimizing the target role action 3D sequence by taking a second loss function as semantic consistency constraint, so that the target role action 3D sequence is aligned with the source action in a semantic feature hidden space, and the action semantics are kept in the redirection process.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for optimizing a 3D sequence of target character actions and an electronic device. Background Art

[0002] Motion retargeting algorithms have been integrated into various motion capture software, but they often suffer from semantic loss. While some solutions have emerged to address this semantic loss, they cannot maintain the semantics of the motion during the retargeting process. Summary of the Invention

[0003] In view of this, an embodiment of the present invention provides a method and electronic device for optimizing a 3D sequence of target character actions. According to the semantic features of the source action and the semantic features of the target action, the 3D sequence of the target character action is optimized by using a second loss function as a semantic consistency constraint, so that the target character action is aligned with the source action in the semantic feature latent space, thereby achieving the preservation of action semantics during the redirection process.

[0004] In a first aspect, an embodiment of the present invention provides a method for optimizing a 3D sequence of target character actions, the method comprising:

[0005] Perform skeleton-level redirection on the source skeleton action sequence to obtain the target skeleton action sequence;

[0006] The source character image is driven by the source skeleton action sequence to obtain a source character action 3D sequence, and the target character image is driven by the target skeleton action sequence to obtain a target character action 3D sequence;

[0007] Processing the source character action 3D sequence and the target character action 3D sequence based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features;

[0008] Based on the semantic features of the source and target actions, the target character action 3D sequence is optimized using a second loss function as a semantic consistency constraint. This embodiment of the present invention splits the retargeting problem into coarse-grained alignment at the skeletal level and fine-grained fine-tuning at the semantic level, thereby preserving the semantics of the actions during the retargeting process.

[0009] In conjunction with the first aspect, in certain implementations of the first aspect, before processing the source character action 3D sequence and the target character action 3D sequence based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features, the method further includes:

[0010] Optimizing the target character action 3D sequence through geometric constraints: After obtaining the target character action 3D sequence, the embodiment of the present invention further introduces geometric constraints to optimize the target character action 3D sequence, thereby preventing the model from being cut through after redirection.

[0011] In conjunction with the first aspect, in certain implementations of the first aspect, redirecting the source skeletal motion sequence at the skeletal level to obtain the target skeletal motion sequence includes:

[0012] Determine the first loss function according to the reconstruction constraint, the cycle consistency constraint, the adversarial constraint and the relative position constraint of the skeleton joint points;

[0013] According to the first loss function, the source skeleton action sequence is input into the skeleton-level redirection network to obtain the target skeleton action sequence. The skeleton-level redirection network adopts the Encoder / Decoder paradigm, and its specific network structure can be selected in different ways (such as graph neural networks, convolutional neural networks, etc.). The constraints of the loss function are crucial for good initialization results. The first loss function used in this stage is as follows:

[0014] L skel =λ r L rec +λ c L cyc +λ a L adv +λ j L jdm

[0015] Where, L skel is the first loss function, L rec To reconstruct the constraints, L cyc is the cycle consistency constraint, L adv To combat the constraints, L jdm is the relative position constraint of the skeleton joints, λ r is the weight of the reconstruction constraint, λ c is the weight of the cycle consistency constraint, λ a is the weight of the adversarial constraint, λ j The weight of the relative position constraint of the skeleton joints.

[0016] In conjunction with the first aspect, in certain implementations of the first aspect, driving a source character image using the source skeletal motion sequence to obtain a source character motion 3D sequence, and driving a target character image using the target skeletal motion sequence to obtain a target character motion 3D sequence, includes:

[0017] Based on a differentiable skinning algorithm, the source character image is driven by the source skeleton action sequence to obtain a source character action 3D sequence;

[0018] Based on the differentiable skinning algorithm, the target character image is driven by the target skeleton action sequence to obtain a target character action 3D sequence.

[0019] In conjunction with the first aspect, in certain implementations of the first aspect, the processing of the source character action 3D sequence and the target character action 3D sequence based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features includes:

[0020] Obtaining a source character action picture sequence by performing differentiable rendering on the source character action 3D sequence;

[0021] Obtaining a target character action picture sequence by performing differentiable rendering on the target character action 3D sequence;

[0022] Inputting the source character action picture sequence into a visual language model to obtain the source action semantic features;

[0023] The target character action picture sequence is input into the visual language model to obtain the target action semantic features. The embodiment of the present invention uses differentiable rendering and a visual language model as supervision to align action semantics. In the absence of semantic annotation, it can significantly improve the effect of preserving action semantics after redirection. The embodiment of the present invention uses differentiable rendering to convert the action domain to the image domain, and uses a visual language model to extract the high-level semantics of the action, so that the model can understand the high-level semantics of the action during the redirection process.

[0024] In conjunction with the first aspect, in certain implementations of the first aspect, inputting the source character action picture sequence into a visual language model to obtain the source action semantic features includes:

[0025] Encoding the source character action picture sequence through an image encoder to obtain a first picture feature;

[0026] Performing visual question answering processing on the first image feature to obtain a first text;

[0027] The first text is encoded using a large language model encoder to obtain the source action semantic features. In this embodiment of the present invention, after inputting the action image sequence into the visual language model, the visual language model first encodes the action image sequence using an image encoder to obtain image features. Visual question answering is then performed on the image features to obtain text. The text is then encoded using the large language model encoder to obtain the latent space features of the action, namely, the action semantic features.

[0028] Usually, visual question answering returns text, but the large language model latent space in the visual language model is more expressive and has a higher information density. Therefore, when performing action semantic alignment, the text returned by visual question answering is not selected. Instead, the features in the latent space output by the large language model encoder are selected as the semantic features of the action, and semantic consistency constraints are performed on these features.

[0029] In conjunction with the first aspect, in certain implementations of the first aspect, inputting the target character action picture sequence into a visual language model to obtain the target action semantic features includes:

[0030] Encoding the target character action picture sequence through an image encoder to obtain a second picture feature;

[0031] Performing visual question answering processing on the second image feature to obtain a second text;

[0032] The second text is encoded using a large language model encoder to obtain the target action semantic features. In this embodiment of the present invention, after inputting the action image sequence into the visual language model, the visual language model first encodes the action image sequence using an image encoder to obtain image features. Visual question answering is then performed on the image features to obtain text. The text is then encoded using the large language model encoder to obtain latent space features of the action, namely, the action semantic features.

[0033] Usually, visual question answering returns text, but the large language model latent space in the visual language model is more expressive and has a higher information density. Therefore, when performing action semantic alignment, the text returned by visual question answering is not selected. Instead, the features in the latent space output by the large language model encoder are selected as the semantic features of the action, and semantic consistency constraints are performed on these features.

[0034] In conjunction with the first aspect, in certain implementations of the first aspect, performing visual question-answering processing on the first image feature to obtain the first text includes:

[0035] Based on the first image feature, a question related to the hand position is asked as a first pre-guiding question to obtain an answer to the first pre-guiding question;

[0036] The answer to the first pre-guiding question is used as context information to continue asking questions related to the semantics of the action, thereby obtaining the first text. When performing visual question answering, the embodiment of the present invention takes into account that the semantics expressed by the hands when performing an action account for the majority of the overall semantics of the action. Therefore, the visual language model is first asked targeted questions related to the hand position as pre-guiding questions, allowing the visual language model to pay more attention to the semantics of the hand area. Subsequently, further questions related to the semantics of the action are asked using the answer to the pre-guiding question as context information, thereby obtaining a more complete semantic feature of the action.

[0037] In conjunction with the first aspect, in certain implementations of the first aspect, performing visual question-answering processing on the second image feature to obtain the second text includes:

[0038] Based on the second image feature, a question related to the hand position is asked as a second pre-guiding question to obtain an answer to the second pre-guiding question;

[0039] The answer to the second pre-guiding question is used as context information to continue asking questions related to the semantics of the action, thereby obtaining the second text. When performing visual question answering, the embodiment of the present invention takes into account that the semantics expressed by the hands when performing an action account for the majority of the overall semantics of the action. Therefore, the visual language model is first asked targeted questions related to the hand position as pre-guiding questions, allowing the visual language model to pay more attention to the semantics of the hand area. Subsequently, further questions related to the semantics of the action are asked using the answer to the pre-guiding question as context information, thereby obtaining a more complete semantic feature of the action.

[0040] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a program, and when the processor runs the program, the electronic device executes the steps of the method described above.

[0041] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program, and when the program is executed by a device, the electronic device executes the method as described above.

[0042] In a fourth aspect, an embodiment of the present invention provides a computer program product, which includes a program. When the program is run on an electronic device or any at least one processor, the electronic device performs the functions / steps in the above method.

[0043] In the technical solution of the target character action 3D sequence optimization method and electronic device provided by the embodiment of the present invention, the method includes: redirecting the source skeleton action sequence at the skeleton level to obtain a target skeleton action sequence; driving the source character image through the source skeleton action sequence to obtain a source character action 3D sequence, and driving the target character image through the target skeleton action sequence to obtain a target character action 3D sequence; processing the source character action 3D sequence and the target character action 3D sequence based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features; based on the source action semantic features and the target action semantic features, optimizing the target character action 3D sequence by using a second loss function as a semantic consistency constraint, so that the target character action 3D sequence is aligned with the source action in the semantic feature latent space, thereby achieving the preservation of action semantics during the redirection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of an action redirection scheme;

[0045] Figure 2 A schematic diagram of another action redirection scheme;

[0046] Figure 3 A schematic diagram of another action redirection solution;

[0047] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention;

[0048] Figure 5 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of an application scenario of the target character action 3D sequence optimization method provided by an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of the overall framework of the target character action 3D sequence optimization method provided by an embodiment of the present invention;

[0051] Figure 8 for Figure 7 Schematic diagram of the detailed framework of the visual language model in [1].

[0052] Figure 9 A flowchart of a method for optimizing a 3D sequence of target character actions provided by an embodiment of the present invention;

[0053] Figure 10 for Figure 9 The source skeleton action sequence is redirected at the skeleton level to obtain the flow chart of the target skeleton action sequence;

[0054] Figure 11 for Figure 9 A flowchart of obtaining a source character action 3D sequence by driving a source character image through a source skeleton action sequence, and obtaining a target character action 3D sequence by driving a target character image through a target skeleton action sequence;

[0055] Figure 12 for Figure 9 In the process, the source character action 3D sequence and the target character action 3D sequence are processed based on differentiable rendering and visual language model to obtain the flow chart of the source action semantic features and the target action semantic features;

[0056] Figure 13 for Figure 12 In the flowchart, the source character action picture sequence is input into the visual language model to obtain the semantic features of the source action;

[0057] Figure 14 for Figure 13 Perform visual question answering on the first image feature to obtain a flowchart of the first text;

[0058] Figure 15 for Figure 12 In the flowchart, the target character action picture sequence is input into the visual language model to obtain the semantic features of the target action;

[0059] Figure 16 for Figure 15 Perform visual question answering on the second image feature to obtain a flowchart of the second text;

[0060] Figure 17 A flowchart of another method for optimizing a 3D sequence of target character actions provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0062] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0063] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0064] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0065] To better understand the embodiments of the present application, the terms or concepts that may be involved in the embodiments are explained below.

[0066] 1. Source Motion Sequence: A sequence consisting of source skeleton structure and source motion information, which is the action sequence that needs to be redirected.

[0067] 2. Target Motion Sequence: A sequence consisting of the target skeleton structure and the redirected motion information.

[0068] 3. Motion Retargeting: The process of migrating the actions of the source action sequence to the target skeleton structure so that the target skeleton actions can be consistent with the source skeleton actions.

[0069] 4. Visual Question Answering (VQA): By providing an image and a question about the image content, the model is expected to output the answer to the question.

[0070] 5. Visual Language Model (VLM): A multimodal model that can align images and natural language in a latent space and perform question-answering tasks about images using natural language.

[0071] 6. Encoder: A module that encodes information from the original domain to the latent space domain, usually accompanied by information compression.

[0072] 7. Decoder: A module that decodes information from the latent space domain to the original domain.

[0073] 8. Latent Space: refers to an abstract, usually high-dimensional space used to represent and store the potential features or structures of data.

[0074] 9. Differentiable Skinning: Skinning algorithms use the character's skeletal motion information (such as rotation angles) to deform the corresponding character's geometric information (such as vertices and triangles) so that the character's pose matches the skeletal pose. Differentiable skinning means that the calculation functions throughout the skinning process meet differentiability constraints.

[0075] 10. Differentiable Rendering: Rendering is the process of mapping 3D content onto a 2D surface through functional transformations. Differentiable rendering means that the computational functions throughout the rendering process satisfy differentiability constraints.

[0076] Digital avatars have become an essential component in industries like gaming and film production. Motion capture technology is widely used to more accurately replicate an actor's performance on a digital avatar. However, due to the ever-changing appearance and skeletal structure of digital avatars, the motion data obtained through motion capture is often not directly usable and must be processed through a motion retargeting module to be correctly mapped to the digital avatar.

[0077] Although motion redirection algorithms have been integrated into various motion capture software, they often face the following limitations:

[0078] (1) Only the similarities of the positions and motion trajectories of the skeleton joints before and after redirection are considered, which may lead to the phenomenon of target character penetrating the model after redirection;

[0079] (2) The semantic information contained in the action is ignored during the redirection process (for example, a specific posture represents a specific semantics in sign language), resulting in the redirection result losing the original semantics.

[0080] Regardless of the above limitations, a large amount of manual work is currently required for post-production repair, which greatly increases the cost of content production and limits production efficiency.

[0081] For example, Figure 1 A schematic diagram of an action redirection scheme is shown in FIG. Figure 1 As shown in the figure, this action redirection solution directly copies the action information (such as rotation angle, local displacement, etc.) of each skeletal joint point in the source skeleton action sequence, and then pastes it to the corresponding target skeletal joint point in the target skeleton topology. This solution is simple and direct, and can achieve good redirection results when the source action skeleton and target action skeleton structures are similar. However, the effect will be significantly reduced when the source action skeleton and target action skeleton structures are different. Moreover, it does not take into account the semantic information carried by the action, and the redirection results are prone to problems such as model penetration, inaccuracy, and semantic loss.

[0082] For example, Figure 2 A schematic diagram of another action redirection scheme is shown in FIG. Figure 2 As shown in Figure 2, this action redirection scheme achieves redirection by training an encoder / decoder network and applying constraints in the latent space. Redirection focuses solely on the similarity between the positions of skeletal joints or the trajectory of the action before and after redirection. This scheme typically optimizes network parameters through cycle consistency loss and adversarial learning. It achieves good redirection results when the source and target action skeletons are structurally similar. However, this redirection fails to consider the semantic information of the action and fails to maintain the semantics of the action before and after redirection.

[0083] Although some solutions have emerged to address the problem of semantic loss, they cannot maintain the semantics of actions during the redirection process.

[0084] For example, Figure 3 A schematic diagram of another action redirection solution is shown in FIG. Figure 3 As shown in Figure 2, this motion redirection scheme uses joint self-contact or the relative relationship between bones as a manually defined semantic constraint during redirection. After redirection, the locations where these constraints are applied are more closely aligned with the source motion. However, these manually defined constraints are local and lack high-level semantic information, making it difficult to maintain the semantics of the motion before and after migration.

[0085] In response to the above-mentioned problem of being unable to maintain the semantics of actions during the redirection process, an embodiment of the present invention proposes a method and electronic device for optimizing a target character action 3D sequence, the method comprising: redirecting a source skeleton action sequence at the skeleton level to obtain a target skeleton action sequence; driving a source character image through the source skeleton action sequence to obtain a source character action 3D sequence, and driving a target character image through the target skeleton action sequence to obtain a target character action 3D sequence; processing the source character action 3D sequence and the target character action 3D sequence based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features; based on the source action semantic features and the target action semantic features, optimizing the target character action 3D sequence by using a second loss function as a semantic consistency constraint to align it with the source action in the semantic feature latent space, thereby automatically maintaining the semantics of the action during the action redirection process, thereby achieving the preservation of the action semantics during the redirection process.

[0086] The target character action 3D sequence optimization method provided in the embodiments of this application is a fundamental capability for digital image content production and can also provide guidance data for robot behavior learning. It can be used in various end-to-end or cloud-based testing scenarios and can be applied to electronic devices such as smartphones, tablets, computers, smart large screens, and robots. The sales model can be software or hardware that includes the software.

[0087] For example, Figure 4 A schematic structural diagram of the electronic device 100 is shown.

[0088] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0089] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0090] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0091] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0092] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0093] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0094] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0095] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0096] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0097] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0098] The mobile communication module 150 can provide solutions for wireless communications, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter, amplify, and perform other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0099] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0100] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc., applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be transmitted from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0101] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite-based augmentation system (SBAS).

[0102] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0103] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-LED, a Micro-o-LED, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0104] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0105] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization for image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0106] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0107] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0108] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0109] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0110] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0111] The internal memory 121 can be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0112] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0113] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0114] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0115] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0116] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0117] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130, or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0118] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0119] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0120] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0121] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0122] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0123] Figure 5 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0124] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0125] The application layer can include a series of application packages.

[0126] like Figure 5 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0127] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0128] like Figure 5 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0129] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0130] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0131] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0132] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0133] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0134] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0135] Android Runtime includes core libraries and a virtual machine. Android Runtime is responsible for scheduling and management of the Android system.

[0136] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0137] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0138] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0139] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0140] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0141] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0142] A 2D graphics engine is a drawing engine for 2D drawings.

[0143] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0144] The overall process of the constraint problem reasoning method provided in the embodiment of the present application will be specifically described below with reference to the accompanying drawings.

[0145] Figure 6 A schematic diagram of an application scenario of the target character action 3D sequence optimization method provided by an embodiment of the present invention.

[0146] like Figure 6As shown, users can provide various forms of motion data as motion sources (also called source motion data) through electronic devices. For example, users can capture motion through video or through professional equipment (such as optical motion capture, inertial motion capture), or they can use animation software (such as Blender, Maya, etc.) to produce motion or use Artificial Intelligence Generated Content (AIGC) algorithms to directly generate motion sources.

[0147] In addition, if Figure 6 As shown, the user also needs to provide the target person (such as a virtual digital human or humanoid robot) for redirection and obtain the target person's relevant geometric information through relevant software. For example, the target person can be a virtual digital human or humanoid robot. For example, the target person's relevant geometric information can be the target person's skeletal topology and surface modeling.

[0148] like Figure 6 As shown, after obtaining the source action and the target character's related geometric information, the target character action 3D sequence optimization method provided by the embodiment of the present invention can obtain the corresponding target skeleton motion data to drive the target character, and then return the driving result to the user.

[0149] For example, the driving result can be rendered into a 2D video or a 3D model with animation, etc.

[0150] Example 1

[0151] Figure 7 A schematic diagram of the overall framework of the target character action 3D sequence optimization method provided by an embodiment of the present invention.

[0152] like Figure 7 The overall framework of the target character action 3D sequence optimization method provided by the embodiment of the present invention is divided into the following two stages:

[0153] Phase 1: Skeleton-level redirection.

[0154] The skeleton-level redirection provides a better initialization result for the semantics-preserving redirection in phase 2. Figure 7 As shown in Figure 3, at this stage, the input of the skeleton-level redirection network is the source skeleton action sequence, and the output is the target skeleton action sequence.

[0155] The skeleton-level redirection network adopts the Encoder / Decoder paradigm, and its specific network structure can be selected in different ways (such as graph neural networks, convolutional neural networks, etc.). The constraints of the loss function are crucial for good initialization results. The first loss function used in this stage is as follows:

[0156] L skel =λ r L rec +λ c L cyc +λ a L adv +λ j L jdm

[0157] Where, L skel is the first loss function, L rec To reconstruct the constraints, L cyc is the cycle consistency constraint, L adv To combat the constraints, L jdm is the relative position constraint of the skeleton joints, λ r is the weight of the reconstruction constraint, λ c is the weight of the cycle consistency constraint, λ a is the weight of the adversarial constraint, λ j The weight of the relative position constraint of the skeleton joints.

[0158] Phase 2: Semantic Preserving Redirection.

[0159] like Figure 7 As shown, after obtaining the initial skeletal retargeting results from stage one, the skeletal motion information is first used in conjunction with a differentiable skinning algorithm to drive the source and target character images, obtaining 3D motion sequences for the corresponding images. The initial skeletal retargeting results from stage one include a source skeletal motion sequence and a target skeletal motion sequence. Specifically, based on the differentiable skinning algorithm, the source skeletal motion sequence is used to drive the source character image to obtain the source character motion 3D sequence; and based on the differentiable skinning algorithm, the target skeletal motion sequence is used to drive the target character image to obtain the target character motion 3D sequence.

[0160] Then, if Figure 7 As shown, based on the differentiable rendering method, a 3D action sequence is rendered into a 2D image sequence, which is then fed into a visual language model. The high-level semantic features expressed in the action are then captured through the cognitive capabilities of the visual language model. Specifically, differentiable rendering is performed on the 3D sequence of source character actions to obtain a source character action image sequence, which is then fed into the visual language model to obtain the source action semantic features. Differentiable rendering is performed on the 3D sequence of target character actions to obtain a target character action image sequence, which is then fed into the visual language model to obtain the target action semantic features.

[0161] Then, if Figure 7As shown, after respectively obtaining the semantic features of the source action and the semantic features of the target action, the 3D sequence of the target character action is optimized by the semantic consistency constraint so that it is aligned with the source action in the semantic feature latent space, thereby achieving the preservation of the action semantics during the redirection process. In an embodiment of the present invention, the second loss function is used as a semantic consistency constraint to optimize the 3D sequence of the target character action based on the semantic features of the source action and the semantic features of the target action. The source action semantic features and the target action semantic features serve as inputs of the second loss function. Exemplarily, the second loss function is the mean square error loss function (MSE Loss), which is also called L2 Loss.

[0162] like Figure 7 As shown, after obtaining the target character action 3D sequence, the embodiment of the present invention further introduces geometric constraints to optimize the target character action 3D sequence, which can prevent the model from being penetrated after redirection.

[0163] Figure 8 for Figure 7 Schematic diagram of the detailed framework of the visual language model in . Figure 8 As shown in the figure, after the action picture sequence is input into the visual language model, the visual language model first encodes the action picture sequence through the image encoder to obtain picture features, and then performs visual question answering on the picture features to obtain text; then the text is encoded through the large language model encoder to obtain the latent space features of the action, that is, the action semantic features.

[0164] Exemplarily, if the action picture sequence is a source character action picture sequence, after the source character action picture sequence is input into the visual language model, the visual language model first encodes the source character action picture sequence through an image encoder to obtain a first picture feature; performs visual question-answering processing on the first picture feature to obtain a first text; and encodes the first text through a large language model encoder to obtain the latent space features of the source action, that is, the semantic features of the source action.

[0165] Exemplarily, if the action picture sequence is a target character action picture sequence, after the target character action picture sequence is input into the visual language model, the visual language model first encodes the target character action picture sequence through an image encoder to obtain a second picture feature; performs visual question answering processing on the second picture feature to obtain a second text; and encodes the second text through a large language model encoder to obtain the latent space features of the target action, that is, the semantic features of the target action.

[0166] The embodiments of the present invention have the following characteristics when extracting action semantic features through a visual language model:

[0167] 1. When conducting visual question answering, considering that the semantics expressed by the hands during an action account for a large portion of the overall action semantics, we first ask the visual language model targeted questions related to hand position. This serves as a pre-guided question, allowing the visual language model to pay more attention to the semantics of the hand area. The answers to these pre-guided questions then serve as context for further questions related to the action semantics, thereby obtaining a more complete action semantic feature.

[0168] 2. Visual question answering typically returns text, but the large language model latent space in the visual language model is more expressive and has greater information density. Therefore, when performing action semantic alignment, the text returned by the visual question answering is not selected. Instead, the features in the latent space output by the large language model encoder are selected as the semantic features of the action, and semantic consistency constraints are applied to these features.

[0169] In summary, the target character action 3D sequence optimization method provided by the embodiment of the present invention has the following four beneficial effects:

[0170] First, the overall framework of the target character action 3D sequence optimization method provided by the embodiment of the present invention includes a two-stage redirection framework, which splits the redirection problem into coarse-grained alignment at the skeletal level and fine-grained fine-tuning at the semantic level, thereby achieving the preservation of action semantics during action redirection.

[0171] Secondly, in the second stage, the target character action 3D sequence optimization method provided by the embodiment of the present invention aligns the action semantics based on differentiable rendering and visual language model as supervision. In the absence of semantic annotation, it can significantly improve the effect of preserving the action semantics after redirection.

[0172] Thirdly, an embodiment of the present invention uses differentiable rendering to convert the action domain to the image domain, and uses a visual language model to extract the high-level semantics of the action, so that the model can understand the high-level semantics of the action during the redirection process.

[0173] Fourthly, the embodiment of the present invention imposes semantic consistency constraints on actions, and can maintain the semantics of the action after redirection.

[0174] Table 1 shows the experimental data metrics of the target character action 3D sequence optimization method provided by an embodiment of the present invention and other methods. In Table 1, Source represents the source action; GT represents the de facto standard (the result of human annotation); Copy represents direct copy; NKN is the abbreviation of the paper "Neural Kinematic Networks for Unsupervised Motion Retargeting"; SAN is the abbreviation of the paper "Skeleton-aware networks for deep motion retargeting"; R2ET is the abbreviation of the paper "Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry"; and Ours represents the target character action 3D sequence optimization method provided by an embodiment of the present invention.

[0175] In Table 1, the mean square error (MSE) and local mean square error (MSE lc ) reflects the error between the predicted joint positions and the true values ​​of the redirected action. Pen.% represents the penetration rate. The image-text matching score (ITM) and semantic consistency loss (SCL) both reflect the semantic match between the predicted and true values ​​of the redirected action. The Frechet Inception Distance (FID) reflects the authenticity of the target character's action image sequence. Lower values ​​(↓) represent better results, while higher values ​​(↑) represent better results.

[0176] As shown in Table 1, compared with other methods, the target character action 3D sequence optimization method provided by the embodiment of the present invention has a smaller error between the predicted joint position and the true value of the redirected action, the smallest penetration ratio, a higher semantic match between the predicted action and the true value action after redirection, and the highest authenticity of the target character action image sequence.

[0177] Table 1 Experimental data indicators of the target character action 3D sequence optimization method provided by the embodiment of the present invention and other methods

[0178] method MSE↓ <![CDATA[MSE lc ↓]]> Pen.%↓ ITM↑ FID↓ SCL↓ Source - - 4.43 0.796 - - GT - - 9.06 0.582 26.99 1.331 Copy - 0.005 9.03 0.581 26.58 1.327 NKN 0.326 0.231 8.71 0.575 27.79 1.414 SAN 0.435 0.255 9.74 0.561 28.33 1.448 R2ET 0.499 0.496 7.62 0.643 5.469 0.405 Ours 0.284 0.229 3.50 0.680 0.436 0.143

[0179] based on Figure 7 and Figure 8 The framework shown, Figure 9 Flowchart of a method for optimizing a 3D sequence of target character actions provided by an embodiment of the present invention. Figure 6 As shown, the method includes: steps 202 to 208.

[0180] Step 202: Redirect the source skeleton action sequence at the skeleton level to obtain a target skeleton action sequence.

[0181] In some possible embodiments, such as Figure 10 As shown, step 202 specifically includes: step 2022-step 2024.

[0182] Step 2022: Determine a first loss function based on the reconstruction constraint, the cycle consistency constraint, the adversarial constraint, and the relative position constraint of the skeletal joints.

[0183] For example, the constraints of the first loss function are as follows:

[0184] L skel =λ r L rec +λ c L cyc +λ a L adv +λ j L jdm

[0185] Where, L skel is the first loss function, L rec To reconstruct the constraints, L cyc is the cycle consistency constraint, L adv To combat the constraints, L jdm is the relative position constraint of the skeleton joints, λ r is the weight of the reconstruction constraint, λ c is the weight of the cycle consistency constraint, λ a is the weight of the adversarial constraint, and λj is the weight of the relative position constraint of the skeleton joints.

[0186] Step 2024: According to the constraints of the first loss function, the source skeleton action sequence is input into the skeleton-level redirection network to obtain the target skeleton action sequence.

[0187] For example, the skeleton-level redirection network adopts the Encoder / Decoder paradigm, and its specific network structure can have different choices (such as graph neural network, convolutional neural network, etc.).

[0188] Step 204: Drive the source character image through the source skeleton action sequence to obtain a source character action 3D sequence, and drive the target character image through the target skeleton action sequence to obtain a target character action 3D sequence.

[0189] In some possible embodiments, such as Figure 11 As shown, step 204 specifically includes: step 2042-step 2044.

[0190] Step 2042: Based on the differentiable skinning algorithm, the source character image is driven by the source skeleton action sequence to obtain a source character action 3D sequence.

[0191] Step 2044: Based on the differentiable skinning algorithm, the target character image is driven by the target skeleton action sequence to obtain a target character action 3D sequence.

[0192] It should be understood that step 2042 and step 2044 may be executed simultaneously or at different times, and the embodiment of the present invention does not specifically limit the execution order of step 2042 and step 2044.

[0193] Step 206 : Process the source character action 3D sequence and the target character action 3D sequence based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features.

[0194] In some possible embodiments, such as Figure 12 As shown, step 206 specifically includes: steps 2062 to 2068.

[0195] Step 2062: Obtain a source character action image sequence by performing differentiable rendering on the source character action 3D sequence.

[0196] Step 2064: Obtain a target character action image sequence by performing differentiable rendering on the target character action 3D sequence.

[0197] It should be understood that step 2062 and step 2064 may be executed simultaneously or at different times, and the embodiment of the present invention does not specifically limit the execution order of step 2062 and step 2064.

[0198] Step 2066: Input the source character action picture sequence into the visual language model to obtain the source action semantic features.

[0199] In some possible embodiments, such as Figure 13 As shown, step 2066 specifically includes: step A2-step A6.

[0200] Step A2: Encode the source character action picture sequence through an image encoder to obtain a first picture feature.

[0201] Step A4: Perform visual question answering processing on the first image feature to obtain a first text.

[0202] In some possible embodiments, such as Figure 14 As shown, step A4 specifically includes: step A42-step A44.

[0203] Step A42: Based on the first image feature, a question related to the hand position is asked as a first pre-guiding question to obtain an answer to the first pre-guiding question.

[0204] Step A44: Use the answer to the first pre-question as context information, continue to ask questions related to the action semantics, and obtain the first text.

[0205] Step A6: Encode the first text using a large language model encoder to obtain source action semantic features.

[0206] Step 2068: Input the target character action picture sequence into the visual language model to obtain the target action semantic features.

[0207] In some possible embodiments, such as Figure 15 As shown, step 2068 specifically includes: step B2-step B6.

[0208] Step B2: Encode the target character action picture sequence through an image encoder to obtain a second picture feature.

[0209] Step B4: Perform visual question answering processing on the second image features to obtain a second text.

[0210] In some possible embodiments, such as Figure 16 As shown, step B4 specifically includes: step B42-step B44.

[0211] Step B42: Based on the second image feature, a question related to the hand position is asked as a second pre-guiding question to obtain an answer to the second pre-guiding question.

[0212] Step B44: Use the answer to the second pre-guiding question as context information, continue to ask questions related to the action semantics, and obtain a second text.

[0213] Step B6: Encode the second text using a large language model encoder to obtain semantic features of the target action.

[0214] It should be understood that step 2066 and step 2068 may be executed simultaneously or at different times, and the embodiment of the present invention does not specifically limit the execution order of step 2066 and step 2068.

[0215] Step 208: Optimize the target character action 3D sequence based on the source action semantic features and the target action semantic features by using the second loss function as a semantic consistency constraint.

[0216] Optionally, Figure 17 A flowchart of another target character action 3D sequence optimization method provided by an embodiment of the present invention is shown in FIG. Figure 17As shown, between step 204 and step 206 , the target character action 3D sequence optimization method provided by the embodiment of the present invention further includes: step 205 .

[0217] Step 205: Optimize the target character's 3D action sequence through geometric constraints.

[0218] Therefore, if Figure 17 As shown, step 206 specifically includes: processing the source character action 3D sequence and the target character action 3D sequence optimized by geometric constraints based on differentiable rendering and visual language model to obtain source action semantic features and target action semantic features.

[0219] After obtaining the target character action 3D sequence, the embodiment of the present invention further introduces geometric constraints to optimize the target character action 3D sequence, thereby preventing the model from being penetrated after redirection.

[0220] In the technical solution of the target character action 3D sequence optimization method provided by an embodiment of the present invention, the method includes: redirecting the source skeleton action sequence at the skeleton level to obtain a target skeleton action sequence; driving the source character image through the source skeleton action sequence to obtain a source character action 3D sequence, and driving the target character image through the target skeleton action sequence to obtain a target character action 3D sequence; processing the source character action 3D sequence and the target character action 3D sequence based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features; based on the source action semantic features and the target action semantic features, optimizing the target character action 3D sequence by using a second loss function as a semantic consistency constraint, so that the target character action 3D sequence is aligned with the source action in the semantic feature latent space, thereby achieving the preservation of action semantics during the redirection process.

[0221] An embodiment of the present application provides an electronic device, which may be a terminal device or a circuit device built into the terminal device. The electronic device may be used to perform the functions / steps of the above method embodiment.

[0222] An embodiment of the present application provides a computer-readable storage medium, in which a program is stored. When the program is executed on a terminal device, the terminal device executes the functions / steps in the above-mentioned method embodiment.

[0223] An embodiment of the present application further provides a computer program product comprising a program. When the computer program product is run on an electronic device or any at least one processor, the electronic device executes the functions / steps in the above method embodiment.

[0224] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0225] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0226] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0227] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0228] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A method for optimizing a 3D sequence of target character actions, characterized in that: The method comprises: Perform skeleton-level redirection on the source skeleton action sequence to obtain the target skeleton action sequence; The source character image is driven by the source skeleton action sequence to obtain a source character action 3D sequence, and the target character image is driven by the target skeleton action sequence to obtain a target character action 3D sequence; Processing the source character action 3D sequence and the target character action 3D sequence based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features; According to the source action semantic features and the target action semantic features, the target character action 3D sequence is optimized by using a second loss function as a semantic consistency constraint.

2. The method according to claim 1, characterized in that Before the source character action 3D sequence and the target character action 3D sequence are processed based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features, the method further includes: The target character action 3D sequence is optimized using geometric constraints.

3. The method according to claim 1 or 2, characterized in that The step of performing skeleton-level redirection on the source skeleton action sequence to obtain a target skeleton action sequence includes: Determine the first loss function according to the reconstruction constraint, the cycle consistency constraint, the adversarial constraint and the relative position constraint of the skeleton joint points; According to the first loss function, the source skeleton action sequence is input into a skeleton-level redirection network to obtain the target skeleton action sequence.

4. The method according to any one of claims 1 to 3, characterized in that The step of driving a source character image by using the source skeleton action sequence to obtain a source character action 3D sequence, and driving a target character image by using the target skeleton action sequence to obtain a target character action 3D sequence, comprises: Based on a differentiable skinning algorithm, the source character image is driven by the source skeleton action sequence to obtain a source character action 3D sequence; Based on the differentiable skinning algorithm, the target character image is driven by the target skeleton action sequence to obtain a target character action 3D sequence.

5. The method according to any one of claims 1 to 4, characterized in that The processing of the source character action 3D sequence and the target character action 3D sequence based on differentiable rendering and a visual language model to obtain source action semantic features and target action semantic features includes: Obtaining a source character action picture sequence by performing differentiable rendering on the source character action 3D sequence; Obtaining a target character action picture sequence by performing differentiable rendering on the target character action 3D sequence; Inputting the source character action picture sequence into a visual language model to obtain the source action semantic features; The target character action picture sequence is input into a visual language model to obtain the target action semantic features.

6. The method according to claim 5, characterized in that Inputting the source character action picture sequence into a visual language model to obtain the source action semantic features includes: Encoding the source character action picture sequence through an image encoder to obtain a first picture feature; Performing visual question answering processing on the first image feature to obtain a first text; The first text is encoded using a large language model encoder to obtain the source action semantic features.

7. The method according to claim 5 or 6, characterized in that The step of inputting the target character action picture sequence into a visual language model to obtain the target action semantic features includes: Encoding the target character action picture sequence through an image encoder to obtain a second picture feature; Performing visual question answering processing on the second image feature to obtain a second text; The second text is encoded using a large language model encoder to obtain the target action semantic features.

8. The method according to claim 6, characterized in that The performing visual question answering processing on the first image feature to obtain a first text includes: Based on the first image feature, a question related to the hand position is asked as a first pre-guiding question to obtain an answer to the first pre-guiding question; The answer to the first preceding guiding question is used as context information, and further questions related to the semantics of the action are asked to obtain the first text.

9. The method according to claim 7, characterized in that The performing visual question answering processing on the second image feature to obtain a second text includes: Based on the second image feature, a question related to the hand position is asked as a second pre-guiding question to obtain an answer to the second pre-guiding question; The answer to the second preceding guiding question is used as context information, and further questions related to the semantics of the action are asked to obtain the second text.

10. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store a program, and when the processor runs the program, the electronic device executes the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and when the program is executed by an electronic device, the electronic device executes the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that The computer program product stores a program, and when the program is executed by an electronic device, the electronic device executes the method according to any one of claims 1 to 9.

Images