Video generation method

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a video generation method, a video generation device, a computer readable medium, electronic equipment and a computer program product. The method comprises the steps that a reference video and a reference image are acquired, the video content of the reference video comprises the posture change process of a first human body in a motion state, and the image content of the reference image comprises the static display effect of a second human body in a specified view angle; performing feature extraction on the reference video to obtain global attitude features and local attitude features; and generating a target video according to the global posture feature, the local posture feature and the reference image, wherein the video content of the target video comprises the posture change process of the second human body in the motion state. The motion detail expressive force generated by the video can be improved, and the global and local posture coordination of the human body can be enhanced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and specifically relates to a video generation method, a video generation device, a computer readable medium, an electronic device and a computer program product. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, portrait video automatic generation technology has become one of the hottest technology applications. This technology can simulate the changes of facial expressions and postures by converting static portraits into dynamic videos, realizing the effect of generating smooth videos from a single photo. However, the existing technology still faces many challenges in realizing portrait video generation, especially in dealing with the detailed performance of human motion postures.

[0003] One of the main bottlenecks of current portrait video generation technology is the processing of hand movements. Due to the uneven quality of training data or the lack of high-quality hand data, the generated video often has problems such as unstable movement and blurred details. Although theoretically, these problems can be improved by increasing a large amount of high-quality hand training data, in practical applications, this method faces multiple restrictions such as difficulty in ensuring data quality, huge consumption of computing resources, excessive storage requirements, and complex training strategies.

[0004] In addition, the existing technology also has deficiencies in dealing with the coordination of global and local postures of the human body. Traditional methods usually use single-view or fixed-region feature extraction methods, which are difficult to balance the overall motion fluency of the human body and the detailed performance of specific body parts (such as hands). Especially when the motion posture in the reference video needs to be transferred to the target image, the existing technology often cannot accurately capture and transfer the motion characteristics of different body parts. SUMMARY

[0005] The present application provides a video generation method, a video generation device, a computer readable medium, an electronic device and a computer program product, aiming to improve the motion detail performance of video generation and enhance the coordination of global and local postures of the human body.

[0006] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned partly through practice of the present application.

[0007] According to an aspect of some embodiments of the present application, a video generation method is provided, the method comprising: obtaining a reference video and a reference image, the video content of the reference video comprising a posture change process of a first human body in a motion state, the image content of the reference image comprising a static display effect of a second human body under a specified view angle; performing feature extraction on the reference video to obtain global posture features and local posture features, the global posture features being used to represent posture information displayed by the first human body in a global field of view area of the reference video, the local posture features being used to represent posture information displayed by the first human body in a local field of view area of the reference video; and generating a target video according to the global posture features, the local posture features, and the reference image, the video content of the target video comprising a posture change process of the second human body in a motion state.

[0008] According to an aspect of some embodiments of the present application, a video generation apparatus is provided, the apparatus comprising: an obtaining module configured to obtain a reference video and a reference image, the video content of the reference video comprising a posture change process of a first human body in a motion state, the image content of the reference image comprising a static display effect of a second human body under a specified view angle; an extracting module configured to perform feature extraction on the reference video to obtain global posture features and local posture features, the global posture features being used to represent posture information displayed by the first human body in a global field of view area of the reference video, the local posture features being used to represent posture information displayed by the first human body in a local field of view area of the reference video; a generating module configured to generate a target video according to the global posture features, the local posture features, and the reference image, the video content of the target video comprising a posture change process of the second human body in a motion state.

[0009] In some embodiments of the present application, based on the above technical solution, the local field of view area is a sub-area that dynamically changes in the global field of view area, and the local field of view area is used to capture posture information of a specified body part of the first human body; the apparatus further comprises: a positioning module configured to position the local field of view area according to the specified body part in each video frame of the reference video.

[0010] In some embodiments of the present application, based on the above technical solution, the specified body part is a hand of the first human body.

[0011] In some embodiments of the present application, based on the above technical solution, the apparatus further comprises: The recognition module is configured to perform semantic recognition on the reference video to obtain a content theme of the reference video. The selection module is configured to select, from a plurality of body parts of the first human body, a specified body part that has semantic correlation with the content theme.

[0012] In some embodiments of the present application, based on the above technical solutions, the extraction module comprises: The first frame combination module is configured to select a plurality of first video frames from the reference video to form a first video frame sequence, the first video frames being video frames that are representative for describing a global pose of the first human body; The first feature extraction module is configured to perform feature extraction on the first video frame sequence to obtain global pose features; The second frame combination module is configured to select a plurality of second video frames from the reference video to form a second video frame sequence, the second video frames being video frames that are representative for describing a local pose of the specified body part; The second feature extraction module is configured to perform feature extraction on the second video frame sequence to obtain local pose features.

[0013] In some embodiments of the present application, based on the above technical solutions, the first frame combination module is further configured to: perform pose recognition on the global field of view in each video frame of the reference video to obtain a global pose parameter of the video frame, the global pose parameter being used to represent an importance degree of the video frame for describing a global pose of the first human body; and select a plurality of first video frames from the reference video to form a first video frame sequence according to the global pose parameters, the global pose parameter of the first video frames being greater than a preset first parameter threshold.

[0014] In some embodiments of the present application, based on the above technical solutions, the second frame combination module is further configured to: perform pose recognition on the local field of view in each video frame of the reference video to obtain a local pose parameter of the video frame, the local pose parameter being used to represent an importance degree of the video frame for describing a local pose of the specified body part; and select a plurality of second video frames from the reference video to form a second video frame sequence according to the local pose parameters, the local pose parameter of the second video frames being greater than a preset second parameter threshold.

[0015] In some embodiments of the present application, based on the above technical solutions, the generation module comprises: The feature fusion module is configured to perform feature fusion on the global pose features and the local pose features to obtain human body pose features. The feature reduction module is configured to perform feature reduction on the human pose feature to obtain a target video by taking the reference image as a prompt word.

[0016] In some embodiments of the present application, based on the above technical solution, the feature fusion module is further configured to: acquire timestamps associated with the global pose feature and the local pose feature respectively, the timestamps being used to mark the positions of the video frames in the reference video; perform feature alignment on the global pose feature and the local pose feature according to the timestamps to obtain feature pairs corresponding to the same timestamp; and splice the feature pairs to form a human pose feature.

[0017] In some embodiments of the present application, based on the above technical solution, the feature reduction module is further configured to: perform layer-by-layer feature extraction on the reference image to obtain a plurality of image features corresponding to different feature depths; and perform layer-by-layer feature reduction on the human pose feature by taking a plurality of the image features as prompt words respectively to obtain a target video.

[0018] According to an aspect of an embodiment of the present application, there is provided a computer readable medium having stored thereon a computer program, which, when executed by a processor, implements the video generation method in the above technical solution.

[0019] According to an aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the video generation method in the above technical solution.

[0020] According to an aspect of an embodiment of the present application, there is provided a computer program product comprising a computer program, which, when executed by a processor, implements the video generation method in the above technical solution.

[0021] In the technical solution provided in the embodiments of the present application, the global and local pose features of the reference video are extracted and combined with the reference image to generate a target video, which solves the problems of blurred hand details and insufficient coordination of human poses in the prior art, and has the advantages of improving the expression of hand movement details and enhancing the coordination of global and local poses of the human body.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application. It is to be understood that the drawings are designed solely for purposes of illustration to be used in conjunction with the description in

[0024] Figure 1 An exemplary system architecture block diagram to which the technical solution of the present application is applied is schematically shown.

[0025] Figure 2 A flow chart of a video generation method in an embodiment of the present application is shown.

[0026] Figure 3 A flow chart of feature extraction on a reference video in an embodiment of the present application is shown.

[0027] Figure 4 A flow chart of a human video generation process in an application scenario of an embodiment of the present application is shown.

[0028] Figure 5 A flow chart of hand feature extraction in an application scenario of an embodiment of the present application is shown.

[0029] Figure 6 A flow chart of constructing a training data set for a hand feature extraction module in an application scenario of an embodiment of the present application is shown.

[0030] Figure 7 A flow chart of training a hand feature extraction module in an application scenario of an embodiment of the present application is shown.

[0031] Figure 8 A flow chart of joint training of a full model in an application scenario of an embodiment of the present application is shown.

[0032] Figure 9 A structural block diagram of a video generation apparatus provided by an embodiment of the present application is schematically shown.

[0033] Figure 10 A computer system structural block diagram of an electronic device for implementing an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0034] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art.

[0035] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the application.

[0036] In embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0037] The block diagrams shown in the accompanying drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0038] The flowcharts shown in the accompanying drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0039] Figure 1 An exemplary system architecture block diagram to which the technical solutions of the present application are applied is schematically shown.

[0040] As Figure 1As shown, the system architecture 100 can include a terminal device 110, a network 120, and a server 130. The terminal device 110 can include various electronic devices such as a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart wearable device, a smart vehicle device, a smart payment terminal, and the like. The server 130 can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The network 120 can be various connection types of communication media capable of providing a communication link between the terminal device 110 and the server 130, such as a wired communication link or a wireless communication link.

[0041] According to the implementation needs, the system architecture in the embodiments of the present application can have any number of terminal devices, networks, and servers. For example, the server 130 can be a server group composed of multiple server devices. In addition, the technical solutions provided by the embodiments of the present application can be applied to the terminal device 110, can be applied to the server 130, or can be jointly implemented by the terminal device 110 and the server 130, and the present application does not make special limitations on this.

[0042] In the related art of the present application, for the portrait video generation method, due to the lack of high-quality hand motion samples in the model training data set, the generated video hand region has the problems of incoherent motion trajectory and blurred detail texture. The global pose feature extraction process usually only focuses on the overall motion trend of the human body, and cannot effectively capture the dynamic change characteristics of the local region such as the hand, resulting in a lack of smooth transition of the hand action in the time dimension. When the reference video contains complex hand interaction actions, the probability of sudden change of the hand joint angle between the video frames generated by the existing method increases, the pixel-level jitter of the hand contour edge occurs, and the knuckle texture details gradually degrade during the motion process.

[0043] For example, in the virtual fitting room application scenario, when the user uploads a static full-body photo and a reference video containing a hand waving action, the wrist rotation angle of the generated video jumps more than 15 degrees between adjacent frames, and the finger belly area produces a mosaic effect during finger stretching. When the reference video contains a fast page turning action, the position offset of the thumb and the book page contact area in the generated video accumulates up to 20 pixels within 5 consecutive frames, resulting in a visual hand penetration phenomenon. The feature fusion module in the system architecture fails to establish the spatiotemporal correlation between the global limb motion and the local hand action, resulting in a phase difference between the hand motion trajectory and the overall body posture.

[0044] If the above problems are not solved, the abnormal motion of the generated video will significantly reduce the credibility of the user experience, and will cause misidentification of operation instructions in virtual teaching and gesture interaction systems that require accurate hand movement display. Video quality defects will also cause cross-modal feature alignment failure, increasing the cost of manual correction in subsequent video editing. The decline in system reliability will limit the application of this technology in medical rehabilitation motion simulation, sports training guidance and other fields that require strict details of the limbs.

[0045] In the face of the above problems, the application first analyzes the limitations of global pose feature extraction in traditional methods and finds that it cannot effectively capture the dynamic details of local areas such as hands. Attempts to optimize hand features by adding a local motion trajectory tracking module are found to introduce additional computational delays. Further consideration is given to decomposing video features into two levels of global and local, with global features maintaining the overall body movement continuity and local features focusing on hand joint angle changes. Experiments show that using global features alone will cause the hand motion trajectory to break, and relying solely on local features will make the limb movements lose coordination. Ultimately, it is determined that a spatiotemporal correlation mechanism between global and local features needs to be established to balance action coordination and detail precision through hierarchical feature extraction and fusion.

[0046] The video generation method, video generation device, computer readable medium, electronic device and computer program product provided by the application are described in detail below in combination with specific embodiments, aiming to improve the video generation quality and other technical solutions.

[0047] Figure 2 A flowchart of a video generation method in an embodiment of the application is shown, which can include the following steps S210 to S230.

[0048] S210: Obtain a reference video and a reference image, the video content of the reference video including a pose change process of a first human body in a motion state, and the image content of the reference image including a static display effect of a second human body at a specified angle.

[0049] S220: Extract features from the reference video to obtain global pose features and local pose features, the global pose features being used to represent pose information displayed by the first human body in a global field of view area of the reference video, and the local pose features being used to represent pose information displayed by the first human body in a local field of view area of the reference video.

[0050] S230: Generate a target video according to the global pose features, the local pose features and the reference image, the video content of the target video including a pose change process of the second human body in a motion state.

[0051] The global pose feature refers to a feature extracted from a video and representing a whole body motion trajectory and limb coordination relationship, and can be specifically implemented by using a 3D human pose estimation algorithm to perform key point detection and trajectory tracking on a video frame, so as to ensure the continuity of the subject action in the generated video.

[0052] In the embodiment of the present application, the local pose feature refers to a feature extracted from a video and representing motion details of a key part such as a hand, which can be implemented by using high-resolution local region segmentation and optical flow analysis technology, for example, to solve the hand detail distortion problem caused by the global feature blur.

[0053] The reference video refers to a dynamic data source containing a first human motion pose change, which can be implemented by using a multi-view motion capture system to collect a standardized action sequence, for example, to provide basic data for modeling the motion rule of the generated target video.

[0054] The reference image refers to a single-frame image containing a second human static display effect, which can be implemented by using a multi-angle high-definition portrait photo, for example, to provide a visual reference and identity feature of the target view for the generated target video.

[0055] The core innovation of the present application lies in the cooperative extraction mechanism of the global pose feature and the local pose feature, which preserves the static identity feature of the target human body while migrating the whole motion rule and hand detail action in the reference video to the generated video in layers, thereby achieving the dual improvement of hand motion stability and detail clarity without the need for a large amount of high-quality training data.

[0056] The main inventive principle of the present application is to first obtain a reference video and a reference image. The reference video contains the motion pose change process of a first human body and provides a dynamic behavior paradigm. The reference image contains the static display effect of a second human body at a specified view angle and serves as a visual reference.

[0057] Then, feature extraction is performed on the reference video to obtain global pose features and local pose features. The global pose feature extraction captures the whole body motion trajectory and limb coordination relationship, ensuring the continuity of the subject action in the generated video. The local pose feature extraction focuses on the motion details of key parts such as hands, solving the detail distortion problem caused by the global feature blur.

[0058] Finally, a target video is generated based on the extracted global pose features, local pose features, and reference image. The global features and local features are fused, and the view angle information of the reference image is combined to reconstruct a target video that conforms to the motion rule based on the preservation of the static features of the second human body. The content of the target video includes the pose change process of the second human body in the motion state, achieving the dual improvement of hand action stability and detail clarity.

[0059] By hierarchical feature extraction and fusion, a spatio-temporal correlation mechanism of global and local features is established to achieve the balance between action coordination and detail accuracy. Global features maintain the coherence of the overall body movement, and local features focus on the change of hand joint angles, avoiding the problem of broken hand movement trajectory caused by using global features alone or losing limb action coordination caused by relying solely on local features.

[0060] In an optional embodiment, the scheme of the present application is implemented as follows: First, the reference video and the reference image are obtained. The reference video can be a video segment containing the complete action sequence of a person, such as the action process of a person waving or turning a book. The reference image can be a full-body photo of the target person, showing the static pose of the person at a specific angle.

[0061] Second, feature extraction is performed on the reference video. In the global pose feature extraction process, a skeleton point detection algorithm is used to identify the key nodes of the human body in the video, such as the head, shoulders, elbows, wrists, etc., to track the movement trajectory of these nodes in the entire video sequence and construct a global pose feature vector. Local pose feature extraction focuses on the hand region, using a hand key point detection algorithm to identify the positions of the palm and finger joints, record their relative motion in the local coordinate system, and generate a local pose feature vector.

[0062] Then, the target video is generated according to the extracted features and the reference image. First, the global pose features and the local pose features are time-aligned and fused to obtain a comprehensive pose feature sequence. Then, using a generative adversarial network, the comprehensive pose feature sequence is mapped to the image space to generate a dynamic video frame sequence of the target person, with the reference image as the conditional input. Finally, the generated video frames are time-smoothed to ensure the coherence of the motion transition.

[0063] Through the above scheme, the present application solves the problems of unstable hand movement and unclear details in traditional portrait video generation. Through hierarchical feature extraction, global pose features ensure the coherence of the overall action, while local pose features improve the detail accuracy of key areas such as hands. The feature fusion mechanism establishes the spatio-temporal correlation of global and local motion, avoiding the incoordination between hand movement and overall body posture. Combined with the static features of the reference image, the generated target video not only maintains the identity features of the person, but also realizes stable and smooth hand movement and clear and realistic details. This method improves the quality of the generated video and enhances the credibility of user experience under the condition of less high-quality training data, providing more reliable technical support for virtual fitting, gesture interaction and other application scenarios.

[0064] In an embodiment of the present application, the local field of view region is a sub-region dynamically changing within the global field of view region, and the local field of view region is used to capture the pose information of the specified body part of the first human body; before feature extraction is performed on the reference video, the local field of view region can be located according to the specified body part in each video frame of the reference video.

[0065] In an embodiment of the present application, the dynamically changing local field of view region can realize real-time adjustment of the spatial position through a target detection algorithm, for example, a coordinate mapping method based on key point detection, taking the body part key point coordinates as the center point coordinates of the local field of view region. The size of the local field of view region can be set to a fixed proportion, for example, a rectangular region extending outward by 50 pixels to 200 pixels with the key point as the center. In the video frame sequence processing process, the motion trajectory of the body part between adjacent frames is established through an optical flow tracking algorithm, and the positioning accuracy of the local field of view region is further optimized.

[0066] Specifically, in the video frame processing stage, first, the spatial coordinates of the specified body part are identified through a human skeleton key point detection model, for example, the two-dimensional coordinates of the hand key point. Then, the boundary range of the local field of view region is calculated according to the coordinate data, so that the region can cover the instantaneous position of the hand motion trajectory. When the hand moves, the center coordinates of the local field of view region are adjusted synchronously at an update frequency of 0.1 second per frame, ensuring that the feature extraction region always covers the target part. Through multi-frame continuous positioning, spatiotemporal correlation features are formed, so that the feature extraction network can capture the acceleration and trajectory curvature change of the hand motion. This dynamic positioning mechanism can improve the spatiotemporal continuity of the hand pose features, effectively solving the feature deviation problem caused by the traditional fixed region, and providing accurate motion parameter input for generating stable and smooth hand motion videos.

[0067] In an optional implementation, the scheme of the present application is implemented as follows: The local field of view region dynamically changes within the global field of view region. The local field of view region is used to capture the pose information of the specified body part of the first human body. Before feature extraction is performed on the reference video, the local field of view region is located according to the specified body part in each video frame of the reference video.

[0068] Specifically, the dynamic positioning of the local field of view region can be realized through the following steps: First, human key point detection is performed on each frame of the reference video to identify the positions of each key point of the first human body. The key points can include the head, neck, shoulder, elbow, wrist, etc.

[0069] Then, according to the pre-set specified body part, for example, the hand, the corresponding key point position is determined in each frame. For the hand, the wrist key point can be selected as the positioning reference point.

[0070] Then, a fixed-size rectangular region is determined as the local field of view region, with the wrist key point as the center. The size of the rectangular region can be set according to actual needs, for example, a region extending a certain number of pixels around the wrist key point as the center can be set.

[0071] Further, in order to ensure that the local field of view region can completely capture the hand movement, the rectangular region can be adjusted according to the motion trajectory of the wrist key point. For example, when it is detected that the moving distance of the wrist key point between consecutive frames exceeds a threshold, the center position of the rectangular region can be adjusted accordingly, so that it always follows the hand movement.

[0072] Thus, by dynamically positioning the local field of view region in each frame, it can be ensured that the posture information of the specified body part is always accurately captured, providing an accurate target region for subsequent feature extraction.

[0073] Through the above technical solutions, the present application realizes accurate capture of the posture of a specific body part during human motion. The local field of view region, as a sub-region within the global field of view region, has dynamic change characteristics that can adjust to the position change of the specified body part in the video frame, avoiding the feature loss problem caused by motion offset when extracting features from a traditional fixed region. By positioning the specified body part in each video frame, the spatial position of the local field of view region can be determined in real time, ensuring that feature extraction always focuses on the target region. This dynamic positioning mechanism not only improves the matching degree of local posture features and the actual motion trajectory of the body part, but also forms spatiotemporal correlation features through continuous positioning of multiple frames, providing key motion information support for subsequent generation of videos with stable hand movements.

[0074] In an embodiment of the present application, the specified body part is the hand of the first human body.

[0075] Generally, the hand contains 21 joint nodes, and the three-dimensional coordinate data of each joint node can be captured in real time by a key point detection model. The positioning range of the local field of view region covers the palm region to the region 10 cm above the wrist joint, ensuring that both the fingertip micro-motion and the overall posture of the palm are included in the feature extraction range. The calculation of the hand posture parameters is based on the joint angle change rate and the motion trajectory curvature, for example, when the finger bending angle exceeds 15 degrees or the motion trajectory curvature exceeds 0.8, high-precision feature sampling is triggered. The hand feature extraction module and the global posture feature extraction module share a timestamp alignment mechanism, so that the hand joint motion features can form a spatiotemporal correlation with the torso posture changes.

[0076] Specifically, in the video generation process, when the hand is specified as the designated body part, the local field of view area is forcibly locked to the area containing the palm and fingers. Through the hand key point detection model, the three-dimensional coordinate data of the 21 joints of the hand is extracted from each frame of the reference video, and the angle change between adjacent joints is calculated. For example, when the pinch action of the thumb and index finger is detected, the joint angle change data is marked as a high priority feature, and is sampled at a density twice that of the regular frame rate in the second video frame sequence. The hand posture parameters are obtained by weighted calculation of the motion trajectory length and the joint displacement amount, for example, the weight coefficient of the fingertip displacement amount is set to 0.6, and the weight coefficient of the palm is set to 0.4. The hand local posture features generated in this way are aligned frame by frame with the global posture features in the feature fusion stage through timestamp matching, ensuring that the hand movement and the overall body movement remain synchronized in the time dimension. This technical solution can effectively control the finger bending angle error in the generated video and reduce the jitter amplitude of the fingertip motion trajectory.

[0077] In an optional embodiment, the scheme of the present application is implemented as follows: the designated body part is the hand of the first human body. In the video generation process, first, the reference video is preprocessed, and the hand region is located through the human body key point detection algorithm. Then, for each frame of video, local features of the hand region are extracted, including finger joint position, palm orientation, finger bending degree, etc. These local features are combined with global posture features to form a complete human body posture representation. When generating the target video, the hand features are given priority to ensure the coherence and detail restoration of the hand movement. For example, a special hand posture generation model can be used, which is trained on a hand action dataset and can accurately capture and reproduce complex hand movements.

[0078] Through the above technical solution, the present application solves the technical defect that the target part is not clear in the local posture feature extraction process. By explicitly specifying the hand as the capture object of the local field of view area, the feature extraction module is forced to focus on the hand joint movement features, avoiding feature omission due to body part selection bias. This method makes the local posture features and global posture features complementary, and effectively improves the coherence of the hand motion trajectory and the texture restoration degree of the finger joints by accurately injecting hand detail features in the video generation stage. Thus, the hand movement instability and detail blur problems mentioned in the background art are overcome, and the quality and realism of the generated video are improved.

[0079] In an embodiment of the present application, before feature extraction is performed on the reference video, semantic recognition can be performed on the reference video to obtain the content theme of the reference video; from the multiple body parts of the first human body, a designated body part that has semantic relevance with the content theme is selected.

[0080] Exemplarily, semantic recognition can employ a natural language processing model to perform topic classification on the audio, subtitles or visual content of the video, such as extracting visual features of key frames of the video by a convolutional neural network, in combination with a long short-term memory network to analyze time series information. Body part selection can be based on a pre-trained semantic correlation model to match the content theme with the action semantics of human body parts, such as preferentially selecting the shoulder and waist in a swimming theme, and selecting the spine and limbs in a yoga theme. The judgment criterion for semantic correlation can be set as a probability threshold, when the frequency of occurrence of a certain body part in the theme-related action library exceeds 60%, it is determined to be strongly correlated. The dynamic selection process forms a time sequence synchronization mechanism with global posture feature extraction, which updates the key capture parts in real time in the video frame sequence.

[0081] Specifically, in the video processing stage, the content theme can be first parsed by the semantic recognition module, for example, when the video content is identified as basketball, the motion analysis submodule is activated to extract action patterns such as shooting and dribbling. Then, according to the typical action features of basketball, the elbow joint flexion angle and trunk rotation amplitude are selected from the human body joint data as key parameters, so as to determine the arm and trunk as the key capture parts. In the feature extraction process, the local field of view area is positioned in real time following the dynamically selected parts, for example, in the dribbling action, the wrist swing trajectory is captured, and in the jump shooting, the coordinated motion of the shoulder and waist is tracked. This dynamic adjustment mechanism based on semantics makes the local feature extraction always focus on the body areas that are strongly related to the motion logic, avoiding the wrong capture of foot actions and the neglect of hand posture details in dance videos, thereby improving the semantic rationality and motion coherence of the human action in the generated video.

[0082] In an optional embodiment, the scheme of the present application is implemented as follows: First, the reference video is subjected to semantic recognition. Semantic recognition can be achieved by a deep learning model, for example, using a pre-trained convolutional neural network to extract features of video frames, and then combining a long short-term memory network to capture time series information. In this way, the content theme of the reference video can be obtained, such as dance, sports or daily activities, etc.

[0083] Further, based on the identified content theme, a specified body part having semantic correlation with the content theme is selected from a plurality of body parts of the first human. For example, if the content theme is dance, the hand and leg can be preferentially selected as the specified body part; if the content theme is basketball, the arm and trunk can be selected as the specified body part. This selection can be achieved by a pre-defined theme-body part mapping table, or by a machine learning model to dynamically determine the correlation.

[0084] By the technical solution, the adaptability and pertinence of local feature extraction are achieved. Thus, the human posture change in the generated video is more in line with the semantic requirements of the video content, improving the quality and realism of the generated video. Meanwhile, by focusing on the body parts highly related to the video theme, unnecessary feature extraction calculation is reduced, improving the efficiency of the system. In addition, the dynamic selection mechanism enables the system to adapt to different types of video content, enhancing the versatility and flexibility of the method.

[0085] Figure 3 A flowchart of feature extraction on a reference video in an embodiment of the present application is shown. As shown in Figure 3 The method of feature extraction on a reference video to obtain global posture features and local posture features can include the following steps S310-S340.

[0086] S310: Select a plurality of first video frames from the reference video to form a first video frame sequence, the first video frame being a video frame representative of the global posture of the first human body.

[0087] S320: Perform feature extraction on the first video frame sequence to obtain global posture features.

[0088] S330: Select a plurality of second video frames from the reference video to form a second video frame sequence, the second video frame being a video frame representative of the local posture of the specified body part.

[0089] S340: Perform feature extraction on the second video frame sequence to obtain local posture features.

[0090] Exemplarily, the selection of the first video frame can be achieved by a posture parameter threshold, for example, when the global posture parameter of the video frame exceeds 0.8, it is determined as a key frame, which can be obtained by calculating the bone joint displacement or motion trajectory curvature. The selection of the second video frame can be combined with the dynamic change characteristics of the local field of view, for example, when the hand joint angle change rate exceeds 30 degrees per second, the local key frame capture is triggered. The extraction of global posture features can adopt a time interval sampling strategy, for example, 5 frames per second are selected to cover the main motion stage, while the local posture feature extraction can adopt a high-resolution feature map for detail capture for the hand region. On the basis of positioning the local field of view, the body parts matching the content theme are automatically selected by a pre-set semantic correlation model, for example, the hand is preferentially selected as the specified part in a dance video, so that the local feature extraction is spatiotemporally associated with the global motion trend.

[0091] For example, in the process of video processing, first, the global pose parameters of each video frame are calculated by a pose recognition algorithm. The parameters can be based on the spatial distribution of human body joint points, the dispersion degree, or the motion speed mutation point detection. When the parameters exceed the preset threshold, the frame is selected into the first video frame sequence. For example, when the trunk inclination angle of three consecutive frames changes by more than 15 degrees, it is marked as a global key frame. Then, a three-dimensional convolutional network is used to encode the features of the filtered sequence, extracting global pose features covering the whole body motion trajectory. For local features, based on the located hand region coordinates, the finger bending degree and joint displacement are quantitatively analyzed. When the starting frame of the gripping action or the gesture shape mutation frame is detected, it is included in the second video frame sequence. Through a dual-path feature extraction architecture, the global and local sequences are processed respectively. The global path uses downsampling to reduce the computational load, and the local path retains the original resolution to capture the fingertip texture. In the feature fusion stage, the global feature vector and the local feature map at the same time are spliced through the timestamp alignment mechanism to form a spatiotemporal consistent human pose representation. This phased processing method concentrates computing resources on key motion nodes, reducing more than 60% of the computational load compared to full-frame processing, while the hand joint coordinate error is reduced to within 3 pixels, effectively improving the smoothness of gesture transitions in the generated video.

[0092] In an optional embodiment, the scheme of the present application is implemented as follows: A plurality of first video frames are selected from the reference video to form a first video frame sequence. The first video frame is a video frame that is representative of the global pose of the first human body. For example, a pose recognition algorithm can be used to analyze each frame in the reference video and calculate the position and connection relationship of the key points in each frame. According to the change amplitude and speed of the key points, frames with significant pose changes are selected as first video frames. Specifically, a threshold value can be set, and when the position of the key points between adjacent frames changes by more than the threshold value, the frame is selected as a first video frame.

[0093] Feature extraction is performed on the first video frame sequence to obtain global pose features. Further, a convolutional neural network can be used to process each frame in the first video frame sequence to extract high-dimensional feature vectors of the human pose. These feature vectors contain information about the overall pose of the human body, such as the position and angle of the torso and limbs.

[0094] A plurality of second video frames are selected from the reference video frames to form a second video frame sequence. The second video frame is a video frame that is representative of the local pose of the specified body part. For example, if the specified body part is the hand, a hand detection algorithm can be used to locate the hand region in each frame and calculate the motion amplitude and complexity of the hand region. When the hand motion amplitude or complexity exceeds the preset threshold, the frame is selected as a second video frame.

[0095] The second video frame sequence is subjected to feature extraction to obtain local pose features. Specifically, a feature extraction network dedicated to hands, such as a hand key point detection network, can be used to process each frame in the second video frame sequence. The extracted features include detailed information such as finger joint positions and palm orientations.

[0096] By the above technical solutions, the present application realizes efficient feature extraction of global poses and local poses in the reference video. Thus, the waste of computing resources caused by indiscriminate processing of all video frames is avoided. Further, by respectively selecting representative frames of global poses and local poses, it is ensured that the extracted features can accurately capture the overall motion trend of the human body and the local detail changes. Specifically, the extraction of global pose features focuses on key frames of large-scale human body movements, ensuring the coherence of the overall action of the character in the generated video. The extraction of local pose features specifically focuses on the motion changes of detailed parts such as hands, improving the stability and detail authenticity of hand movements in the generated video. Thus, while ensuring the quality of feature extraction, the computational complexity is significantly reduced, providing a reliable feature basis for subsequent generation of high-quality target videos.

[0097] In an embodiment of the present application, the method of selecting a plurality of first video frames from the reference video to form the first video frame sequence can further include: in each video frame of the reference video, performing pose recognition on the global field of view region to obtain a global pose parameter of the video frame, the global pose parameter being used to represent the importance of the video frame for describing the global pose of the first human body; and selecting a plurality of first video frames from the reference video to form the first video frame sequence according to the global pose parameter, the global pose parameter of the first video frame being greater than a preset first parameter threshold.

[0098] Exemplarily, the generation of the global pose parameter can be realized by a human key point detection algorithm, for example, by calculating the key point displacement amplitude or pose change angle between adjacent frames, and marking the frames with motion amplitude exceeding the set threshold as high importance frames. The first parameter threshold can be set as a dynamically adjusted value, for example, self-adaptively adjusted according to the total length of the reference video or the motion complexity, and the specific numerical range can be selected between 0.7 and 0.9. In the basketball dunk scene, the starting frames of the three stages of take-off, in-air stretch, and dunk action can be identified as high global pose parameter frames. The parameter calculation process is complementary to the dynamic positioning of the local field of view region in the above embodiment. When the global field of view region captures the overall motion trend, the local field of view region simultaneously focuses on the details of hand movements, and the two are time-stamped to realize the spatiotemporal consistency modeling of the motion trajectory.

[0099] For example, by using a pose recognition algorithm to detect skeletal key points in the global field of view, multi-dimensional pose data including torso inclination angle and limb extension degree are extracted. Further, normalization processing is used to map each dimension of data to the 0-1 interval, and a global pose parameter is generated by weighted summation. When the parameter exceeds a first parameter threshold, it is determined that the frame contains a significant pose change node. For example, in a taijiquan scene, the transition frame of the wild horse comb action has a torso rotation angle of 35 degrees, and its global pose parameter reaches 0.85, which exceeds the preset threshold of 0.8 and is selected into the first video frame sequence. This screening mechanism, combined with the feature alignment based on the timestamp in claim 5, ensures that the global pose feature and the local hand action feature are accurately matched in time, so that when the target video is generated, the overall movement trajectory of the second human body can be restored, and the continuity of the hand details can be maintained. The video frame sequence formed thereby effectively avoids the omission of action key frames caused by traditional equal-interval sampling, and makes the turning, jumping and other complex actions of the second human body in the generated target video appear more natural.

[0100] In an optional embodiment, the scheme of the present application is implemented as follows: In each video frame of the reference video, pose recognition is performed on the global field of view. Specifically, a human pose estimation algorithm is used to process each frame to extract human key point coordinate information. These key points include head, shoulder, elbow, wrist, hip, knee and ankle, etc.

[0101] According to the extracted key point information, the global pose parameter of each frame is calculated. The global pose parameter can be obtained by multiple indicators, such as the relative position change between key points, motion speed, acceleration, etc. These parameters collectively reflect the importance of the frame in describing the global pose of the first human body.

[0102] A preset first parameter threshold is set to filter representative video frames. This threshold can be adjusted according to the specific application scenario and requirements. For example, the threshold can be set to the average value of the global pose parameter plus a standard deviation.

[0103] The global pose parameter of each frame is compared with the preset first parameter threshold. If the parameter value of a certain frame is greater than the threshold, it is selected into the first video frame sequence. In this way, it can be ensured that the selected frames can fully reflect the key pose changes of the first human body during the movement.

[0104] Finally, a plurality of first video frames that meet the conditions are selected from the reference video to form a first video frame sequence. This sequence contains key frames that are representative of the global pose of the first human body, providing high-quality input data for subsequent feature extraction and video generation.

[0105] By the technical solution, the application realizes automatic screening of key posture frames in the reference video. The global posture parameter is introduced as a quantitative index to avoid subjectivity and inconsistency of manual selection. By setting a threshold for screening, it is ensured that the selected video frames can accurately reflect the key posture changes of the first human body in the motion state. This data-driven frame selection method improves the coherence and naturalness of the posture changes of the second human body in the subsequently generated target video, effectively solving the problem of missing key posture information in the video frame selection process.

[0106] In an embodiment of the application, the method of selecting a plurality of second video frames from the reference video frames to form a second video frame sequence can further include: performing posture recognition on the local field of view in each video frame of the reference video to obtain a local posture parameter of the video frame, the local posture parameter being used to represent the importance of the video frame in describing the local posture of a specified body part; and selecting a plurality of second video frames from the reference video to form a second video frame sequence according to the local posture parameter, the local posture parameter of the second video frame being greater than a preset second parameter threshold.

[0107] For example, the generation of the local posture parameter can be quantitatively calculated by the posture recognition model on the joint angle change rate in the local field of view, such as generating a parameter value based on the displacement amplitude and speed of the hand joint in consecutive frames. The setting of the parameter threshold can be dynamically adjusted according to the total number of video frames, for example, when the total number of video frames exceeds 500 frames, the second parameter threshold can be set to 0.85 to improve the screening accuracy. When selecting the second video frame, a sliding window mechanism can be used to compare the local posture parameters of adjacent frames, and frames with parameter mutations are retained as key posture change nodes. By aligning the timestamps of the local posture parameters and the global posture parameters, the time sequence consistency of local features and overall actions can be ensured, avoiding the misplacement of limb motion trajectories during feature fusion.

[0108] For example, in each frame of the reference video, the local field of view is recognized by a pre-trained skeletal key point detection model to extract joint coordinate data of a specified body part. Based on the joint coordinates, the motion vector module length between adjacent frames is calculated, and the module length is normalized with a preset reference value to obtain a local posture parameter in the range of 0 to 1. When the local posture parameter exceeds the second parameter threshold, it is determined that the frame contains key posture change information. Further, the second parameter threshold can be dynamically adjusted according to the video frame rate, for example, in a 30 frames / second video, the threshold can be set to 0.75 to capture subtle motion changes. Through this screening mechanism, redundant frames with static or repetitive hand movements can be effectively eliminated, so that only high-information-content postures such as hand opening, fist clenching or rotation are retained in the second video frame sequence, thereby improving the modeling accuracy of the hand motion trajectory during subsequent feature extraction.

[0109] In an optional embodiment, the scheme of the present application is implemented as follows: In each video frame of the reference video, the local field of view region is recognized. A deep learning model can be used to detect and locate the human body key points in the local field of view region to obtain the coordinate information of the key points. Based on the key point coordinate information, the relative positions and angle relationships between the key points are calculated to generate a feature vector describing the local pose. The feature vector is input into a pre-trained pose evaluation model to output a numerical score representing the importance of the specified body part local pose in the video frame, i.e., the local pose parameter.

[0110] Further, a predetermined second parameter threshold, for example 0.7, can be set. All video frames in the reference video are traversed, and the video frames with a local pose parameter greater than 0.7 are selected to form a second video frame sequence. Thus, each frame in the second video frame sequence has a high local pose description value and can effectively capture the key pose changes of the specified body part.

[0111] Through the above technical scheme, the present application realizes the quantitative evaluation of the importance of the video frame. Through the calculation and threshold screening of the local pose parameter, the video frames containing the key pose information of the specified body part can be accurately identified and retained. This screening mechanism based on objective numerical indicators avoids the subjective bias of manual judgment, and improves the accuracy and representativeness of the selected video frames in describing the local pose. Thus, in the subsequent feature extraction and video generation process, the movement details of the specified body part can be restored more accurately, and the action continuity and detail clarity of the local region in the generated video can be significantly improved.

[0112] In an embodiment of the present application, generating the target video according to the global pose feature, the local pose feature, and the reference image can further include: performing feature fusion on the global pose feature and the local pose feature to obtain a human pose feature; and performing feature restoration on the human pose feature to obtain the target video, taking the reference image as a prompt word.

[0113] For example, the feature fusion stage realizes spatio-temporal alignment through a timestamp matching mechanism, for example, difference calculation is performed on the timestamps of the global pose feature and the local pose feature, and when the time difference is less than a preset threshold, it is determined that the feature pair is synchronous. The feature splicing adopts a channel dimension stacking manner, for example, the 256-dimensional vector of the global pose feature and the 128-dimensional vector of the local pose feature are spliced in the channel dimension to form a 384-dimensional fusion feature. The feature restoration stage adopts a hierarchical guidance mechanism, for example, the shallow edge feature, the middle texture feature, and the deep semantic feature extracted from the reference image through a convolutional neural network are respectively calculated with the fusion feature through cross-modal attention, and the attention weight can be set as an adjustable parameter in the range of 0.3-0.7.

[0114] For example, in the feature fusion process, the skeleton motion trajectory carried by the global pose feature and the joint displacement recorded by the local pose feature are aligned at the frame level through timestamp matching, such as performing feature synchronization verification every 5 frames. When the hand joint displacement is detected to exceed the preset threshold, the local feature resampling mechanism is automatically triggered, and the missing frame data is supplemented through the interpolation algorithm. In the feature restoration stage, the face contour feature of the reference image guides the head motion trajectory generation through the spatial attention mechanism, and the hand texture feature controls the knuckle bending angle through the channel attention mechanism. For example, in the third layer of the decoder, the feature map resolution of the hand region is increased to 512x512 pixels, at which time the nail texture details provided by the reference image are injected into the generation process through residual connection, so that the fingertip shape maintains a position deviation of less than 2 pixels between consecutive frames. The scheme ensures the continuity of the hand motion trajectory through the spatio-temporal alignment mechanism, and realizes detail enhancement using the multi-level features of the reference image, so that the finger bending angle error is controlled within ±5 degrees in the motion sequence, and the joint position deviation is stable below the 3 pixel threshold.

[0115] In an optional implementation, the scheme of the present application is implemented as follows: In the video generation process, the time dimension of the global pose feature and the local pose feature is synchronized by the feature alignment module, wherein the global pose feature is composed of a human body skeleton key point sequence, and the local pose feature is composed of a hand joint sequence. Specifically, the global skeleton sequence and the local hand joint sequence are mapped to the same time axis through a timestamp matching mechanism, and a bidirectional recurrent neural network is used to interpolate and compensate the feature sequence, so that the global feature and the local feature completely correspond in time steps. In the feature fusion stage, a cross-modal attention mechanism is used to splice the synchronized global skeleton feature and the local hand feature into a three-dimensional tensor, and a spatial pyramid pooling layer is used for multi-scale feature fusion to form a human pose feature tensor containing overall motion trend and detailed action. In the feature restoration stage, the reference image is disassembled into multi-layer semantic features, wherein the shallow features contain hand texture details, and the deep features contain limb shape information. Through cross-attention mechanism, the human pose feature tensor is dynamically associated with each layer of image features, and image details are injected layer by layer in the video decoding process, such as using shallow image features to correct the skin texture of hand joints, and using deep image features to maintain the coherence of limb motion.

[0116] By the technical solution, the application effectively solves the problems of hand action fracture and detail ambiguity in the video generation process. The time dimension alignment mechanism ensures the time sequence consistency of global motion and local action, avoiding the appearance of faults in the time axis of the hand movement trajectory; the cross-modal feature fusion mechanism realizes the collaborative expression of skeleton movement trend and hand detail action, ensuring that the hand action conforms to the overall movement law and retains the fine action characteristics; the dynamic guidance mechanism based on multi-layer image features continuously injects static image detail features in the video decoding process, making the hand texture, joint bending angle and other details in the generated video highly consistent with the reference image, while maintaining the stability of the shape in the movement process.

[0117] In an embodiment of the application, the feature fusion of the global pose feature and the local pose feature to obtain the human pose feature can further include: acquiring time stamps associated with the global pose feature and the local pose feature respectively, the time stamps being used to mark the positions of the video frames in the reference video; performing feature alignment on the global pose feature and the local pose feature according to the time stamps to obtain feature pairs corresponding to the same time stamp; and splicing the feature pairs to form the human pose feature.

[0118] Illustratively, the time stamp marking mechanism can use absolute time coding or relative time offset coding based on the video frame sequence, such as millisecond-level time stamps based on the starting point of the video. The feature alignment operation can establish a mapping relationship between the time stamp and the feature vector through a hash table, and when the time stamp difference of the global pose feature and the local pose feature is less than a preset threshold, for example, the difference does not exceed the time interval of two frames, it is determined that it is an alignable feature pair. The feature splicing can use channel dimension splicing or spatial dimension splicing, for example, splicing the 256-dimensional vector of the global pose feature and the 128-dimensional vector of the local pose feature into a 384-dimensional fusion feature. In the feature alignment process, if there are time stamps that cannot be matched, an interpolation algorithm can be used to generate intermediate features, for example, using linear interpolation to supplement the missing local pose features.

[0119] For example, in generating human pose features, first, the timestamp corresponding to each feature is extracted from the global pose feature sequence, and the corresponding timestamp is extracted from the local pose feature sequence. By traversing the timestamps of the two feature sequences, features with a timestamp difference less than or equal to one frame interval are paired, for example, in a 30 frame / s video, features with a timestamp difference of not more than 33 milliseconds are considered as feature pairs at the same time point. The paired global pose features and local pose features are spliced in the channel dimension to form a fusion feature containing the overall motion trend and local details. In this process, the timestamp difference threshold can be dynamically adjusted according to the video frame rate, for example, set to 16 milliseconds in a 60 frame / s video. For unpaired local pose features, the global pose features of the adjacent two frames are used to generate supplementary features by weighted average, ensuring the continuity on the time axis. Thus, the fused human pose features not only retain the coherence of global motion, but also enhance the detail accuracy of hand movements, making the generated target video natural and smooth in human pose changes and clear and accurate in key part movements.

[0120] In an optional embodiment, the scheme of the present application is implemented as follows: In the feature fusion stage, the acquisition time information accurate to milliseconds is parsed from the video frame metadata as a timestamp. The global pose feature sequence and the local pose feature sequence respectively carry the timestamps of the respective source video frames, and the time sequence mapping relationship across the sequences is established through timestamp comparison. Specifically, when there is a sample with a timestamp of T_12:34:56.789 in the global feature sequence, the system automatically retrieves the matching sample in the local feature sequence with a timestamp difference within ±50 milliseconds, forming a time-aligned feature pair. The matched feature pair is fused through a tensor splicing operation, in which the global feature vector and the local feature vector are concatenated along the channel dimension to form a fusion feature vector containing time-space alignment information.

[0121] Through the above technical scheme, the time sequence misalignment phenomenon caused by different sampling frequencies is effectively eliminated, ensuring that the global motion trajectory and the local joint movement are accurately synchronized in the time dimension. The feature alignment mechanism enables the human pose features to contain both macro motion trends and micro movement details, which can accurately guide the displacement trajectory of the hand joints and the finger joint bending angle in the video generation process, and finally output a target video with stable hand motion trajectory and clear finger texture.

[0122] In an embodiment of the present application, taking the reference image as the prompt word, the feature restoration of the human pose feature to obtain the target video can further include: performing feature extraction on the reference image layer by layer to obtain multiple image features corresponding to different feature depths; and taking the multiple image features as prompt words respectively, performing feature restoration on the human pose feature layer by layer to obtain the target video.

[0123] Exemplarily, different depths of neural network layers can be adopted when performing layer-by-layer feature extraction on the reference image, such as shallow convolutional layers extracting edge contour information and deep convolutional layers extracting texture detail information. During the layer-by-layer feature restoration process, the human pose feature and the image feature interact through the cross-modal attention mechanism, wherein the low-level feature restoration stage adopts a global attention module to establish the correspondence between the hand movement trajectory and the joint position, and the high-level feature restoration stage adopts a local attention module to focus on the detail matching of the finger bending angle and the skin wrinkle. When the human pose feature is spliced after the timestamp alignment of the feature pair, the time dimension information thereof can be dynamically weighted with the image features at different levels, such as giving higher weight to the shallow features in the movement trajectory generation stage and increasing the fusion proportion of the deep features in the detail rendering stage.

[0124] For example, in the feature restoration process, first, the reference image is processed in layers through a pre-trained feature extraction network, such as using the first three residual blocks of ResNet-50 to output image features of different granularities. Subsequently, in the initial stage of generating the target video, the shallow image features are input into the generation model as a condition to guide the model to decode the hand joint displacement parameters in the human pose feature, ensuring smooth transition of the hand movement trajectory between adjacent video frames. When generating to the middle layer, the middle image features correct the finger bending angle parameters in the human pose feature through the channel attention mechanism, eliminating the movement jitter caused by joint misplacement. In the final rendering stage, the deep image features are pixel-level fused with the skin texture parameters in the human pose feature, and the details of the fingernail surface reflection and fingerprint wrinkles are enhanced through adversarial training. Through this hierarchical progressive feature guiding mechanism, the global motion information and local detail parameters in the human pose feature can be accurately matched with the image features at the corresponding level in different generation stages, thereby simultaneously improving the coherence of hand movement and the authenticity of details while reducing the amount of training data.

[0125] In an optional embodiment, the scheme of the present application is implemented as follows: the reference image is input into the pre-trained deep convolutional neural network for feature extraction, and different levels of image features are obtained through layer-by-layer forward propagation. Specifically, shallow, medium and deep features are extracted at the third, fourth and fifth layers of the convolutional neural network, respectively, wherein the shallow features correspond to the basic geometry of the hand joint position, the medium features correspond to the morphological features of the palm contour, and the deep features correspond to the microscopic details of the fingerprint texture. Subsequently, in the decoding stage of the video generation network, the shallow features are used as the prompt word of the initial generation stage to guide the coherence generation of the hand motion trajectory, the medium features are used as the prompt word of the intermediate generation stage to optimize the natural transition of the finger bending angle, and the deep features are used as the prompt word of the final generation stage to strengthen the detailed performance of the nail gloss and skin wrinkles. Through this hierarchical progressive generation mechanism, each network level receives semantic guidance of corresponding granularity during the generation process.

[0126] Through the above technical scheme, the present application effectively solves the problem of confusion of hand detail features in the traditional single-stage generation. By decomposing the semantic information of the reference image into multiple granularity feature levels, the corresponding detail guidance signals are injected at different stages of video synthesis, so that the spatiotemporal continuity of the hand motion trajectory is guaranteed while the generation quality of the microscopic texture is significantly improved. Especially in the scene of fast finger movement, the shallow features ensure the smooth transition of the joint position change, and the deep features avoid the texture blur phenomenon caused by high-speed movement, thereby realizing high-fidelity video generation of hand details under the condition of low training data volume.

[0127] The human video generation process and the corresponding model training process involved in the present application will be described in detail below in conjunction with specific application scenarios.

[0128] Figure 4 A flowchart of the human video generation process of an embodiment of the present application in an application scenario is shown.

[0129] As Figure 4As shown, the embodiment of the present application proposes a two-stage training method for decoupling training of a hand feature extractor in the field of human video generation, which improves the hand detail quality of the generated video and the generalization of different scale poses. The video generation process is as follows: first, the target video is input into a pre-trained human pose estimation model and a hand pose estimation model to obtain the target full-body action sequence and hand action picture sequence. Then, the human pose feature extraction module and the hand feature extraction module are used to obtain the required features. The hand feature extraction module uses an image restoration training method and introduces position disturbance and other methods to enhance the model's understanding of hand features. Finally, the reference picture, the full-body pose feature, and the optimized hand feature are jointly input to guide the human video generation model to perform video synthesis. This method not only improves the quality of hand details in the generated video, but also enhances the adaptability and generalization ability of the model to different scale poses.

[0130] For example, in order to generate a high-quality dance performance video, first, a pre-trained human pose estimation model and a hand pose estimation model are used to extract the full-body action sequence and hand action picture sequence of the dancer. Then, the full-body action sequence is processed by a specially designed human pose feature extraction module, and the hand action picture sequence is processed by an optimized hand feature extraction method to obtain more detailed hand detail features. Finally, a reference picture of the dancer is input into the human video generation model together with the extracted full-body and hand pose features to guide the model to generate a new video that not only accurately reproduces the dancer's full-body action, but also particularly finely displays every detail of the hand action, significantly improving the overall quality of the generated video and the hand performance. This method solves the problems of blurred hand details and inaccurate poses in traditional techniques, and enhances the generalization ability to different scale poses.

[0131] Figure 5 A flowchart of the hand feature extraction process in an application scenario is shown. The human action video generation method proposed by the embodiment of the present application is different from the conventional training method of directly using the feature extraction module and the video generation model to restore the target video. First, a picture dataset of the person is constructed, and the video generation model is guided by the hand feature extraction module to learn to restore the hand of the person in the picture, and the feature extraction ability of the position disturbance enhancement module is added.

[0132] As Figure 5 shown, the hand feature extraction process can include the following steps S510 to S530.

[0133] In step S510, a human image dataset is constructed.

[0134] Figure 6A flowchart of the embodiment of the present application for constructing a training dataset for a hand feature extraction module in an application scenario is shown. As shown in Figure 6 First, the human image dataset is obtained, such as 800,000 clear hand human pictures, including real data and high-quality character data generated by a text image model, wherein the characters cover all age groups, genders, and body types as much as possible. The image is pre-cropped to meet the input resolution requirements of the video generation model. The hand position in the image is detected by the hand detection model, a label is given, and the hand grid structure of the image is predicted by using the hand pose estimation model. The pose result is obtained by using random translation and scaling operation, and then restored into a hand pose picture. After the above processing, a series of original pictures P, corresponding hand position masks M, and corresponding hand pose pictures G can be obtained. The hand position mask M around all original pictures P is changed to all white, and the picture P ̃ without hand is obtained as training data. The training label is the original picture P, and thus the dataset T is constructed.

[0135] In step S520, the hand feature extraction module is trained.

[0136] The conventional training feature extraction module often directly extracts features from all kinds of posture pictures, and performs end-to-end training by using the guided video generation model to generate a video. The direct connection between the hand feature and the hand in the video generation is not close.

[0137] The embodiment of the present application proposes a two-stage training strategy for decoupling the hand feature extraction module. In data processing, the embodiment of the present application also improves the generalization of the model by adding random disturbance to the hand pose estimation result. Unlike the traditional end-to-end training method, the embodiment of the present application first uses the decoupled hand feature extraction module to independently perform the first stage of training. This stage focuses on accurately extracting hand features from the input hand pose image. By introducing a loss function specifically for the hand pose region, the feature extraction module is optimized to learn more robust and accurate hand representations without interference from other non-hand factors. This separate training method helps to enhance the consistency and accuracy between the hand feature and the hand performance in the finally generated video.

[0138] The hand feature input video generation model generates a picture as When the hand feature extraction module is trained, the formula for generating the picture with the restored hand is:

[0139] wherein, represents the hand pose feature extraction module, is a sequence of hand pose pictures, is a video generation model, for picture style reference network, for original picture, for hand position mask.

[0140] Because the video generation model The input of the model is the original picture processed by VAE (here, the original picture after hand removal to avoid answer leakage), the style reference feature obtained by the style reference network, and the hand feature.

[0141] During training, the loss function uses MSE loss, and only calculates the loss at the hand mask position. The formula is:

[0142] Figure 7 A flowchart of training the hand feature extraction module in an application scenario is shown. As Figure 7 shown, the specific training steps of the hand feature extraction module proposed in the embodiment of the application are as follows: First, the hand pose estimation model is applied to extract hand pose information from the picture of the person. Next, a series of random perturbations are applied to the extracted hand pose, including but not limited to scale change, position translation to the center area, and size scaling, etc. operations, and the processed hand pose information is restored to a hand pose image. Then, the hand feature extraction module is used to extract features from the hand pose image, and the obtained features are input into the video generation model as one of the inputs, which is used to generate a single frame image. At the same time, the original picture of the person is processed, in which the hand region is set to be blank, and the position of the blank region is determined by expanding the mask provided by the hand pose estimation model to a certain range, which is provided to the video generation model as another input.

[0143] In the loss calculation stage, the mean square error (MSE) loss function is adopted. Since the video generation model is based on the latent diffusion model (LDM), the loss calculation is performed in the hidden space after the variational autoencoder (VAE) encoding. In particular, in order to deal with the problem of picture imbalance and emphasize the importance of hand details, a higher weight is given to the loss in the hand mask area, so as to enhance the generation quality of the hand region.

[0144] This method predicts the recovery of the missing hand picture through the hand feature extraction module and the video generation model, and specifically strengthens the training of the hand region, thereby improving the hand feature extraction capability of the hand feature extraction module in the final generated video, mainly the understanding of hand action details, and enhancing the detail authenticity of the hand in the final video generation model.

[0145] In an application scenario of an embodiment of the present application, the training process of the pose feature extraction module can include: first, using a human pose estimation model to extract human pose information from the original character action picture. Then, a series of random disturbances are applied to the extracted human pose, including but not limited to translation, body scale, etc., to ensure that other possible body structures are generated under the condition that the action is unchanged. This approach helps to reduce the sensitivity of the feature extraction module to the absolute position and body scale of a specific pose, and enhances its versatility in different scenarios. Subsequently, the processed pose information is restored to a pose picture, and the optimized pose feature extraction module is used to extract features therefrom. These features will be injected as one of the inputs into the backbone network of the video generation model.

[0146] At the same time, in order to obtain the original character action picture as the target generated by the model, first, according to the diffusion model process, gradually add noise to the picture, and then input it as a condition into the video generation model. At the same time, the CLIP model is also used to extract features from the original character action picture, and after obtaining the overall style features through the appearance reference network, they are input into the backbone network of the video generation model together.

[0147] Finally, in the loss calculation stage, the mean square error (MSE) loss between the original character action picture is used to evaluate the quality of the generated picture. This not only trains the feature extraction capability of the human pose extraction module and the appearance reference network, but also improves the ability of the video generation model to generate high-quality pictures under the guidance of the first two features, without considering the continuity between frames.

[0148] This method significantly enhances the robustness of the pose feature extraction module and the expressiveness of the video generation model through data enhancement means and multi-source feature fusion mechanism, especially in adaptability and authenticity when facing different body structures and action details.

[0149] Figure 8 A flowchart of the full-model joint training process of an embodiment of the present application in an application scenario is shown.

[0150] After the first stage of picture-based pre-training, each pose feature extraction module has the ability to guide the model to generate corresponding pose pictures, and the data disturbance makes the pose feature extraction module focus more on the action itself rather than the absolute position of the pose when guiding the video generation model.

[0151] Through the second stage of full-model joint training, the guiding ability of the two types of pose features can be further enhanced on the basis of the first stage, and the introduction of the timing module adds timing information, so that the inter-frame continuity of the generated video is improved.

[0152] As shown in Figure 8 In this application scenario, the full-model joint training can mainly include the following contents.

[0153] (1) Pose feature fusion.

[0154] For the target human pose picture sequence and the target hand pose picture sequence , the human pose feature extraction module and the hand pose feature extraction module trained in the first stage are used for feature extraction, and after feature fusion, the final feature used to guide the human video generation model is obtained.

[0155]

[0156] wherein, is a feature alignment module, which can obtain the aligned human pose feature by aligning the hand pose feature and the human pose feature. is a feature fusion module, which fuses the final feature used to guide the human video generation model.

[0157] (2) Pose feature introduction.

[0158] In order to better play the guiding role of the pose feature, the present scheme uses a NAF-Net module based on a convolutional neural network CNN. The module is designed as an encoder mode of layer-by-layer feature extraction plus down-sampling. The number of layers matches the number of layers of the U-Net structure backbone used in the human video generation model, and the number of channels of each layer also aligns with the corresponding U-Net layer, thereby ensuring the consistency and compatibility of the feature representation. In the step-by-step denoising process of the human video generation model, the feature after each layer processed by the NAF-Net is added to the same layer feature of the main U-Net after zero convolution, which plays a role of layer-by-layer guidance. This layer-by-layer guidance mechanism not only enhances the effective transmission of the pose feature, but also helps the model better understand and generate motion sequences that conform to the expected pose.

[0159] (3) Reference picture feature introduction.

[0160] To make the generated video consistent with the character reference picture, the features of the character reference picture are introduced using two branches. The first branch is directly used as a conditional input branch that the video generation model can accept. This branch encodes the character reference picture into the latent space through a variational autoencoder (VAE), so that the information of the reference picture can be effectively utilized in the video generation process. This process ensures that the input dimension matches the requirements of the video generation model, so that the style features of the reference picture can directly affect the final generated video content. The second branch focuses on the extraction and injection of image fine-grained features, and adopts the Reference UNet structure. This branch first extracts detailed image features from the reference picture, and then adds them layer by layer to the human video generation model. The combination of the two branches ensures the consistency of the generated video and the character reference picture in terms of overall style, and improves the richness and authenticity of the video content through layer-by-layer feature injection. This method significantly enhances the expressiveness of the video generation model.

[0161] (4) Introduce a timing module.

[0162] In the video generation task, it is crucial to introduce a timing module to ensure good coherence and consistency between generated frames. This scheme designs an efficient timing module to process features in the time dimension after feature extraction based on the NAF-Net module and before adding features to the U-Net backbone network. The module mainly consists of inter-frame attention and self-attention mechanisms, and uses the RWKV module to reduce the exponential overhead of traditional attention mechanisms, increasing the calculation speed.

[0163] As can be seen from the above embodiments and application scenarios, the video generation scheme provided by the embodiments of the present application can make full use of hand pose features and human pose features, improve the quality of generated videos, reduce the dependence on a large number of high-quality videos for training, and enhance the adaptability and generalization ability of the model to different proportions of poses by introducing data perturbation.

[0164] It should be noted that although the steps of the method in the present application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired results. In addition or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps, etc.

[0165] The following describes the device embodiments of the present application, which can be used to perform the video generation method in the above embodiments of the present application.

[0166] Figure 9 The structure block diagram of the video generation device provided by the embodiments of the present application is schematically shown. As shown in FIG. 6, the video generation device includes a feature extraction module 601, a feature injection module 602, a video generation module 603, and a data storage module 604. Figure 9As shown, the video generation apparatus 900 comprises: an acquisition module 910 configured to acquire a reference video and a reference image, wherein video content of the reference video comprises a posture change process of a first human body in a motion state, and image content of the reference image comprises a static display effect of a second human body in a specified view angle; an extraction module 920 configured to perform feature extraction on the reference video to obtain global posture features and local posture features, wherein the global posture features are used to represent posture information displayed by the first human body in a global field of view area of the reference video, and the local posture features are used to represent posture information displayed by the first human body in a local field of view area of the reference video; a generation module 930 configured to generate a target video according to the global posture features, the local posture features, and the reference image, wherein video content of the target video comprises a posture change process of the second human body in a motion state.

[0167] In some embodiments of the present application, based on the above technical solutions, the local field of view area is a sub-area that dynamically changes in the global field of view area, and the local field of view area is used to capture posture information of a specified body part of the first human body; the apparatus further comprises: a positioning module configured to position the local field of view area according to the specified body part in each video frame of the reference video.

[0168] In some embodiments of the present application, based on the above technical solutions, the specified body part is a hand of the first human body.

[0169] In some embodiments of the present application, based on the above technical solutions, the apparatus further comprises: an identification module configured to perform semantic identification on the reference video to obtain a content theme of the reference video; a selection module configured to select, from a plurality of body parts of the first human body, a specified body part that has semantic correlation with the content theme.

[0170] In some embodiments of the present application, based on the above technical solutions, the extraction module 920 comprises: a first frame combination module configured to select a plurality of first video frames from the reference video to form a first video frame sequence, wherein the first video frames are representative video frames for describing global posture of the first human body; a first feature extraction module configured to perform feature extraction on the first video frame sequence to obtain global posture features; a second frame combination module configured to select a plurality of second video frames from the reference video frames to form a second video frame sequence, the second video frames being video frames representative of local poses of the specified body part; a second feature extraction module configured to perform feature extraction on the second video frame sequence to obtain local pose features.

[0171] In some embodiments of the present application, based on the above technical solutions, the first frame combination module is further configured to: perform pose recognition on the global field of view in each video frame of the reference video to obtain global pose parameters of the video frame, the global pose parameters being used to represent the importance of the video frame in describing the global pose of the first human body; and select a plurality of first video frames from the reference video according to the global pose parameters to form a first video frame sequence, the global pose parameters of the first video frames being greater than a preset first parameter threshold.

[0172] In some embodiments of the present application, based on the above technical solutions, the second frame combination module is further configured to: perform pose recognition on the local field of view in each video frame of the reference video to obtain local pose parameters of the video frame, the local pose parameters being used to represent the importance of the video frame in describing the local pose of the specified body part; and select a plurality of second video frames from the reference video according to the local pose parameters to form a second video frame sequence, the local pose parameters of the second video frames being greater than a preset second parameter threshold.

[0173] In some embodiments of the present application, based on the above technical solutions, the generation module 930 includes: a feature fusion module configured to perform feature fusion on the global pose features and the local pose features to obtain human body pose features; a feature restoration module configured to perform feature restoration on the human body pose features with the reference image as a prompt word to obtain a target video.

[0174] In some embodiments of the present application, based on the above technical solutions, the feature fusion module is further configured to: respectively acquire timestamps associated with the global pose features and the local pose features, the timestamps being used to mark the positions of the video frames in the reference video; perform feature alignment on the global pose features and the local pose features according to the timestamps to obtain feature pairs corresponding to the same timestamp; and splice the feature pairs to form human body pose features; In some embodiments of the present application, based on the above technical solutions, the feature restoration module is further configured to: perform layer-by-layer feature extraction on the reference image to obtain a plurality of image features corresponding to different feature depths; and perform layer-by-layer feature restoration on the human pose feature by taking the plurality of image features as prompt words respectively to obtain a target video.

[0175] The specific details of the video generation apparatus provided in the embodiments of the present application have been described in detail in the corresponding method embodiments, and will not be repeated here.

[0176] Figure 10 The computer system structure block diagram of the electronic device for implementing the embodiments of the present application is schematically shown.

[0177] It should be noted that, Figure 10 The computer system 1000 of the electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0178] As Figure 10 shown, the computer system 1000 includes a central processor 1001 (CPU), which can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 (ROM) or loaded from a storage part 1008 to a random access memory 1003 (RAM). In the random access memory 1003, various programs and data required for system operation are also stored. The central processor 1001, the read-only memory 1002, and the random access memory 1003 are connected to each other through a bus 1004. An input / output interface 1005 (I / O interface) is also connected to the bus 1004.

[0179] The following components are connected to the input / output interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a Local Area Network card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output interface 1005 as necessary. A removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1010 as necessary, so that a computer program read out therefrom is installed in the storage portion 1008 as necessary.

[0180] In particular, according to embodiments of the present application, the processes described in the various method flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 1009, and / or installed from the removable media 1011. When the computer program is executed by the central processing unit 1001, various functions defined in the system of the present application are performed.

[0181] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In this application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium that can transmit, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium can be transmitted or propagated using any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0182] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code that contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by special-purpose hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0183] It should be noted that, although several modules or units for a device for action execution are mentioned in the above detailed description, such a division is not mandatory. Indeed, according to an embodiment of the application, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into several modules or units embodied.

[0184] Those skilled in the art can easily understand, through the above description of the embodiments, that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes several instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the methods according to the embodiments of the present application.

[0185] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the present application, along with all of the equivalents thereof. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0186] It should be understood that the application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should only be limited by the appended claims.

Claims

1. A video generation method, characterized in that, The method includes: Acquire reference videos and reference images. The video content of the reference video includes the posture change process of a first human body in motion, and the image content of the reference image includes the static display effect of a second human body from a specified viewpoint. Feature extraction is performed on the reference video to obtain global pose features and local pose features. The global pose features are used to represent the pose information of the first human body displayed in the global field of view of the reference video, and the local pose features are used to represent the pose information of the first human body displayed in the local field of view of the reference video. A target video is generated based on the global pose features, the local pose features, and the reference image. The video content of the target video includes the pose change process of the second human body in motion.

2. The video generation method according to claim 1, characterized in that, The local field of view is a sub-region that dynamically changes within the global field of view, and the local field of view is used to capture the posture information of a specified body part of the first human body. Before performing feature extraction on the reference video, the method further includes: The local field of view region is located in each video frame of the reference video based on the specified body part.

3. The video generation method according to claim 2, characterized in that, The designated body part is the hand of the first human body.

4. The video generation method according to claim 2, characterized in that, Before performing feature extraction on the reference video, the method further includes: Semantic recognition is performed on the reference video to obtain the content theme of the reference video; From multiple body parts of the first human body, select a specified body part that is semantically relevant to the content topic.

5. The video generation method according to claim 2, characterized in that, The step of extracting features from the reference video to obtain global pose features and local pose features includes: A first video frame sequence is formed by selecting multiple first video frames from the reference video, wherein the first video frame is a video frame that is representative of the global posture of the first human body. Feature extraction is performed on the first video frame sequence to obtain global pose features; A second video frame sequence is formed by selecting multiple second video frames from the reference video frame, wherein the second video frame is a video frame that is representative of the local posture of the specified body part. Feature extraction is performed on the second video frame sequence to obtain local pose features.

6. The video generation method according to claim 5, characterized in that, The step of selecting multiple first video frames from the reference video to form a first video frame sequence includes: In each video frame of the reference video, pose recognition is performed on the global field of view region to obtain the global pose parameters of the video frame. The global pose parameters are used to represent the importance of the video frame in describing the global pose of the first human body. Based on the global pose parameters, multiple first video frames are selected from the reference video to form a first video frame sequence, wherein the global pose parameters of the first video frames are greater than a preset first parameter threshold.

7. The video generation method according to claim 5, characterized in that, The step of selecting multiple second video frames from the reference video frames to form a second video frame sequence includes: In each video frame of the reference video, pose recognition is performed on the local visual field region to obtain the local pose parameters of the video frame. The local pose parameters are used to represent the importance of the video frame in describing the local pose of the specified body part. Based on the local pose parameters, multiple second video frames are selected from the reference video to form a second video frame sequence, wherein the local pose parameters of the second video frames are greater than a preset second parameter threshold.

8. The video generation method according to claim 1, characterized in that, The step of generating the target video based on the global pose features, the local pose features, and the reference image includes: Human posture features are obtained by fusing the global posture features and the local posture features. Using the reference image as a prompt, the human posture features are reconstructed to obtain the target video.

9. The video generation method according to claim 8, characterized in that, The process of fusing the global pose features and the local pose features to obtain human pose features includes: The timestamps associated with the global pose feature and the local pose feature are obtained respectively, and the timestamps are used to mark the position of the video frame in the reference video; Based on the timestamp, feature alignment is performed on the global pose feature and the local pose feature to obtain feature pairs corresponding to the same timestamp; The aforementioned features are then combined to form human posture features.

10. The video generation method according to claim 8, characterized in that, The step of using the reference image as a prompt word to perform feature reconstruction on the human posture features to obtain the target video includes: The reference image is subjected to layer-by-layer feature extraction to obtain multiple image features corresponding to different feature depths; Using multiple image features as prompts, the human posture features are reconstructed layer by layer to obtain the target video.

Images