Virtual camera pose adjusting method and system and computer program product

Abstract

The invention relates to the field of computer vision, and discloses a virtual camera pose optimization method. The method comprises the following steps: determining an implicit expression model corresponding to a preset reference video through a dynamic neural radiation field, and determining a scene rendering video through the implicit expression model; inputting the rendered video and a preset reference video into a preset virtual camera pose optimization model to determine optical flow diagrams, figure joint point thermodynamic diagrams and depth diagrams of video frames in the reference video and the rendered video; determining an optical flow loss function, a joint point loss function and a depth loss function, and determining a guide area; and for any pixel point of the video frame in the rendered video, determining a first return gradient, a second return gradient and a third return gradient according to the optical flow loss function, the joint point loss function, the depth loss function and the guide region, and performing return so as to optimize shooting parameters of the virtual camera. Memory consumption can be reduced, a mechanism similar to attention can be provided, and convergence of areas with rich attention information is helped.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and virtual scene shooting, and in particular to a virtual camera posture adjustment method, system and computer program product. Background Art

[0002] The purpose of the background technology description provided here is to give an overall background of the present application. The statements in this section merely provide background related to the present application and do not necessarily constitute prior art.

[0003] Under the existing technology, for a traditional performing arts program, the real video shooting process generally includes multiple steps such as scene design, performance arrangement, plot design and camera position setting. Each shot is affected by the above factors. Due to the limitations of objective conditions, the video producer may not be able to shoot the best shot. At the same time, due to the uncertainty of such programs, directors often choose to shoot a preview in advance, which requires a lot of manpower and material resources, and is also affected by various factors such as time and weather. Therefore, high cost, low controllability and the need for advance rehearsal are the main problems of traditional performing arts programs. However, if the program can be transferred to a virtual scene for shooting, the impact of various factors can be reduced, helping the director to find the best shooting plan quickly and easily, obtain the best visual effects, improve the production efficiency of the program and reduce a lot of cost consumption. However, although shooting in virtual scenes has many advantages, virtual scenes are currently usually built manually using virtual engines. The process of manually modeling related performance scenes is too complicated and time-consuming. It requires artificial creation of object materials, textures, and lighting, etc. At the same time, it is difficult to ensure the realism of the modeling. Some objects with special materials (such as glass, fog, etc.) are difficult to present well through manual modeling, resulting in the problem of time-consuming and labor-intensive reconstruction of virtual scenes and poor results. In summary, there is still a lack of methods to quickly express a virtual scene.

[0004] In the shooting of performing arts programs in virtual scenes, in addition to the creativity and quality of the performance itself, the camera movement is also very important, because it is directly linked to the audience's perception. The reasonable use of lens language will better enhance the artistic expression of digital performing arts content, improve the quality and creativity of digital performing arts content, and thus better meet the needs of the audience and immersive experience. Therefore, how to determine the camera trajectory in the virtual scene is also crucial. In order to conveniently determine a shooting trajectory, the "visual tribute" method is used in traditional program shooting, that is, performing artists often imitate some classic shots in movies or performing arts programs, which can help the photographers better convey specific emotions or themes, thereby enhancing the depth of the work and resonating with the audience. Inspired by "visual tribute", for performing arts photographers, in virtual scenes, if they can intelligently analyze the shooting trajectory and techniques of the camera based on the existing shooting videos and apply them to the virtual performing arts scenes, they can determine the camera posture in the virtual scene without in-depth knowledge of lens language knowledge and camera operation technology. At the same time, they only need to change the existing reference videos to obtain different flexible and changeable virtual scene video clips. This technology, which facilitates precise manipulation and reproduction of visual elements such as camera movement, will help create more creative and visually impactful performances.

[0005] In order to optimize the camera pose and ensure that the reference video and the rendered video are shot in the same way, it is necessary to consider whether multiple features between the two are consistent. Existing methods often consider features such as character action, composition, camera motion, and aesthetic evaluation. Among them, the composition feature generally considers the composition of the character, using the character's joint features to represent the character's position and size, etc., but the position of the character on the screen is only a 2D composition information. In real shooting, it is often necessary to fully consider the perspective relationship in the picture, that is, 3D composition information. Therefore, there is still a lack of a feature that can represent the 3D composition information of the character to assist in guiding the optimization of the camera pose.

[0006] Therefore, there is an urgent need for a new camera pose optimization method to overcome and solve the above-mentioned defects. Summary of the invention

[0007] In response to the above problems, the present application proposes a virtual camera posture adjustment method, a virtual camera posture adjustment system and a storage medium.

[0008] The first aspect of the present application provides a virtual camera posture adjustment method, comprising:

[0009] Determine an implicit expression model corresponding to a preset reference video through a dynamic neural radiation field, and determine a rendered video under a current camera trajectory pose through the implicit expression model according to any camera trajectory pose in a preset camera trajectory pose set;

[0010] Inputting the rendered video and the preset reference video into a preset virtual camera pose optimization model to respectively determine an optical flow map, a character joint point heat map, and a depth map of each video frame in the preset reference video and the rendered video;

[0011] Respectively determine an optical flow loss function corresponding to the optical flow map, a joint point loss function corresponding to the character joint point heat map, and a depth loss function corresponding to the depth map;

[0012] According to the preset reference video and the optical flow map of the rendered video, the character joint point heat map and the depth map, a guide area is determined by a preset guide area determination model to highlight the key area whose gradient contribution meets the preset conditions;

[0013] For any pixel point of the video frame in the rendered video, a first gradient is determined according to the optical flow loss function, a second gradient is determined according to the joint point loss function, and a third gradient is determined according to the depth loss function; and a first return gradient corresponding to the first gradient, a second return gradient corresponding to the second gradient, and a third return gradient corresponding to the third gradient are determined according to the guide area, and the first return gradient, the second return gradient, and the third return gradient are returned to optimize the shooting parameters of the virtual camera.

[0014] Furthermore, the preset virtual camera pose optimization model includes:

[0015]

[0016] in, is the camera’s extrinsic matrix, is the time, Φ is the focal length of the camera, F Θ () is an implicit expression model, is a video frame of the reference video;

[0017]

[0018]

[0019] Where i is the video frame number of the reference video, I * The video frame for the rendered video.

[0020] Furthermore, the preset guidance area determination model includes:

[0021]

[0022] in, is the second characteristic difference map, H * To render the heat map of the joint points of the video frame, is the heat map of the joint points of the reference video frame; is the first characteristic difference map, O * To render the optical flow map of a video frame, is the optical flow map of the reference video frame; is the third characteristic difference map, D * To render the depth map of the video frame, is the depth map of the reference video frame; f * () indicates normalization processing.

[0023] Further, determining the first gradient according to the optical flow loss function, determining the second gradient according to the joint point loss function, and determining the third gradient according to the depth loss function includes:

[0024] Performing a derivative operation on the optical flow loss function to obtain the first gradient;

[0025] Performing a derivative operation on the joint point loss function to obtain the second gradient;

[0026] A derivative operation is performed on the depth loss function to obtain the third gradient.

[0027] Further, the determining, according to the guide area, a first return gradient corresponding to the first gradient, a second return gradient corresponding to the second gradient, and a third return gradient corresponding to the third gradient, respectively, includes:

[0028] Taking the dot product result of the guide region and the first gradient as the first return gradient;

[0029] Taking the dot product result of the guide area and the second gradient as the second return gradient;

[0030] A dot product result of the guide region and the third gradient is used as the third return gradient.

[0031] Furthermore, the optical flow loss function includes:

[0032]

[0033] in, and Respectively represent the horizontal and vertical optical flow components of the rendered video, and Represent the horizontal optical flow component and vertical optical flow component of the reference video respectively.

[0034] Furthermore, the joint point loss function includes:

[0035]

[0036] Among them, d w is the Wasserstein distance, H * and Represents the heat map of the character joints of the rendered video frame and the reference video frame, H i It represents the value of the i-th joint point in the character joint point heat map, and M is the number of joint points.

[0037] Furthermore, the depth loss function includes:

[0038]

[0039] Where N is the number of pixels in the video frame, and They represent the depth value of the i-th pixel of the rendered video frame and the reference video frame respectively, H is the height of the reference video frame, and W is the width of the reference video frame.

[0040] A second aspect of the present application provides a virtual camera posture adjustment system, comprising:

[0041] A scene video rendering module, used to determine an implicit expression model corresponding to a preset reference video through a dynamic neural radiation field, and determine a rendered video under a current camera trajectory pose through the implicit expression model according to any camera trajectory pose in a preset camera trajectory pose set;

[0042] A feature estimation module, used for inputting the rendered video and the preset reference video into a preset virtual camera pose optimization model to respectively determine the optical flow map, character joint point heat map and depth map of each video frame in the preset reference video and the rendered video;

[0043] A loss function determination module, used to respectively determine an optical flow loss function corresponding to the optical flow map, a joint point loss function corresponding to the character joint point heat map, and a depth loss function corresponding to the depth map;

[0044] A guide area determination module is used to determine the guide area through a preset guide area determination model according to the preset reference video and the optical flow map of the rendered video, the character joint point heat map and the depth map, so as to highlight the key area whose gradient contribution meets the preset conditions;

[0045] A gradient feedback module is used to determine, for any pixel point of a video frame in the rendered video, a first gradient according to the optical flow loss function, a second gradient according to the joint point loss function, and a third gradient according to the depth loss function; and to determine, according to the guide area, a first feedback gradient corresponding to the first gradient, a second feedback gradient corresponding to the second gradient, and a third feedback gradient corresponding to the third gradient, and feedback the first feedback gradient, the second feedback gradient, and the third feedback gradient to optimize the shooting parameters of the virtual camera.

[0046] According to a third aspect of the present application, a computer program product is provided, wherein the computer program product comprises a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps of the method described above are implemented.

[0047] Compared with the prior art, the advantages or beneficial effects of the technical solution of the present application include:

[0048] This application is based on implicit neural expression of performing arts scenes and a method of collaborative optimization using three features: optical flow, joint point heat map, and depth. It is more suitable for optimizing the camera pose in performing arts scenes. The effect that can be achieved is relatively consistent with the reference video, and can significantly improve the quality of generated camera pose sequences and reduce the workload of scene reconstruction.

[0049] Based on the problems existing in the existing camera pose optimization schemes, this application uses some pre-trained models in existing work as the basis for data preprocessing, and discloses a multi-feature collaborative virtual camera pose optimization model based on implicit neural expression of performing arts scenes. The scene is implicitly neurally expressed using a dynamic neural radiation field to obtain rendered videos under different camera trajectories. At the same time, three differentiable downstream networks are designed to extract optical flow, character joint heat maps, and depth features from rendered videos and reference videos, respectively, and calculate the loss between these three features to measure the differences in camera and character motion, character composition, and picture perspective between the reference video and the neural radiation field rendered video, and finally back-propagate to optimize the camera pose. Since the overall model is differentiable, the camera pose can be iteratively optimized end-to-end to reduce the cumulative error, and finally a rendered video with a similar shooting effect to the reference video is obtained. At the same time, in order to solve the problem that the back propagation of gradients of all pixels will lead to excessive computational consumption and the background pixels in some losses provide less information, the present application proposes to calculate a guidance map through the joint difference map generated by the joint point heat map, optical flow map and depth map to highlight the key areas whose gradient contribution meets the preset conditions, and selectively backpropagate the gradients of these areas to reduce memory consumption and provide a mechanism similar to attention to help convergence and focus on information-rich areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in the relevant field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0051] It should also be noted that, for the convenience of description, only the parts related to the present disclosure are shown in the drawings. The drawings constituting a part of the specification of this application are used to provide a further understanding of the present application. The illustrative embodiments and their descriptions in this application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0052] Figure 1 A flowchart of a virtual camera posture adjustment method provided in an embodiment of the present application;

[0053] Figure 2 is a flowchart of another virtual camera posture adjustment method according to an embodiment of the present application;

[0054] Figure 3 A logical schematic diagram of a virtual camera pose optimization generation method provided in an embodiment of the present application;

[0055] Figure 4 A diagram showing the joint point heat map feature extraction results provided in the embodiment of the present application;

[0056] Figure 5 A diagram showing the results of deep feature extraction provided in the embodiment of the present application;

[0057] Figure 6 A diagram showing the optical flow feature extraction results provided in the embodiment of the present application;

[0058] Figure 7 A result display diagram of the guidance area provided in the embodiment of the present application;

[0059] Figure 8 A visualization diagram of the experimental results provided in the embodiments of the present application. DETAILED DESCRIPTION

[0060] The following will describe the implementation methods of the present application in detail with reference to the accompanying drawings and embodiments, so that the implementation process of how the present application applies technical means to solve technical problems and achieve corresponding technical effects can be fully understood and implemented accordingly. The embodiments of the present application and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the protection scope of the present application.

[0061] It should be clear that the embodiments described below are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by technicians in the relevant field without making creative work are within the scope of protection of the present application.

[0062] Below, some technical terms in the embodiments of the present application and / or the prior art are explained to facilitate those skilled in the art to understand the technical solutions of the present application:

[0063] RAFT (Recurrent All-Pairs Field Transforms, RAFT for short) is a type of optical flow feature extraction network. RAFT consists of three main parts: the first part is the feature encoder and the semantic feature encoder. The feature encoder is used to extract pixel information between two consecutive frames, while the semantic feature encoder only extracts pixel information of the first frame of the video; the second part is the correlation calculation module, which calculates the 4D correlation space (Correlation Volume) by the dot product between the feature vectors of the two images, and the last two dimensions in this space introduce feature pyramids for multi-scale sampling to capture motion at different scales; the third part is the iterative update module, which is the core module of RAFT and contains multiple GRU activation units, which are used to process features and information at different times in the feature pyramid and iteratively update the optical flow.

[0064] Monodepth2 is a deep feature extraction network that can infer the corresponding depth information from a monocular RGB (Red, Green, Blue) image. The model is mainly composed of a deep network and a pose estimation network, both of which consist of an encoder and a decoder. The deep network is used to estimate the depth information of the target frame, while the pose network uses the target frame and the previous frame of the target frame as input to predict the changes in the angle and position of the camera relative to the scene, and finally outputs the rotation matrix and translation matrix of the camera.

[0065] LitePose is a joint point heat map extraction network that uses fused deconvolution layers to eliminate the redundancy of high-resolution branches, allowing a single-branch structure to perform scale-aware multi-resolution fusion. At the same time, large convolution kernels are used in the backbone network to improve computational efficiency and reduce model parameters.

[0066] Colmap is an open source computer vision software that can be used to reconstruct the structure and camera pose of a 3D scene. It includes functions for image feature extraction, matching, 3D reconstruction, and dense point cloud generation. With Colmap, users can create an accurate 3D scene model from a set of images.

[0067] NeRF (Neural Radiance Fields, NeRF for short) is a neural network model for rendering realistic 3D scenes. It can learn a 3D representation of a scene from a set of 2D images, and then use this representation to generate new viewpoints, perspectives, lighting, and shadow effects. NeRF is based on the concept of neural radiance fields, which represents the amount of radiation at each 3D point in the scene as a function of the input viewpoint and direction. Through large-scale training and optimization, NeRF is able to generate highly realistic rendering results.

[0068] Embodiment 1

[0069] This embodiment provides a virtual camera posture adjustment method, and takes a performance scene as an example to illustrate the virtual camera posture adjustment method disclosed in this embodiment.

[0070] In this embodiment, the virtual camera pose adjustment method is mainly implemented based on implicit neural expression and multi-feature optimization, which can optimize the camera pose sequence in the virtual scene according to a given reference video so that the shooting effect of the rendered video is consistent with that of the reference video.

[0071] First, in this embodiment, the definition of the camera pose adjustment or optimization task (i.e., the preset virtual camera pose optimization model) can be expressed as:

[0072]

[0073] in, Φ is the camera parameter to be optimized, is the camera's extrinsic matrix, is the time, Φ represents the focal length of the camera. Θ is the parameter of the trained NeRF model, F Θ is the NeRF model with parameter Θ, is the video frame of the reference video.

[0074] Figure 1 A flowchart of a virtual camera posture adjustment method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method disclosed in this embodiment includes the following steps:

[0075] Step 110: determine an implicit expression model corresponding to a preset reference video through a dynamic neural radiation field, and determine a rendered video under a current camera trajectory pose through the implicit expression model according to any camera trajectory pose in a preset camera trajectory pose set.

[0076] It can be understood that the preset camera trajectory pose set may be an array or a set, and a plurality of preset camera trajectory poses are stored in the preset camera trajectory pose set.

[0077] In some embodiments, the preset virtual camera pose optimization model includes:

[0078]

[0079] in, is the camera’s extrinsic matrix, is the time, Φ is the focal length of the camera, Θ is the parameter of the implicit expression model, F Θ () is the implicit expression model corresponding to parameter Θ, is a video frame of the reference video;

[0080]

[0081] Where i is the video frame number of the reference video (e.g., the first frame, the second frame, etc.), * The video frame for the rendered video.

[0082] In this step, we first use the dynamic neural radiation field to implicitly neurally express a performance scene, taking a scene video shot from multiple angles as input, and obtain the function model corresponding to this scene through implicit neural expression. After obtaining the implicit expression of the scene, we use the implicit expression model of the scene and the randomly initialized camera trajectory pose as input, and use this model to render the scene video shot by the camera under this trajectory.

[0083] For a given scene S, we need to first express the scene with a function through the dynamic neural radiation field. Unlike manually building a scene in a virtual engine, we only need to give a video V of the scene. S A NeRF model can be trained. For a given scene video V S , we can get each video frame The corresponding camera extrinsic matrix. Specifically, Colmap is used to obtain the camera's extrinsic matrix.

[0084]

[0085] Since both the scene and the characters may move, a dynamic neural radiation field is used, which takes into account the time variable exist At this moment, from the camera external parameter matrix with focal length Θ The camera's viewing angle can be rendered and compared with the corresponding video frame Together they are used to optimize the NeRF model F Θ , get the NeRF parameters Θ of scene S S :

[0086]

[0087] After training, the dynamic NeRF model F of the scene is obtained. Θ After that, given any observation angle time And the focal length Φ of the camera, we can get the rendered image I at the corresponding perspective at the corresponding time * :

[0088]

[0089] It should be noted that the sizes of the video frame of the rendered video and the video frame of the reference video (eg, the height H and width W of the video frame) may remain the same.

[0090] For example, first, a multi-angle video of a performance scene is used as input, and Colmap is used to estimate the camera's 4×4 pose transformation matrix, and then the dynamic neural radiation field model of the corresponding scene is trained. After obtaining the dynamic neural radiation field model of the scene, different camera parameters are used as input to obtain scene rendering videos at different angles. Among them, the camera parameters used mainly include the observation angle time and the focal length Φ of the camera.

[0091] Step 120: input the rendered video and the preset reference video into a preset virtual camera pose optimization model to respectively determine the optical flow map, character joint point heat map and depth map of each video frame in the preset reference video and the rendered video.

[0092] It can be understood that the role of the preset reference video is to allow the model to learn the shooting style from the reference video, and finally generate a video output with a similar shooting style to the reference video in the virtual scene.

[0093] In this step, the reference video and the rendered video are used as input, and the optical flow information, character joint point information and depth information of the reference video and the rendered video are calculated respectively. These three are used to represent the movement of the camera and the character, the character composition and the perspective relationship of the picture in the reference video and the rendered video respectively.

[0094] Specifically, in order to make the generated neural rendering virtual video as consistent as possible with the reference video in terms of shooting effect, appropriate features are extracted based on the following three questions:

[0095] 1. Composition problem. Since characters are the key in performing arts programs, ensuring the consistency of composition means ensuring that the position and size of the actors in the reference video and the neural rendering video are as consistent as possible;

[0096] 2. Perspective problem: It is necessary to ensure that the distances between the elements of different scenes and the camera in the reference video and the neural rendering video are as consistent as possible. At the same time, the perspective problem also involves the size of the foreground and the middle scene;

[0097] 3. The problem of camera movement and character movement is also the core issue of the research. It is necessary to ensure that the movement of elements of different scenes in the reference video and the neural rendering video is as consistent as possible.

[0098] For the first composition problem, the joint features of the person are used to describe the composition of a single video. In order to ensure the differentiability of the overall model, the joint heat map is used to describe the joints of the human body. The joint heat map represents the confidence that each pixel position is a specific joint point. At the same time, the joint heat map also contains richer information than the coordinates of the person's joint points. A pre-trained downstream network LitePose is used to predict the joint heat map:

[0099]

[0100] in, Represent the heat map of the rendered video frame and the heat map of the reference video frame respectively. denote the rendered video and the reference video frame respectively, and Pose(·) denotes the processing process of the LitePose model.

[0101] Optionally, a pre-trained downstream network LitePose is used to predict joint point heat maps, a pre-trained downstream depth estimation network Monodepth2 is used to estimate the depth map of the video, and a pre-trained downstream optical flow estimation network RAFT is used to perform optical flow estimation.

[0102] Optionally, the joint point heat map feature uses Wasserstein (EMD) distance as the loss function, the optical flow feature uses EPE loss (End-Point Error Loss) as the loss function, and the deep feature uses MSE (Mean Square Error loss) as the loss function.

[0103] like Figure 4 As shown, Figure 4 This is a schematic diagram of the results of joint point heat map feature extraction. Pixels that may be joint points are represented by red and yellow. The higher the probability value, the brighter the color.

[0104] For the second perspective problem, the depth features of the video are used to describe the perspective information of a single video. The perspective of the picture is also part of the shooting composition. The perspective is related to the depth and stereoscopic sense of the scene being shot. It is used to reflect the distance between people or objects on the screen. At the same time, it can also divide the foreground, middle ground, and background of the two videos so that the area is consistent. The depth represents the distance between the object and the camera in three-dimensional space. It is used to determine the distance between objects. The two are relatively consistent. A pre-trained downstream depth estimation network Monodepth2 is used to estimate the depth map of the video:

[0105]

[0106] in, Respectively represent the depth map of the rendered video frame and the depth map of the reference video frame, They represent rendered video and reference video frames respectively, and Depth(·) represents the processing process of the Monodepth2 model.

[0107] like Figure 5 As shown, Figure 5 This is a schematic diagram of the result of depth map feature extraction. In the visualization result of the depth map, the lighter the color of the pixel, the farther the pixel is from the camera; conversely, the darker the color of the pixel, the closer the pixel is to the camera.

[0108] For the third problem of camera motion and character motion, optical flow is used to describe the motion information in a single video. Optical flow is a relatively important feature. Optical flow estimates the pixel-level displacement information between consecutive frames and can capture the motion information of characters in the video. In performing arts scenes, the background does not move in most cases, so the optical flow of the background can also relatively represent the motion of the shooting camera. Use the pre-trained downstream optical flow estimation network RAFT to perform optical flow estimation:

[0109]

[0110] in, and represents the horizontal and vertical optical flow components of the reference video, and and Represent the horizontal and vertical optical flow components of the rendered video, respectively. and They represent adjacent frames of rendered video and adjacent frames of reference video respectively, and Flow(·) represents the processing process of RAFT model.

[0111] like Figure 6 As shown, Figure 6This is a schematic diagram of the results of optical flow map feature extraction. In the visualization of the optical flow map, different hues express different directions of optical flow, and saturation represents the size of the optical flow at that point.

[0112] Step 130: respectively determine an optical flow loss function corresponding to the optical flow map, a joint point loss function corresponding to the character joint point heat map, and a depth loss function corresponding to the depth map.

[0113] In some embodiments, the optical flow loss function includes:

[0114]

[0115] in, and Respectively represent the horizontal and vertical optical flow components of the rendered video, and Represent the horizontal optical flow component and vertical optical flow component of the reference video respectively.

[0116] In some embodiments, the joint point loss function includes:

[0117]

[0118] Among them, d w is the Wasserstein distance, H * and They represent the character joint point heat maps of the rendered video frame and the reference video frame respectively, Hx represents the value of the i-th joint point in the character joint point heat map, and M is the number of joint points.

[0119] In some embodiments, the depth loss function includes:

[0120]

[0121] Where N is the number of pixels in the video frame, and They represent the depth value of the i-th pixel of the rendered video frame and the reference video frame respectively, H is the height of the reference video frame, and W is the width of the reference video frame.

[0122] Corresponding to the three different features extracted in step 120, three corresponding loss functions are designed in this step. Specifically:

[0123] 1. For the joint point heat map features used, the joint point loss of the character is introduced This loss function helps to optimize the problem of inconsistent positions and sizes of characters in the generated neural rendering video and the reference video, and can also constrain the movement of characters in the two videos to a certain extent. The Wasserstein (EMD) distance is used in the loss function. The Wasserstein distance pays more attention to the spatial layout and positional relationship between pixels in the heat map, that is, the similarity between the whole, rather than emphasizing the difference between pixels (single points) like metrics such as mean square error (MSE). That is, the Wasserstein distance pays more attention to metrics such as the overall size and shape of the character, while the MSE pays more attention to the difference between a single local joint point. By minimizing the EMD distance between the heat maps of the joint points of the two characters, the composition information can be transferred from the reference video to the differentiable rendering space of the model. For an image of size H×W, its joint point heat map Where M is the number of joints detected by the joint estimation network (for example, the number of joints detected by the pre-trained downstream network LitePose on an image of size H×W). The joint loss between the final joint heatmaps The calculation formula is as follows:

[0124]

[0125] Among them, d w represents the Wasserstein distance, H * and Represents the joint point heat map of the rendered video frame and the reference video frame, H i Represents the value of the i-th joint point in the heat map. The summation term in the formula represents the sum of the Wasserstein distances between the same joint points in the two images, while the regularization term S(H) in the formula represents the sum of the Wasserstein distances between different joint points on the same heat map. This term is used to ensure the similarity of the joint point shapes in the rendered video generated during the optimization process and the reference video.

[0126] 2. For deep features, depth loss is introduced This loss helps optimize the perspective relationship between the reference video and the generated neural rendering video to make them as consistent as possible. The loss function uses MSE as the loss function, and its calculation formula is as follows:

[0127]

[0128] Among them, N represents the number of pixels in the image. and Represent the depth value of the i-th pixel of the rendered video frame and the reference video frame respectively.

[0129] 3. For optical flow features, optical flow loss is introduced In terms of loss function, EPE loss (End-Point Error Loss) is used. EPE is a commonly used evaluation indicator in optical flow estimation tasks. It represents the Euclidean distance between the estimated optical flow and the true optical flow. Its calculation formula is as follows:

[0130]

[0131] in, and represents the horizontal and vertical optical flow components of the rendered video, and and Represent the horizontal optical flow component and vertical optical flow component of the reference video respectively.

[0132] The final overall model loss is composed of the sum of the loss functions of the above three downstream networks:

[0133]

[0134] Among them, α, β, and γ are the weights corresponding to each loss, Represents the overall model loss.

[0135] Among them, the weights corresponding to each loss are hyperparameters, and it is necessary to conduct multiple experiments and select better parameters from them, and then make corresponding settings.

[0136] Step 140, determining a guide area through a preset guide area determination model according to the optical flow map, character joint point heat map and depth map of the preset reference video and the rendered video, so as to highlight the key area whose gradient contribution meets the preset conditions.

[0137] In this step, the optical flow map, character joint heat map and depth map of the reference video and the rendered video are used as input, and the feature difference map between the two is calculated. After the difference map is obtained, it is normalized and summed, and finally the guidance area is output to distinguish the pixels containing large amounts of information and perform gradient back propagation, thereby accelerating the running speed of the virtual camera pose optimization model.

[0138] Optionally, the reference video and the rendered video optical flow map, the joint point heat map and the depth map are subtracted to calculate the difference, and the difference map of each feature is normalized using Min-Max normalization, and the normalized results are finally added to obtain the final guide area.

[0139] In some embodiments, the determining the guide area by a preset guide area determination model according to the optical flow map, character joint point heat map and depth map of the reference video and the rendered video includes:

[0140] Determine a first feature difference map according to the optical flow maps of the reference video and the rendered video; determine a second feature difference map according to the heat maps of the character joints of the reference video and the rendered video; determine a third feature difference map according to the depth maps of the reference video and the rendered video;

[0141] The guide area is determined by the preset guide area determination model according to the first feature difference map, the second feature difference map and the third feature difference map.

[0142] In some embodiments, the preset guidance area determination model includes:

[0143]

[0144] in, is the second characteristic difference map, H * To render the heat map of the joint points of the video frame, is the heat map of the joint points of the reference video frame; is the first characteristic difference map, O * To render the optical flow map of a video frame, is the optical flow map of the reference video frame; is the third characteristic difference map, D * To render the depth map of the video frame, is the depth map of the reference video frame.

[0145] During back propagation, the optimization process of camera parameters needs to calculate the gradient of the loss function with respect to all MLP (Multi-Layer Perceptron, MLP) parameters, which requires that all MLP parameters need to be considered for each pixel, resulting in increased computational complexity and a large amount of memory usage. The input of the downstream network requires a complete image, so it is impossible to reduce operations by sampling pixels; at the same time, some losses are not evenly distributed across the image, especially joint point losses, where many background areas cannot provide effective information. This uneven distribution results in the gradient of pixels in information-free areas being back propagated, which will reduce the optimization performance of the model or mislead the optimization direction of the model.

[0146] In order to solve the above problems, this embodiment proposes a guidance area calculated based on optical flow, depth and joint point heat map. In the final gradient backpropagation process, only the gradients of some important pixels will be backpropagated. This method not only avoids destroying the end-to-end training of the model, but also reduces the computational and memory overhead, while making the model more focused on pixels that contribute to the loss function without wasting resources on invalid pixels, thereby improving the efficiency and performance of the model. The guidance map G is calculated by the joint difference map between the generated joint point heat map, the optical flow map and the depth map to highlight the key areas that contribute more to the gradient (the preset condition can be set to a larger gradient contribution). Selectively backpropagating the gradients of these pixels can reduce memory consumption and provide a mechanism similar to attention to help convergence and focus on information-rich areas. As shown in the formula:

[0147]

[0148] Among them, H, O, and D represent the joint point heat map, optical flow map, and depth map, respectively, |·| represents the absolute value, and f * (x) is the standard Min-Max normalization, which is used to map the value of x to between [0-1]:

[0149]

[0150] like Figure 7 As shown, Figure 7 It is a schematic diagram of the result of the guide area. In the visualization of the result of the guide area, black pixels represent pixels with less information that are not back-propagated by the gradient, while white pixels represent pixels with more information that are back-propagated by the gradient.

[0151] Step 150: for any pixel point of the video frame in the rendered video, determine a first gradient according to the optical flow loss function, determine a second gradient according to the joint point loss function, and determine a third gradient according to the depth loss function; and determine a first return gradient corresponding to the first gradient, a second return gradient corresponding to the second gradient, and a third return gradient corresponding to the third gradient according to the guide area, and return the first return gradient, the second return gradient, and the third return gradient to optimize the shooting parameters of the virtual camera.

[0152] When the final gradient is returned, the gradient of all pixels calculated by the derivation of the loss function is multiplied with the guide area, and the gradient of some pixels is returned, that is, only the gradient value of the pixels in the information-rich area is returned to optimize the camera's shooting parameters. Repeat the above steps to iteratively optimize the camera's shooting parameters.

[0153] In some embodiments, determining the first gradient according to the optical flow loss function, determining the second gradient according to the joint point loss function, and determining the third gradient according to the depth loss function includes:

[0154] Performing a derivative operation on the optical flow loss function to obtain the first gradient;

[0155] Performing a derivative operation on the joint point loss function to obtain the second gradient;

[0156] A derivative operation is performed on the depth loss function to obtain the third gradient.

[0157] Optionally, the gradient of the loss is multiplied by the guide area to obtain the partial pixel gradient value to be returned, and this partial pixel gradient value is used to back-propagate and optimize the camera parameters. After multiple rounds of iterative optimization, the camera parameters and camera pose parameters that are most consistent with the reference video are finally obtained.

[0158] In some embodiments, determining, according to the guide area, a first return gradient corresponding to the first gradient, a second return gradient corresponding to the second gradient, and a third return gradient corresponding to the third gradient, respectively, comprises:

[0159] Taking the dot product result of the guide region and the first gradient as the first return gradient;

[0160] Taking the dot product result of the guide area and the second gradient as the second return gradient;

[0161] A dot product result of the guide region and the third gradient is used as the third return gradient.

[0162] After obtaining the guide region G, G acts as a "mask". The gradient obtained through the loss is multiplied by the guide region G to obtain the final returned pixel gradient value:

[0163]

[0164] in, Represents the gradient of the returned pixel containing important information, Represents the gradient of all pixels, G represents the guide area, and ⊙ represents the dot multiplication operation. The shooting parameters are optimized using the Adam optimization algorithm.

[0165] Furthermore, the above steps 120 to 150 are repeated to iteratively optimize the shooting parameters of the camera.

[0166] Finally, after multiple rounds of iterative optimization, the camera's shooting parameters gradually converge to the optimal value. In each round of iteration, the pixel gradient calculated by the derivation of the loss function is multiplied by the dot operation of the guide area, so that only the pixel gradient with important information is transferred and updated, thereby retaining the key details and structures in the image. This sophisticated gradient transfer method makes the optimization process more efficient and can better adapt to the shooting needs of different scenes and complex lighting conditions.

[0167] Further, in order to facilitate understanding of the technical solution of the application, you can also refer to Figure 2 and Figure 3 .in, Figure 2 is a flowchart of another virtual camera posture adjustment method according to an embodiment of the present application. Figure 3 A logical schematic diagram of a virtual camera pose optimization generation method provided in an embodiment of the present application.

[0168] In the virtual camera pose optimization method disclosed in this embodiment, a video of a performance scene is first used as input to train the dynamic neural radiation field, implicitly express the virtual performance scene, and obtain the function expression model under the scene; then a randomly initialized virtual camera pose sequence is used as input, and the rendered video under the camera pose trajectory is obtained in combination with the neural radiation field model; the reference video and the rendered video are used as input to extract the joint point heat map features, depth features and optical flow features of the two respectively; the three shooting features of the reference video and the rendered video are used as input to calculate the loss between each feature of the reference video and the rendered video in turn; the three shooting features of the reference video and the rendered video are used as input to calculate the difference map of each feature of the reference video and the rendered video in turn, and the guide area is obtained after normalization, and the information-rich area is selected for gradient feedback. Finally, the gradient is calculated through the loss function, and the gradient and the guide area are multiplied to obtain the final gradient to be returned, and the gradient is returned to optimize the camera parameters.

[0169] In summary, in the virtual camera pose optimization method disclosed in this embodiment, firstly, a performance scene video is given, and the virtual performance scene is expressed by the dynamic neural radiation field implicit neural expression, so that the rendered video of different angles can be rendered according to different camera pose sequences; then three differentiable downstream networks are used to extract the optical flow, character joint point heat map and depth features of the reference video and the rendered video respectively, which are used to characterize the motion information, composition information and camera perspective information of the camera and the character, calculate the loss between the features, and back-propagate the gradient to optimize the camera pose parameters, so that the final neural rendered video and the reference video input by the user (i.e., the preset reference video) have similar shooting techniques. In the gradient return, by calculating the guide area, the pixels with more information are selectively returned, which reduces the computational overhead.

[0170] Embodiment 2

[0171] Based on the above-mentioned embodiments, this embodiment discloses a specific example.

[0172] In this embodiment, the data set is selected using a performance held offline in a theater shot by a multi-angle camera array and a virtual scene video corresponding to the performance program.

[0173] In order to intuitively characterize the performance of this method on the existing data set, the root mean square error RMSE-ATE (Absolute Trajectory Error) of the final camera pose data and the root mean square error RMSE-RPE (Relative Pose Error) of the relative pose error are used as evaluation indicators for the accuracy of the camera pose. The relative pose error represents the error of the camera pose transformation between two frames within a fixed time interval, and the absolute trajectory error is used to measure the overall similarity between the two camera trajectories. It is used to measure the accuracy of the pose change between consecutive frames and represents the local accuracy.

[0174] According to the steps described in the above specific implementation manner, the experimental results obtained are shown in Table 1 below.

[0175] Table 1 Comparison of results between this application and existing methods

[0176]

[0177] As shown in Table 1, the present application has achieved higher performance on the existing datasets. It can be seen that the method based on implicit neural expression of the performing arts scene and the collaborative optimization of three features, namely optical flow, joint point heat map and depth, is more suitable for the optimization of camera pose in the performing arts scene. The effect that can be achieved is relatively consistent with the reference video, which can significantly improve the quality of the generated camera pose sequence and reduce the workload of scene reconstruction.

[0178] In order to more clearly reflect the performance and usability of the virtual camera pose optimization method of this application, this application uses a virtual scene video as a test scene. Through the steps in the method disclosed in this application, the camera pose trajectory is visualized in the scene. The results can be referred to Figure 8 .

[0179] like Figure 8 As shown, the reference video is shot in a way that the camera gradually moves backward, while the camera in the virtual rendering video predicted by EIIFM also gradually moves away, which is consistent with the reference video.

[0180] Embodiment 3

[0181] Based on the above-mentioned embodiments, this embodiment provides a virtual camera posture adjustment system.

[0182] This system embodiment can be used to execute the method embodiment of this application. For details not disclosed in this system embodiment, please refer to the method embodiment of this application. The system disclosed in this embodiment includes:

[0183] A scene video rendering module, used to determine an implicit expression model corresponding to a preset reference video through a dynamic neural radiation field, and determine a rendered video under a current camera trajectory pose through the implicit expression model according to any camera trajectory pose in a preset camera trajectory pose set;

[0184] A feature estimation module, used for inputting the rendered video and the preset reference video into a preset virtual camera pose optimization model to respectively determine the optical flow map, character joint point heat map and depth map of each video frame in the preset reference video and the rendered video;

[0185] A loss function determination module, used to respectively determine an optical flow loss function corresponding to the optical flow map, a joint point loss function corresponding to the character joint point heat map, and a depth loss function corresponding to the depth map;

[0186] A guide area determination module is used to determine the guide area through a preset guide area determination model according to the preset reference video and the optical flow map of the rendered video, the character joint point heat map and the depth map, so as to highlight the key area whose gradient contribution meets the preset conditions;

[0187] A gradient feedback module is used to determine, for any pixel point of a video frame in the rendered video, a first gradient according to the optical flow loss function, a second gradient according to the joint point loss function, and a third gradient according to the depth loss function; and to determine, according to the guide area, a first feedback gradient corresponding to the first gradient, a second feedback gradient corresponding to the second gradient, and a third feedback gradient corresponding to the third gradient, and feedback the first feedback gradient, the second feedback gradient, and the third feedback gradient to optimize the shooting parameters of the virtual camera.

[0188] It should be understood by those skilled in the art that the above modules or steps of the present application can be implemented by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation.

[0189] Technicians in the relevant field can clearly understand that, for the convenience and simplicity of description, the specific working process of each module in the virtual camera posture adjustment system can refer to the corresponding process in the aforementioned method embodiment, and this embodiment will not be repeated here.

[0190] Embodiment 4

[0191] Based on the above embodiments, this embodiment provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which can implement the method steps in the above method embodiments when executed by a processor, and this embodiment will not be repeated here.

[0192] Among them, the computer-readable storage medium may also include computer programs, data files, data structures, etc. alone, or include a combination thereof. The computer-readable storage medium or computer program may be specifically designed and understood by a technician in the field of computer software, or the computer-readable storage medium may be known and available to a technician in the field of computer software. Examples of computer-readable storage media include: magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as CDROM disks and DVDs; magneto-optical media, such as optical disks; and hardware devices, specifically configured to store and execute computer programs, such as read-only memory (ROM), random access memory (RAM), flash memory; or servers, app application stores, etc. Examples of computer programs include machine code (e.g., code generated by a compiler) and files containing high-level code, which can be executed by a computer using an interpreter. The described hardware device may be configured to be used as one or more software modules to perform the operations and methods described above, and vice versa. In addition, the computer-readable storage medium may be distributed in a networked computer system, and program code or computer programs may be stored and executed in a decentralized manner.

[0193] Embodiment 5

[0194] Based on the above embodiments, this embodiment provides a computer program product. The computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processor, all or part of the steps of the method in the above method embodiment are implemented, and this embodiment will not be repeated here.

[0195] Further, the computer program product may include one or more computer executable components configured to perform the embodiments when the program is run; the computer program product may also include a computer program tangibly contained on a readable medium, the computer program including program code for performing any method in the embodiments of the present disclosure. In such an embodiment, the computer program may be downloaded and installed from a network through a communication portion, and / or installed from a removable medium.

[0196] Embodiment 6

[0197] Based on the above embodiments, this embodiment provides an electronic device, which may include: one or more processors, a memory, a multimedia component, an input / output (I / O) interface, and a communication component.

[0198] One or more processors are used to execute all or part of the steps in the above method embodiments. The memory is used to store various types of data, which may include instructions of any application or method in the electronic device, and data related to the application.

[0199] One or more processors may be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and are used to execute the method in the aforementioned method embodiment.

[0200] The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0201] The multimedia component may include a screen and an audio component, the screen may be a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone, and the microphone is used to receive external audio signals. The received audio signal may be further stored in a memory or sent via a communication component. The audio component also includes at least one speaker for outputting audio signals.

[0202] The I / O interface provides an interface between one or more processors and other interface modules, which may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons.

[0203] The communication component is used for wired or wireless communication between the electronic device and other devices. Wired communication includes communication through network ports, serial ports, etc.; wireless communication includes: Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, 5G, or one or a combination of them.

[0204] In addition, it should be understood that the method or system disclosed in the embodiments provided in the present application can also be implemented in other ways. The method or system embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the methods and systems according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a computer program segment or a part of a computer program, and the module, computer program segment or a part of a computer program contains one or more computer programs for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings, and can actually be executed in parallel. They can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer programs.

[0205] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements that are inherent to such process, method, article, or apparatus. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, device or equipment including the elements; if there is a description of "first", "second", etc., it is only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features; in the description of this application, unless otherwise specified, the terms "multiple" and "multiple" mean at least two; if there is a description of a server, it should be noted that the server can be an independent physical server or terminal, or a server cluster composed of multiple physical servers, or a cloud server that can provide basic cloud computing services such as cloud servers, cloud databases, cloud storage and CDN; if there is a description of a smart terminal or mobile device in this application, it should be noted that the smart terminal or mobile device can be a mobile phone, a tablet computer, a smart watch, a netbook, a wearable electronic device, a personal digital assistant (PDA), an augmented reality technology device (AR), a virtual reality device (VR), a smart TV, a smart speaker, a personal computer (PC), a tablet computer, a smart phone, a smart TV, a smart speaker, a smart phone ... Computer, referred to as PC), etc., but not limited to this, the present application does not specifically limit the specific form of the smart terminal or mobile device.

[0206] Finally, it should be noted that, in the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "an example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0207] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary, and the contents described are only embodiments adopted for the convenience of understanding the present application, and are not intended to limit the present application. Any technician in the technical field to which the present application belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present application, but the scope of protection of the present application shall still be based on the scope defined in the attached claims.

Claims

1. A virtual camera posture adjustment method, characterized in that: The method comprises: Determine an implicit expression model corresponding to a preset reference video through a dynamic neural radiation field, and determine a rendered video under a current camera trajectory pose through the implicit expression model according to any camera trajectory pose in a preset camera trajectory pose set; Inputting the rendered video and the preset reference video into a preset virtual camera pose optimization model to respectively determine an optical flow map, a character joint point heat map, and a depth map of each video frame in the preset reference video and the rendered video; Respectively determine an optical flow loss function corresponding to the optical flow map, a joint point loss function corresponding to the character joint point heat map, and a depth loss function corresponding to the depth map; According to the preset reference video and the optical flow map of the rendered video, the character joint point heat map and the depth map, a guide area is determined by a preset guide area determination model to highlight the key area whose gradient contribution meets the preset conditions; For any pixel point of the video frame in the rendered video, a first gradient is determined according to the optical flow loss function, a second gradient is determined according to the joint point loss function, and a third gradient is determined according to the depth loss function; and a first return gradient corresponding to the first gradient, a second return gradient corresponding to the second gradient, and a third return gradient corresponding to the third gradient are determined according to the guide area, and the first return gradient, the second return gradient, and the third return gradient are returned to optimize the shooting parameters of the virtual camera.

2. The virtual camera posture adjustment method according to claim 1, characterized in that: The preset virtual camera pose optimization model includes: in, is the camera’s extrinsic matrix, is the time, Φ is the focal length of the camera, F Θ () is an implicit expression model, is a video frame of the reference video; Where i is the video frame number of the reference video, I * The video frame for the rendered video.

3. The virtual camera posture adjustment method according to claim 1, characterized in that: The preset guidance area determination model includes: in, is the second characteristic difference map, H * To render the heat map of the joint points of the video frame, is the heat map of the joint points of the reference video frame; is the first characteristic difference map, O * To render the optical flow map of a video frame, is the optical flow map of the reference video frame; is the third characteristic difference map, D * To render the depth map of the video frame, is the depth map of the reference video frame; f * () indicates normalization processing.

4. The virtual camera posture adjustment method according to claim 1, characterized in that: The determining of the first gradient according to the optical flow loss function, the determining of the second gradient according to the joint point loss function, and the determining of the third gradient according to the depth loss function comprises: Performing a derivative operation on the optical flow loss function to obtain the first gradient; Performing a derivative operation on the joint point loss function to obtain the second gradient; A derivative operation is performed on the depth loss function to obtain the third gradient.

5. The virtual camera posture adjustment method according to claim 1, characterized in that: The determining, according to the guide area, a first return gradient corresponding to the first gradient, a second return gradient corresponding to the second gradient, and a third return gradient corresponding to the third gradient, respectively, comprises: Taking the dot product result of the guide region and the first gradient as the first return gradient; Taking the dot product result of the guide area and the second gradient as the second return gradient; A dot product result of the guide region and the third gradient is used as the third return gradient.

6. The virtual camera posture adjustment method according to claim 1, characterized in that: The optical flow loss function includes: in, and Respectively represent the horizontal optical flow component and vertical optical flow component of the rendered video, and Represent the horizontal optical flow component and vertical optical flow component of the reference video respectively.

7. The virtual camera posture adjustment method according to claim 1, characterized in that: The joint point loss function includes: Among them, d w is the Wasserstein distance, H * and Represents the heat map of the character joints of the rendered video frame and the reference video frame, H i It represents the value of the i-th joint point in the character joint point heat map, and M is the number of joint points.

8. The virtual camera posture adjustment method according to claim 1, characterized in that: The depth loss function includes: Where N is the number of pixels in the video frame, and They represent the depth value of the i-th pixel of the rendered video frame and the reference video frame respectively, H is the height of the reference video frame, and W is the width of the reference video frame.

9. A virtual camera posture adjustment system, characterized in that: include: A scene video rendering module, used to determine an implicit expression model corresponding to a preset reference video through a dynamic neural radiation field, and determine a rendered video under a current camera trajectory pose through the implicit expression model according to any camera trajectory pose in a preset camera trajectory pose set; A feature estimation module, used for inputting the rendered video and the preset reference video into a preset virtual camera pose optimization model to respectively determine the optical flow map, character joint point heat map and depth map of each video frame in the preset reference video and the rendered video; A loss function determination module, used to respectively determine an optical flow loss function corresponding to the optical flow map, a joint point loss function corresponding to the character joint point heat map, and a depth loss function corresponding to the depth map; A guide area determination module is used to determine the guide area through a preset guide area determination model according to the preset reference video and the optical flow map of the rendered video, the character joint point heat map and the depth map, so as to highlight the key area whose gradient contribution meets the preset conditions; A gradient feedback module is used to determine, for any pixel point of a video frame in the rendered video, a first gradient according to the optical flow loss function, a second gradient according to the joint point loss function, and a third gradient according to the depth loss function; and to determine, according to the guide area, a first feedback gradient corresponding to the first gradient, a second feedback gradient corresponding to the second gradient, and a third feedback gradient corresponding to the third gradient, and feedback the first feedback gradient, the second feedback gradient, and the third feedback gradient to optimize the shooting parameters of the virtual camera.

10. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images