Virtual camera behavior generation method and system

Abstract

The application provides a virtual camera behavior generation method and system, which fuses global performance scene state features and global camera behavior trend features into performance scene global multi-data fusion features, and pre-constructs a camera behavior trend coding module based on the global camera behavior trend features and a camera behavior generation decoding module based on the global performance scene state features; fuses a performance scene multi-data loss function to construct a camera behavior generation model based on the virtual performance global multi-data fusion features to predict a camera behavior sequence, and transmits the camera behavior sequence into a performance scene visualized interactive interface based on a Unity editor to display the generated camera behavior trajectory. The application fully utilizes the global correlation of the camera behavior trend and the strong correlation between the camera behavior trend and the global state of the performance scene to predict the performance scene camera behavior trajectory, and has the advantage of high camera behavior prediction accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer vision technology and virtual scene performance technology, and more specifically, to a method and system for generating camera behavior based on global multi-dimensional data fusion features in virtual performance. Background Technology

[0002] With the rapid development of the media industry, the ever-increasing demand for video content production has brought considerable challenges to high-quality camera movement control. Traditional methods of manually controlling camera movement throughout the entire process consume a significant amount of time and manpower, leading to a growing need for automated generation of camera motion trajectories. Simultaneously, the explosive growth in video volume in the self-media sector has increased the speed requirements of content production, further complicating manual camera movement control. The cost of achieving high-efficiency, high-quality, and high-stability shooting is gradually rising. Therefore, the industry's demand for automated camera movement control is increasing, and intelligent shooting is gradually becoming a necessity.

[0003] However, besides fulfilling low-dimensional factors such as composition, camera movement direction, and character visibility, camera movement also involves higher-dimensional, albeit implicit, but industry-recognized, rules of camera movement. During filming, the depiction of a story or the expression of emotion within a segment requires certain rules to control the way the camera moves. The validity of this higher-dimensional emotional characteristic is related to the director's and audience's perceived understanding; it is a form of implicit encoding.

[0004] In existing camera behavior calculation methods, a common approach to achieve low-dimensional control is keyframe specification control. This involves the user pre-setting several keyframes, specifying their positions and the camera's state at each keyframe. The algorithm then automatically fills in the gaps between keyframes to predict camera motion frame by frame. However, this method only addresses low-dimensional geometric features in the photographic process and does not consider the implicit camera movement rules or the learning and expression of high-dimensional photographic styles.

[0005] To achieve high-dimensional emotional expression, a common approach in existing camera behavior computation is data-driven style feature extraction. This method associates frame-by-frame camera movements or the relationship between the camera and the actor with specific style categories, supervising the network to form a specific data-style mapping relationship. This ensures the network has the ability to learn the implicit shooting style of the photographic sequence. While style feature extraction methods can achieve network training results at the implicit encoding level, the entire camera behavior generation process is left to the neural network. For the user, this means controlling the camera movement style only by specifying a reference sequence, lacking control over the camera behavior during the generation process.

[0006] Existing methods for calculating camera behavior that consider both keyframes and camera movement style combine keyframe-specified data format with style feature extraction, employing an end-to-end approach to train the network using a style feature extraction network and a camera behavior prediction network. However, this method, which uses recurrent neural networks to process information and calculate camera behavior, does not fully utilize the features of camera behavior data and scene performance state characteristics during the performance process. This is mainly reflected in two aspects: First, the keyframe control information, reference camera movement style features, and scene performance state information are directly concatenated and fused during the data input stage, failing to highlight the strong correlation between the predicted camera data and the data at the keyframes and scene performance state information. Second, the use of recurrent neural networks or long short-term memory recurrent neural networks to output and provide feedback on camera behavior frame by frame does not fully utilize the data correlation of distant but related camera behaviors in the virtual camera motion sequence, and as the amount of camera behavior sequence data increases, it is prone to loss gradient explosion.

[0007] Therefore, there is an urgent need for a camera behavior calculation scheme that can overcome the above-mentioned defects. Summary of the Invention

[0008] In view of the above-mentioned problems in the prior art, the purpose of the present invention is to provide a virtual camera behavior generation method and system, which is a camera behavior generation method based on the global multi-dimensional data fusion features of virtual performance scene state features and global camera behavior trend features, so as to improve the global modeling performance of performance process and camera motion state and optimize the camera behavior prediction effect while realizing camera behavior analysis and generation that takes into account both high and low dimensions.

[0009] According to one aspect of the present invention, a method for generating virtual camera behavior is provided, comprising:

[0010] The Unity editor's performance scene visualization interface allows for the acquisition of global performance scene state features and keyframe camera state features.

[0011] Based on the global performance scene state features and keyframe camera state features, virtual performance global multi-dimensional data fusion features are obtained; wherein, the virtual performance global multi-dimensional data fusion features include global camera behavior trend features and global performance scene state features, the global camera behavior trend features include historical camera motion features, the keyframe camera state features, and reference camera movement style features extracted from a preset reference video through a preset global camera behavior trend feature extraction module.

[0012] Based on the preset camera behavior trend encoding module, the global camera behavior trend features are used as input, and motion trend vectors and arrival time signals related to key frame information are introduced. The intermediate vector of global camera trend encoding is encoded by using self-attention and layer-by-layer convolutional distillation calculation methods.

[0013] Based on the preset camera behavior generation and decoding module, the global performance scene state features are used as input, and the intermediate vector of the global camera trend encoding is fused. The cross self-attention calculation method is used to obtain the predicted camera behavior sequence.

[0014] The camera behavior trajectory of the predicted camera behavior sequence is obtained through the Unity editor's performance scene visualization interface.

[0015] One optional approach is that the global camera behavior trend feature extraction module extracts reference camera movement style features from a preset reference video, including: the global camera behavior trend feature extraction module performs skeleton recognition, coordinate extraction, and style feature mapping on the video sequence of the preset reference video to obtain the reference camera movement style features.

[0016] One optional approach is to perform skeletal recognition, coordinate extraction, and style feature mapping on the video sequence of the preset reference video to obtain the reference camera movement style features, including:

[0017] For each frame in the reference video, the screen coordinates of seven joints—head, neck, left shoulder, right shoulder, hip, left hip, and right hip—are extracted using a DeKR network.

[0018] Calculate the joint features of the actors in a frame using the coordinates of 7 joint points of the two actors in that frame.

[0019] Based on the feature estimation network pre-trained model of convolutional layer and 8-frame sliding window, the coordinate system of the screen joint point of the actor is mapped frame by frame, and the relevant parameters are output.

[0020] Based on the relevant parameters and the screen position information of the actor's head, the actor-camera parameter pair in the complex surface coordinate system of the frame in the reference video is obtained;

[0021] The reference video actor-camera parameter pairs are processed using a long short-term memory network and mapped into low-dimensional vectors containing implicit camera movement rules, serving as reference camera movement styles.

[0022] Among them, the optional solutions are:

[0023] The preset camera behavior trend encoding module takes the global camera behavior trend features as input and introduces motion trend vectors and arrival time signals related to keyframe information, including:

[0024] Determine the input frame data for the camera behavior trend encoding module corresponding to the next camera behavior to be tested;

[0025] The input data of the camera behavior trend encoding module is stitched together using a preset time window;

[0026] The input data of the stitched camera behavior trend encoding module is embedded and encoded using the Transformer model; at the same time, the input data is positionally encoded to ensure that each frame carries sequence information during the self-attention calculation process.

[0027] One optional approach is that the intermediate vector for encoding the global camera trend using the self-attention and layer-by-layer convolutional distillation calculation method includes:

[0028] The global self-attention mechanism after computational position encoding;

[0029] Convolutional distillation is performed on the self-attention results of each layer to adjust the feature weights between samples in a longer sequence.

[0030] One possible approach is that the camera behavior trend encoding module employs two self-attention layers, each containing a self-attention calculation process and a convolutional distillation operation; wherein, for the encoded computational data, after computation by the camera behavior trend encoding module, the output latent variable serves as the intermediate vector of the global camera trend encoding.

[0031] One optional approach is that the camera behavior-based decoding module, using the global performance scene state features as input, includes:

[0032] The arrival information of keyframes before and after the global performance scene state features and the actor state features are used as feature data for the input of the frame to be predicted.

[0033] At the same time, the frame to be predicted is spliced ​​with the data of the known camera behavior of the previous 50 frames at the corresponding frame position to become the prior data in the data;

[0034] The input data of the stitched camera behavior generation decoding module is subjected to data transformation encoding and position encoding.

[0035] One possible approach is that the camera behavior generation and decoding module predicts the 5D complex surface coordinate camera behavior vector of the corresponding frame and fills it with a 5D place_holder zero vector.

[0036] One possible approach is to fuse the intermediate vector of the global camera trend encoding and use a cross-self-attention calculation method to obtain the predicted camera behavior sequence, including:

[0037] The single-layer global self-attention is calculated based on the data obtained after performing data transformation encoding and position encoding on the input data of the stitched camera behavior generation decoding module.

[0038] Cross-self-attention calculation is performed based on the single-layer global self-attention and the hidden variable H;

[0039] The cross-attention calculation results are processed through dropout and norm layers, and then through a linear layer to obtain actor information and camera behavior information of the frame to be predicted after decoding.

[0040] According to another aspect of the present invention, a virtual camera behavior generation system is provided, which is based on the global multi-dimensional data fusion features of virtual performance, comprising:

[0041] The performance scene feature acquisition unit is used to acquire global performance scene state features and key frame camera state features through the Unity editor's performance scene visualization and interactive interface.

[0042] The fusion feature acquisition unit is used to acquire virtual performance global multi-dimensional data fusion features based on the global performance scene state features and key frame camera state features; wherein, the virtual performance global multi-dimensional data fusion features include global camera behavior trend features and global performance scene state features, the global camera behavior trend features include historical camera motion features, the key frame camera state features, and reference camera movement style features extracted from a preset reference video by a preset global camera behavior trend feature extraction module;

[0043] The camera behavior trend encoding unit is used to encode the intermediate vector of the global camera behavior trend based on the preset camera behavior trend encoding module, taking the global camera behavior trend features as input, and introducing the motion trend vector and arrival time signal related to the key frame information. It uses self-attention and layer-by-layer convolutional distillation calculation methods to encode the intermediate vector of the global camera trend encoding.

[0044] The camera behavior generation and decoding unit is used to obtain the predicted camera behavior sequence based on the preset camera behavior generation and decoding module, taking the global performance scene state features as input, fusing the intermediate vector of the global camera trend encoding, and using the cross self-attention calculation method.

[0045] The camera behavior trajectory generation unit is used to obtain the camera behavior trajectory of the predicted camera behavior sequence through the Unity editor's performance scene visualization and interactive interface.

[0046] The virtual camera behavior generation method and system of the present invention, by integrating global camera behavior trend features and global performance scene state features, achieves both high- and low-dimensional camera behavior analysis and generation, while effectively improving the global modeling performance of the performance process and camera motion state, and optimizing the camera behavior prediction effect. Furthermore, the global camera behavior trend features include reference cloud scene style features, historical camera motion features, and keyframe camera state features. By using a Transformer-like model encoding structure to process global camera data, global feature encoding can be achieved while avoiding the risk of gradient explosion.

[0047] To achieve the foregoing and related objectives, one or more aspects of the invention include the features that will be described in detail below and particularly pointed out in the claims. The following description and accompanying drawings illustrate certain exemplary aspects of the invention. However, these aspects indicate only a few of the various ways in which the principles of the invention can be used. Furthermore, the invention is intended to include all such aspects and their equivalents. Attached Figure Description

[0048] Other objects and results of the invention will become more apparent and readily understood with reference to the following description taken in conjunction with the accompanying drawings and the contents of the claims, and with a more complete understanding of the invention. In the drawings:

[0049] Figure 1 This is a flowchart of a virtual camera behavior generation method according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the data processing logic of the virtual camera behavior generation method according to an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram illustrating the effect of visual keyframe control of the Unity interface according to an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram of the global camera behavior trend feature extraction module according to an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of a camera behavior trend encoding module according to an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of a camera behavior generation decoding module according to an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the logical structure of a virtual camera behavior generation system according to an embodiment of the present invention;

[0056] Figure 8According to embodiments of the present invention, camera behavior generation results are generated under different reference video styles;

[0057] Figure 9 According to embodiments of the present invention, camera behavior is driven by different keyframe information to generate results.

[0058] In all the accompanying drawings, the same reference numerals indicate similar or corresponding features or functions. Detailed Implementation

[0059] In the following description, numerous specific details are set forth for illustrative purposes and to provide a thorough understanding of one or more embodiments. However, it will be apparent that these embodiments may also be implemented without these specific details. In other instances, well-known structures and devices are shown in block diagram form for ease of description of one or more embodiments.

[0060] Addressing the problems in existing camera behavior calculation schemes, this invention uses some pre-trained models from existing work as the basis for data preprocessing and designs a camera behavior generation model for virtual performance scenes based on global multi-source data fusion features. The camera behavior generation model based on global multi-source data fusion features adopted in this invention uses the basic encoder-decoder structure of the Informer model. At the encoding end, attention is calculated on the global camera behavior trend features formed by fusing reference camera movement style features, historical camera motion features, and keyframe camera state features. At the decoding end, the global performance scene state features formed by fusing keyframe position features and actor state features are analyzed to further predict camera behavior, improve the global correlation of camera sequences, the global correlation of performance scene states, and the correlation of keyframe information, thereby improving the camera behavior prediction effect.

[0061] To better illustrate the technical solution of the present invention, some of the technical terms involved in the present invention will be briefly explained below.

[0062] Transformer: A Natural Language Processing (NLP) model used to convert human natural language into computer-readable instructions. It is entirely based on a self-attention mechanism, making it suitable for parallel computation. The complexity of its model itself leads to higher accuracy and performance than previously popular Recurrent Neural Networks (RNNs).

[0063] Unity is a real-time 3D interactive content creation and operation platform.

[0064] The Informer model is a Long Short-Term Memory (LSTF) network based on the Transformer architecture, designed to solve the problem of long-sequence time series prediction. It has three key features: 1. ProbSparse Self-Attention Mechanism: This innovative self-attention mechanism achieves O(Llog L) time complexity and memory usage. By simplifying self-attention computation, it effectively captures long-term dependencies between sequences, improving prediction accuracy. 2. Self-Attention Distillation: The Informer model employs self-attention distillation, which reduces the input to cascaded layers, effectively handling extremely long input sequences. This technique enhances the model's ability to handle long sequences, making it suitable for a wider range of applications. 3. Generative Decoder: The Informer model uses a generative decoder, which can predict the entire long-sequence at once, rather than making predictions incrementally. This significantly improves the inference speed for long-sequence prediction, making the Informer model more competitive in practical applications.

[0065] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0066] Figure 1 The flowchart of the virtual camera behavior generation method according to the present invention is shown. Figure 2 The data processing logic of a virtual camera behavior generation method according to an embodiment of the present invention is shown.

[0067] The virtual camera behavior generation method provided by this invention is based on the global multi-dimensional data fusion features of virtual performances. Here, virtual camera behavior refers to the frame-by-frame change in the position of the scene camera during a virtual performance; the global multi-dimensional data fusion features of virtual performances refer to the global performance scene state features, including keyframe position features and actor state features, as well as the global camera behavior trend features, including reference camera movement style features, historical camera motion features, and keyframe camera state features. For example... Figure 1 and Figure 2 As shown in the figure, the virtual camera behavior generation method provided in this embodiment mainly includes the following steps:

[0068] S110: Obtain global performance scene state features and keyframe camera state features through the Unity editor's performance scene visualization and interactive interface;

[0069] S120: Based on the global performance scene state features and keyframe camera state features, obtain virtual performance global multi-dimensional data fusion features; wherein, the virtual performance global multi-dimensional data fusion features include global camera behavior trend features and global performance scene state features, the global camera behavior trend features include historical camera motion features, the keyframe camera state features, and reference camera movement style features extracted from a preset reference video by a preset global camera behavior trend feature extraction module;

[0070] S130: Based on the preset camera behavior trend encoding module, the global camera behavior trend features are used as input, and motion trend vectors and arrival time signals related to key frame information are introduced. The intermediate vector of global camera trend encoding is encoded by using self-attention and layer-by-layer convolutional distillation calculation methods.

[0071] S140: Based on the preset camera behavior generation and decoding module, the global performance scene state features are used as input, and the intermediate vector of the global camera trend encoding is fused. The cross self-attention calculation method is used to obtain the predicted camera behavior sequence.

[0072] S150: Obtain the camera behavior trajectory of the predicted camera behavior sequence through the Unity editor's performance scene visualization interface.

[0073] As can be seen from the above steps, the virtual camera behavior generation method in this embodiment is mainly based on the global multi-dimensional data fusion features of virtual performance, and can drive the generation of virtual scene camera behavior sequences according to specific reference styles, specific performance scenes and specific keyframe information.

[0074] The virtual camera behavior generation method provided by the present invention will be described in more detail below through specific embodiments.

[0075] Figure 2 This is a schematic diagram of the data processing logic of the virtual camera behavior generation method according to an embodiment of the present invention, such as... Figure 2 As shown, the virtual camera behavior generation method provided by the present invention mainly includes five parts in terms of implementation logic: a performance scene visualization interactive interface, a reference video camera behavior trend feature extraction, a virtual performance global multi-dimensional data fusion feature acquisition, a camera behavior trend encoding model, and a camera behavior generation decoding module. The following will provide a detailed illustrative description of the above steps S110 to S150 in conjunction with these five parts.

[0076] Step S110: Obtain global performance scene state features and keyframe camera state features through the Unity editor's performance scene visualization interface.

[0077] Figure 3This is a schematic diagram illustrating the effect of visual keyframe control of the Unity interface according to an embodiment of the present invention. Figure 2 and Figure 3 As shown, by outputting keyframe information and actor information through a performance scene visualization interface, the overall performance scene state features and the keyframe camera state features within the overall camera behavior trend features are obtained. Specifically, as an example, this process includes:

[0078] 1a) Specify the keyframe position information for the performance scene. Use the Slide UI component in Unity to drag the slider to control the actor's performance progress, thus representing the keyframe position information, such as... Figure 3 As shown in region A. The value range of the slider is [0,1]. For the current slider position... Keyframe positions, the recorded keyframe positions The data is as follows:

[0079]

[0080] Where N represents the number of frames in the entire performance process, and to ensure reasonableness, this step S110 uses rounding down.

[0081] 1b) Specify the camera state information for keyframes in the performance scene, and use the keyboard to control the camera position in the current frame, which will be used as the camera position for this keyframe. During the operation, the program reads the coordinates of the camera and the heads of the two actors:

[0082] pos Cam =(x Cam y Cam , z Cam ), pos A =(x A y A , z A ), pos B =(x B y B , z B )

[0083] 1c) Construct keyframe state features for the performance process. Based on the network structure data requirements of this embodiment, call relevant program functions to transform the camera coordinates to a complex surface coordinate system:

[0084]

[0085] Among them, (p A p A ), (p B p B θ represents the coordinates of the two actors' heads in the camera's view. These represent the yaw and pitch angles of the camera relative to the actor in space, respectively. After outputting the camera's complex surface coordinates for the current frame, this frame is marked as a keyframe, and combined with the keyframe position from step 1a) as the keyframe state feature of the performance process:

[0086]

[0087] 1d) Construct arrival information before and after keyframes, including arrival distance information of the previous keyframe and smooth departure information of the next keyframe.

[0088] Based on the keyframe position data in the global performance scene state characteristics, the arrival time signal z is used for each frame i in the time series. tta，i This represents the distance between the current frame and the next keyframe, ensuring the smoothness of the camera behavior prediction process as it approaches the next keyframe and avoiding abrupt changes in camera behavior at the keyframe. It is represented using a sine vector:

[0089]

[0090] In this embodiment, tta obtains the number of frames from the current frame to the next keyframe, with a base value of 10000 and a data dimension of 32, meaning the arrival time signal is a 32-dimensional sine vector.

[0091] Based on the keyframe camera behavior data in the global performance scene state features, calculate the velocity parameters of the camera motion state at the next keyframe immediately preceding each frame i. This ensures that the camera behavior prediction process between keyframes smoothly moves away from the previous keyframe, avoiding abrupt changes in camera behavior after the keyframe.

[0092]

[0093] Specifically, as an example, in a specific embodiment of the present invention, the number of motion vector estimation steps is 5, and the last 4 frames of the camera's motion direction at the keyframe are used as parameters.

[0094] 1e) Extract actor state features. Using the actor output mode of the script in the program, output frame-by-frame actor information:

[0095]

[0096] Where AB is a straight line passing through the midpoint of the two actors' bodies; h A h B These represent the directions of the two actors' heads and the angle between them and line AB, respectively; h AB Indicates the angle between the directions the two actors' heads are facing; d AB Indicates the distance between two actors; s A s BLet s represent the angle between the normal vector of the line connecting the shoulders of the two actors and the angle between line AB; AB M represents the angle between the normal vectors of the lines connecting the shoulders of the two actors; M is a two-dimensional vector that specifies the main character in the current scene.

[0097] Thus, the keyframe information features and actor state information features together constitute the global performance scene state features; at the same time, the keyframe camera position and the arrival information before and after the keyframe together constitute the keyframe camera state features.

[0098] Figure 4 This is a schematic diagram of a global camera behavior trend feature extraction module according to an embodiment of the present invention. Figure 2 and Figure 4 As shown in step S120, based on a pre-built global camera behavior trend feature extraction module based on reference video, existing methods are used to perform skeleton recognition, coordinate extraction, and style feature mapping on the input reference video sequence to obtain reference camera movement style features. These features, together with historical camera motion features and scene keyframe camera state features during the performance, constitute the global camera behavior trend features. The historical camera motion features can be specified by the user or are default camera motion features.

[0099] Specifically, as an example, the process includes:

[0100] 2a) Reference video human pose estimation: Extract key point information of the main characters from the input reference video. For each frame f in the input reference video... t The DeKR network is used to extract the screen coordinates of a preset number of joints. In a specific embodiment of the present invention, the screen coordinates of seven joints—head, neck, left shoulder, right shoulder, hip, left hip, and right hip—can be extracted.

[0101]

[0102] in, This represents the screen coordinates of a certain joint point n, and DeKR(·) represents the processing procedure of the human pose estimation model.

[0103] Calculate the human joint features of the actors in a frame using the coordinates of 7 joint points of two actors:

[0104]

[0105] Where AB is a straight line passing through the midpoint of the two actors' bodies, p A p B For the screen position information of the two actors' heads, n A ,n B The h represents the joint positions of the two actors' necks, left shoulders, and right shoulders, respectively.A ,h B The angle between the orientation of the two actors' heads and the line AB;

[0106] 2b) Reference video shooting feature estimation: Map the coordinates representing the positions of people and cameras in the reference video to a specific coordinate space, using the existing convolutional layer-based feature estimation network pre-trained model Esti(·). Based on a preset number of frames (e.g., 8 frames) sliding window, map the frame-by-frame actor screen joint coordinate system obtained in step 2a) to output relevant parameters:

[0107]

[0108] y t =Esti(u t-4 u t-3 u t-2 u t-1 u t u t+1 u t+2 u t+3 )

[0109] Where y represents the output data of the feature estimation network, and Φ and θ in the data represent the yaw angles of the camera relative to the actor in the vertical and horizontal directions, respectively; s A s B Let s represent the angle between the normal vector of the line connecting the shoulders of the two actors and the angle between line AB; AB This represents the angle between the normal vectors of the lines connecting the shoulders of the two actors; u t-4 This represents the keypoint features from the previous four frames, and similarly, the others represent the data within the input time window of the feature estimation network.

[0110] Based on the y-value of the current reference video frame t And computable screen position information of the actor's head p A p B This allows you to directly obtain the actor-camera parameter pair in the complex surface coordinate system for that frame in the reference video:

[0111]

[0112] in, and Represent the actor's state and the camera's state in the complex surface coordinate system in frame t, respectively:

[0113]

[0114] 2c) Reference camera movement style extraction: The reference video actor-camera parameter pairs obtained in step 2b) are processed using a long short-term memory network and mapped into low-dimensional vectors containing implicit camera movement rules, serving as the reference camera movement style.

[0115] z c =(FC(LSTM(X) clip ))) 120

[0116] Where FC(·) and LSTM(·) represent the Long Short-Term Memory network and the fully connected network in the style feature extraction process, respectively, X clip This represents a data segment from the input data in the style feature extraction process, represented by x in the above. t The style vector z is composed of 120 frames, with the output of the last frame of the network used as the result. c .

[0117] Specifically, as an example, in one specific embodiment of the present invention, 120 frames are randomly selected from the input sequence. As a reference style sequence, the network's initial hidden states are set to all zeros. The output style code is mapped to a 4-dimensional vector, and the style code z of the last frame's output is taken as the latent representation of the camera movement rules. c .

[0118] The style code representing the reference camera movement style characteristics is combined with the historical camera behavior sequence of the performance scene, and combined with the key frame camera state characteristics obtained in step S110, to jointly constitute the global camera behavior trend characteristics.

[0119] Figure 5 This is a schematic diagram of a camera behavior trend encoding module according to an embodiment of the present invention.

[0120] like Figure 2 and Figure 5 As shown in step S130, based on the pre-selected camera behavior trend encoding module constructed based on global camera behavior trend features, the global camera behavior trend features are used as input, and motion trend vectors and arrival time signals related to keyframe information are introduced. Using self-attention and layer-by-layer convolutional distillation calculation methods, the intermediate vector of the global camera trend encoding is encoded. Specifically, as an example, this process includes:

[0121] 3a) Construct global camera behavior trend feature encoding data. The global camera behavior trend features obtained in step S120 are used as input to the camera behavior trend encoding module in this step through simple data preprocessing.

[0122] In one specific embodiment of the present invention, the input time window is 200 frames. For the case where the frame index of the next camera behavior to be tested is i, the corresponding input frame data for the camera behavior trend encoding module is:

[0123]

[0124] Where Concat indicates concatenation along the last dimension; x i-200 The actor-camera parameter pair data is the first 200 frames of the frame to be predicted; z c For reference camera movement style; The camera state for the next adjacent keyframe; v i The velocity constraint parameter is 25-dimensional, representing the velocity parameters of the camera motion state at the immediately preceding keyframe. It is calculated by moving 5 frames along the camera motion direction at the keyframe. tta This is the arrival time signal, which indicates the number of time steps from the current frame to the next keyframe;

[0125]

[0126] Where tta represents the number of frames from the previous keyframe to the current frame, b is the baseline value of 10000, and d is the data dimension. In a specific embodiment of the present invention, the value of d is 512.

[0127] In this embodiment, the camera behavior trend encoding module input is a 200-frame data stitching S:

[0128] S = Concat t (s t ), t∈[i, i+200]

[0129] Among them, Concat t This indicates splicing along the time (frame) dimension.

[0130] 3b) Input data encoding: Perform Transformer model data embedding encoding on the module input data S:

[0131]

[0132] Among them, Q e ,K e V e The data q corresponding to each frame e k e Both v and v are 512-dimensional. Simultaneously, positional encoding is applied to the data to ensure that each frame carries sequence information during the self-attention computation process:

[0133]

[0134] X = X + PE

[0135] Where X refers to the input data. pos represents the position of the current frame data in this data segment, b is the baseline value of 10000, and d is the data dimension. In a specific embodiment of the present invention, d is 512.

[0136] 3c) Self-attention operation: Calculate the global self-attention mechanism based on the positionally encoded Q, K, and V.

[0137]

[0138] Specifically, as an example, in the encoder constructed in this embodiment, the number of heads in the multi-head self-attention layer is set to 8, and the number of layers is 2. After attention calculation, a dropout layer is used to deactivate some units to further prevent overfitting when the amount of data and the number of iterations increase.

[0139] 3d) Layer-by-layer convolutional distillation calculation. To further remove redundant data from weakly correlated frames, the self-attention results of each layer are processed. Convolutional distillation is performed to adjust the feature weights between samples in longer sequences, improving the model's performance in the context of this invention. When the computation reaches layer l, the results of layer l-1 are processed. After performing self-attention computation, the vector passes through a convolutional layer with a length half that of the current vector, and then undergoes ELU function and pooling operations to obtain the result of the l-th layer self-attention computation.

[0140]

[0141] in,[·] AB Refers to the self-attention computation process.

[0142] In one specific embodiment of the present invention, the camera behavior trend encoding module employs two self-attention layers, each containing a self-attention calculation process and a convolutional distillation operation. For the encoded computational data... After this module performs the calculations, the implicit variables are output.

[0143] Figure 6 This is a schematic diagram of a camera behavior generation decoding module according to an embodiment of the present invention.

[0144] Combined Figure 1 , Figure 2 , Figure 6 As shown in step S140, based on the pre-constructed camera behavior generation and decoding module based on global performance scene state features, the global performance scene state features are used as input as strong control feature data for camera behavior generation. This is then fused with the global camera trend encoding intermediate vector obtained in step 3), and a self-attention calculation method is used to obtain the predicted camera behavior sequence. Specifically, as an example, this process includes:

[0145] 4a) Construct global performance scene state feature data, using the arrival information before and after keyframes and actor state features from the global performance scene state features as feature data accompanying the input of the frame to be predicted. Camera information of the frame to be predicted is used as a 5-dimensional placeholder zero vector. The above-mentioned feature data are concatenated to form the frame to be predicted in the data:

[0146]

[0147] In one specific embodiment of the present invention, the camera behavior generation decoder module predicts the 5D complex surface coordinate camera behavior vector of the corresponding frame and fills it with a 5D placeholder zero vector.

[0148] Simultaneously, the frame to be predicted is concatenated with the data of the known camera behavior in the 50 frames preceding the corresponding frame position to form the prior token data in the data. Ensure a smooth match between generated camera behavior and historical camera behavior:

[0149]

[0150] In one specific embodiment of the present invention, the input size of the camera behavior generation decoder module is 200 frames, of which the prior token data part is 50 frames and the frame to be predicted part is 150 frames:

[0151]

[0152] The stitched camera behavior generates the input data D for the decoding module:

[0153]

[0154] Among them, Concat t (·) indicates that the data is spliced ​​in the time (frame) dimension.

[0155] 4b) Input data encoding: For the module input data D, the same data transformation encoding and position encoding as in step 3c) are used to obtain Q for self-attention calculation. d ,K d V d The specific calculation details will not be elaborated in this step.

[0156] 4c) Cross-self-attention calculation, based on the position-encoded result obtained in step 4b). Calculate single-layer global self-attention

[0157] In one specific embodiment of the present invention, the camera behavior generation decoding module does not add a mask for the frames following the current prediction frame to the data, so as to ensure that the self-attention calculation process can take into account the correlation between the current frame and the subsequent key frames.

[0158] Self-attention Perform cross-self-attention calculation with the latent variable H calculated in step 3d):

[0159]

[0160] Results After processing by the dropout and normalization layers, and then by the linear layer, it becomes... The vector is used to obtain the actor information and camera behavior information of the frame to be predicted after decoding:

[0161]

[0162] in, This refers to the predicted coordinates of the complex surface camera behavior in the current frame.

[0163] 4d) Post-processing of data: Since the complex surface coordinate system represents the position and angle relationship between the camera and the current actor, obtaining only the camera behavior in the complex surface coordinate system cannot directly drive the virtual camera in the Unity program. Therefore, the obtained frame-by-frame complex surface coordinate camera behavior vector is connected to the actor's head coordinates in three dimensions:

[0164]

[0165] data i Write it into the corresponding text document for the Unity program to read and use.

[0166] Finally, the camera behavior trajectory of the predicted camera behavior sequence can be obtained through the Unity editor's performance scene visualization interface.

[0167] Based on the above embodiments, the virtual camera behavior generation method of the present invention is implemented based on the virtual camera behavior generation model of global multi-dimensional data fusion features of virtual performance.

[0168] In one specific embodiment of the present invention, the virtual camera behavior generation method further includes the step of constructing a virtual camera behavior generation model based on the global multi-dimensional data fusion features of virtual performance.

[0169] In this embodiment, global camera behavior trend features and global performance scene state features are used as global multi-dimensional data fusion features of the performance scene. These features are combined with a global camera behavior encoding module and a camera behavior generation and decoding module, and fused with a loss function based on multi-dimensional data of the performance scene, to construct a virtual camera behavior generation model based on the global multi-dimensional data fusion features of the virtual performance. Specifically, as an example, this process includes:

[0170] 5a) The training method of this invention is end-to-end model training, combining the reference video camera style extraction network in step 2c), the camera behavior trend encoding module in step S130, and the camera behavior generation decoding module in step S140 into a whole, constructing the virtual camera behavior generation model based on global multi-dimensional data fusion features of virtual performance according to this invention. The model generates and predicts the five-dimensional camera behavior in the complex surface coordinate system.

[0171] 5b) Reconstruction frame loss.

[0172] Supervise all frames predicted on the training set to ensure the overall network performance in predicting camera behavior, and calculate the loss function L for predicting camera behavior for all n frames. rec :

[0173]

[0174] 5c) Keyframe loss.

[0175] For camera behavior generation based on keyframe control, the fitting performance at keyframes is a crucial supervision parameter. For the keyframe set K during training, the camera behavior loss function L is calculated for all keyframes. K :

[0176]

[0177] 5d) Actor's performance loss.

[0178] In one specific embodiment of the present invention, to increase the weight of actor behavior and keyframe information in the camera behavior generation decoder module's computation process, frame-by-frame actor information is input into this module and then output. To ensure network accuracy, an actor information loss L is added. Ch :

[0179]

[0180] 5e) Add the reconstructed frame loss obtained in step 5b), the keyframe loss obtained in step 5c), and the actor state loss obtained in step 5d) to obtain the total loss L, and optimize the camera behavior computation generation model end-to-end:

[0181]

[0182] Finally, according to Figure 2 The framework shown inputs the fusion features of global multi-dimensional data from virtual performances into the camera behavior trend encoding module and the camera behavior generation decoding module to predict camera behavior sequences. These sequences are then passed to a Unity editor-based performance scene visualization interface to display the generated camera behavior trajectories. Specifically, as an example, the process includes:

[0183] 6a) Operate in the Unity editor's performance scene visualization interface and output the required data features.

[0184] 6b) For each reference video in the test set, extract the reference camera movement style features according to step S120, and combine them with historical camera motion data to form global camera behavior trend features.

[0185] 6c) Using a pre-trained camera behavior generation model based on global multi-dimensional data fusion features of virtual performances, input the global multi-dimensional data fusion features to generate a predicted camera trajectory file.

[0186] 6d) Send the camera trajectory file into the Unity program to drive the virtual scene camera movement.

[0187] The above method will be applied to the following embodiments to demonstrate the technical effects of the present invention. The specific steps in the embodiments will not be repeated.

[0188] This embodiment uses a segment dataset categorized by camera movement style and shooting strategy, as provided in existing work. This dataset includes 30 virtual scene data points with preset camera trajectories provided by the authors, as well as some real film data, which comprises approximately 10% of the entire dataset. It includes four camera movement styles:

[0189] Direct trajectory: The camera is primarily in front of one of the characters, and the camera's angle relative to the character remains stationary.

[0190] Relative trajectory: Similar to a direct trajectory, while maintaining a relatively stationary position relative to the two characters, with the camera positioned so that both characters can be captured simultaneously.

[0191] Side trajectory: The two characters are mainly in a walking state, and the camera shooting angle line is mainly perpendicular to the walking direction.

[0192] Track trajectory: Commonly used in long-shot dialogue scenes, the camera movement trajectory mainly interweaves between the two characters.

[0193] In the dataset, each camera movement style and shooting strategy consists of multiple video clips in frames. Each frame is a stitched image of the actor's state in a complex surface coordinate system and the camera's state in a complex surface coordinate system.

[0194]

[0195] In this embodiment, a fixed-step downsampling operation is used for each sequence segment. Since the sequence frame rate of the integrated dataset is high and the data change of each frame is small, the downsampling results reduce the amount of data while ensuring the continuity and availability of frame-by-frame data.

[0196] To intuitively characterize the performance of this method on existing virtual camera movement datasets, the mean absolute error of the generated camera behavior data is used as the evaluation metric, and the absolute error values ​​at keyframes are statistically analyzed. Following the steps described in the specific implementation, the experimental results are shown in Table 1 below.

[0197] Method Name <![CDATA[Reconstruction Frame Average Error (×10 -2 )]]> <![CDATA[Average error of key frames (×10 -4 )]]> This invention 1.548 0.023 Existing methods 3.211 0.053

[0198] Table 1 Comparison of results between the present invention and existing methods on existing virtual performance camera movement datasets.

[0199] As shown in Table 1 above, the present invention achieves higher performance on existing datasets. It can be seen that the mechanism based on global multi-dimensional data fusion features is more suitable for generating camera behavior sequences with data-driven factors and requiring attention to global data change relationships. It can significantly improve the quality of generated camera behavior sequences and the prediction accuracy of known camera behavior sequences.

[0200] To more clearly demonstrate the performance and usability of the virtual camera behavior generation method based on visualized keyframe control and global self-attention distillation, this invention uses a self-built Unity virtual scene as the test scene. The camera behavior trajectory is visualized in the scene using the steps described in this invention, and the results are as follows: Figure 8 , Figure 9 As shown.

[0201] like Figure 8 As shown, different style reference sequences are input, and the camera-generated trajectories of different styles, driven by different reference sequences, all have a style presentation effect. The upper image segment is a close-up shot of two people talking, slowly zooming in, while the lower image is an alternating close-up and wide shot of a dance segment.

[0202] like Figure 9As shown, for running instances with the same style reference sequence paired with different keyframes, this invention generates different camera trajectories based on the different keyframe information. For the same style reference sequence, for different keyframe specified sets, in order to ensure overall smoothness during model generation, the trajectory fitting driven by the same keyframe information also differs.

[0203] Depend on Figure 8 , Figure 9 It can be shown that the technical solution of the present invention can achieve the generation of the required camera trajectory in the simultaneous control of style data and keyframes.

[0204] The virtual camera behavior generation method according to an embodiment of the present invention has been described above. This virtual camera behavior generation method involves: 1) operating the performance scene visualization interface in the Unity editor to output global performance scene state features and keyframe camera state features; 2) constructing a global camera behavior trend feature extraction module based on reference video; 3) constructing a camera behavior trend encoding module based on the global camera behavior trend features; 4) constructing a camera behavior generation decoding module based on the global performance scene state features; 5) using the global camera behavior trend features and global performance scene state features as global multi-dimensional data fusion features of the performance scene, combining them with the global camera behavior encoding module and the camera behavior generation decoding module, and fusing them with a multi-dimensional data loss function based on the performance scene, to construct a camera behavior generation model based on the global multi-dimensional data fusion features of the virtual performance scene; 6) inputting the global multi-dimensional data fusion features into the global camera behavior encoding module and the camera behavior generation decoding module to predict the camera behavior sequence, and then inputting it into the performance scene visualization interface based on the Unity editor to display the generated camera behavior trajectory. As can be seen, this invention, through self-attention distillation computation, fully utilizes the global correlation of camera behavior trends and their strong correlation with the global state of the performance scene to predict the camera behavior trajectory of the performance scene. Compared with the prior art, it has the advantage of high accuracy in camera behavior prediction.

[0205] The virtual camera behavior generation method of the present invention can be implemented in software, in hardware, or in a combination of software and hardware.

[0206] Corresponding to the above-described virtual camera behavior generation method, the present invention also provides a virtual camera behavior generation system. Figure 7 A block diagram of a virtual camera behavior generation system 700 according to an embodiment of the present invention is shown. Figure 7 As shown, the virtual camera behavior generation system 700 includes an entertainment scene feature acquisition unit 710, a global camera behavior trend feature acquisition unit 730, a fusion feature acquisition unit 720, a camera behavior trend encoding unit 740, a camera behavior generation decoding unit 750, and a camera behavior trajectory generation unit 760.

[0207] Among them, the performance scene feature acquisition unit 710 is used to acquire global performance scene state features and key frame camera state features through the Unity editor's performance scene visualization interaction interface.

[0208] Global camera behavior trend feature 720 is used to obtain global camera behavior trend features, which include historical camera motion features, key frame camera state features, and reference camera movement style features extracted from a preset reference video by a preset global camera behavior trend feature extraction module.

[0209] The fusion feature acquisition unit 730 is used to acquire virtual performance global multi-dimensional data fusion features based on the global performance scene state features and key frame camera state features; wherein, the virtual performance global multi-dimensional data fusion features include global camera behavior trend features and global performance scene state features;

[0210] The camera behavior trend encoding unit 740 is used to encode the intermediate vector of the global camera behavior trend based on the preset camera behavior trend encoding module, taking the global camera behavior trend features as input, and introducing the motion trend vector and arrival time signal related to the key frame information, and using self-attention and layer-by-layer convolutional distillation calculation methods to obtain the global camera trend encoding.

[0211] The camera behavior generation and decoding unit 750 is used to obtain the predicted camera behavior sequence based on the preset camera behavior generation and decoding module, taking the global performance scene state features as input, fusing the intermediate vector of the global camera trend encoding, and using the cross self-attention calculation method.

[0212] The camera behavior trajectory generation unit 760 is used to obtain the camera behavior trajectory of the predicted camera behavior sequence through the Unity editor's performance scene visualization interaction interface.

[0213] In one specific embodiment of the present invention, the virtual camera behavior generation system 700 further includes a reference video camera behavior trend feature extraction unit (not shown in the figure), which is used to extract camera behavior trend features based on a preset reference video.

[0214] Specifically, as an example, the reference video camera behavior trend feature extraction unit includes a human pose estimation unit, a shooting feature estimation unit, and a camera style extraction unit.

[0215] The human pose estimation unit is used to extract key point information of the main characters from the input reference video. For each frame f in the input reference video... t The screen coordinates of seven joints (head, neck, left shoulder, right shoulder, hip, left hip, and right hip) were extracted using a DeKR network.

[0216]

[0217] in, This represents the screen coordinates of a certain joint point n, and DeKR(·) represents the processing procedure of the human pose estimation model.

[0218] Calculate the joint features of the actors in a frame using the coordinates of seven joint points of the two actors:

[0219]

[0220] Where AB is a straight line passing through the midpoint of the two actors' bodies, p A p B For the screen position information of the two actors' heads, n A n B The h represents the joint positions of the two actors' necks, left shoulders, and right shoulders, respectively. A h B Let AB be the angle between the orientation of the two actors' heads and the line AB.

[0221] The shooting feature estimation unit maps the coordinates representing the positions of people and the camera in the reference video to a specific coordinate space, employing the existing convolutional layer-based feature estimation network pre-trained model Esti(·). Based on an 8-frame sliding window, it maps the frame-by-frame actor screen joint coordinate system obtained in the human pose estimation unit, outputting relevant parameters:

[0222]

[0223] y t =Esti(u t-4 ,u t-3 ,u t-2 ,u t-1 ,u t ,u t+1 ,u t+2 ,u t+3 )

[0224] Where y represents the output data of the feature estimation network, and Φ and θ represent the yaw angles of the camera relative to the actor in the vertical and horizontal directions, respectively; s A s B Let s represent the angle between the normal vector of the line connecting the shoulders of the two actors and the angle between line AB; AB This represents the angle between the normal vectors of the lines connecting the shoulders of the two actors; u t-4 This represents the keypoint features from the previous four frames, and similarly, the others represent the data within the input time window of the feature estimation network.

[0225] Based on the y-value of the current reference video framet And computable screen position information of the actor's head p A p B This allows you to directly obtain the actor-camera parameter pair in the complex surface coordinate system for that frame in the reference video:

[0226]

[0227] in, and Represent the actor's state and the camera's state in the complex surface coordinate system in frame t, respectively:

[0228]

[0229] The camera style extraction unit is used to process the reference video actor-camera parameter pairs obtained in the shooting feature estimation unit using a long short-term memory network, mapping them into a low-dimensional vector of implicit camera movement rules as a reference camera movement style:

[0230] z c =(FC(LSTM(X) clip ))) 120

[0231] Where FC(·) and LSTM(·) represent the Long Short-Term Memory network and the fully connected network in the style feature extraction process, respectively, X clip This represents a data segment from the input data in the style feature extraction process, represented by x in the above. t The style vector z is composed of 120 frames, with the output of the last frame of the network used as the result. c .

[0232] Specifically, as an example, in one specific embodiment of the present invention, 120 frames are randomly selected from the input sequence. As a reference style sequence, the network's initial hidden states are set to all zeros. The output style code is mapped to a 4-dimensional vector, and the style code z of the last frame's output is taken as the latent representation of the camera movement rules. c .

[0233] As an example, in another specific embodiment of the present invention, the fusion feature acquisition unit 730 further includes a global camera behavior trend feature acquisition unit and a global performance scene state feature acquisition unit (not shown in the figure).

[0234] Among them, the global camera behavior trend feature acquisition unit is used to perform skeleton recognition, coordinate extraction and style feature mapping on the input reference video sequence to obtain reference camera movement style features, which together with the historical camera motion features and scene key frame camera state features during the performance constitute the global camera behavior trend features.

[0235] The global performance scene state feature acquisition unit is used to output scene keyframe information and actor information through the performance scene visualization interactive interface to obtain the global performance scene state features.

[0236] For more specific implementations of the virtual camera behavior generation system described above, please refer to the specific embodiments of the human video compression method based on the generative model described above, which will not be described in detail here.

[0237] The virtual camera behavior generation method and system according to the present invention have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various modifications can be made to the virtual camera behavior generation method and system proposed in the present invention without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the content of the appended claims.

Claims

1. A method for generating virtual camera behavior, based on the global multi-dimensional data fusion features of virtual performances, comprising: By using the Unity editor's performance scene visualization interface, the position information of key frames and the camera status information of key frames in the performance scene are specified. The key frame status features and the arrival information before and after key frames in the performance process are constructed, and the actor status features are extracted to obtain the global performance scene status features and key frame camera status features. The arrival information before and after key frames includes the arrival distance information of the previous key frame and the smooth departure information of the subsequent key frame. Based on the global performance scene state features and keyframe camera state features, virtual performance global multi-dimensional data fusion features are obtained; wherein, the virtual performance global multi-dimensional data fusion features include global camera behavior trend features and global performance scene state features, the global camera behavior trend features include historical camera motion features, the keyframe camera state features and reference camera movement style features extracted from a preset reference video by a preset global camera behavior trend feature extraction module, and the global performance scene state features include keyframe arrival information state features, actor state features and camera zero vector data of the frame to be predicted; Based on the preset camera behavior trend encoding module, the global camera behavior trend features are used as input, and motion trend vectors and arrival time signals related to key frame information are introduced. The self-attention and layer-by-layer convolutional distillation calculation method of the Informer model are used to encode the intermediate vector of the global camera trend encoding. Based on the preset camera behavior generation and decoding module, the global performance scene state features are used as input, and the intermediate vector of the global camera trend encoding is fused. The cross self-attention calculation method is used to obtain the predicted camera behavior sequence. The camera behavior trajectory of the predicted camera behavior sequence is obtained through the Unity editor's performance scene visualization interface.

2. The virtual camera behavior generation method as described in claim 1, wherein, The global camera behavior trend feature extraction module extracts reference camera movement style features from a preset reference video, including: The global camera behavior trend feature extraction module performs skeleton recognition, coordinate extraction, and style feature mapping on the video sequence of the preset reference video to obtain the reference camera movement style features.

3. The virtual camera behavior generation method as described in claim 2, wherein, The process of performing skeletal recognition, coordinate extraction, and style feature mapping on a video sequence of a preset reference video to obtain the reference camera movement style features includes: For each frame in the reference video, the screen coordinates of a preset number of key points are extracted using a DeKR network; Calculate the joint features of the actors in a frame by using the joint coordinates of the two actors in that frame. Based on a pre-trained model of a feature estimation network using convolutional layers and a preset number of frame sliding windows, the coordinate system of the actor's screen joint points is mapped frame by frame, and relevant parameters are output. Based on the relevant parameters and the screen position information of the actor's head, the actor-camera parameter pair in the complex surface coordinate system of the frame in the reference video is obtained; The actor-camera parameter pairs are processed using a long short-term memory network and mapped into a low-dimensional vector containing implicit camera movement rules, which serves as a reference camera movement style.

4. The virtual camera behavior generation method as described in claim 3, wherein, The preset camera behavior trend encoding module takes the global camera behavior trend features as input and introduces motion trend vectors and arrival time signals related to keyframe information, including: Determine the input frame data for the camera behavior trend encoding module corresponding to the next camera behavior to be tested; The input data of the camera behavior trend encoding module is stitched together using a preset time window; The input data of the stitched camera behavior trend encoding module is embedded and encoded using the Transformer model; at the same time, the input data is positionally encoded to ensure that each frame carries order information during the self-attention calculation process.

5. The virtual camera behavior generation method as described in claim 4, wherein, The method of using the self-attention and layer-by-layer convolutional distillation of the Informer model to encode the intermediate vector of the global camera trend encoding includes: The global self-attention mechanism after computational position encoding; Convolutional distillation is performed on the self-attention results of each layer to adjust the feature weights between samples in a longer sequence.

6. The virtual camera behavior generation method as described in claim 5, wherein, The camera behavior trend encoding module employs two self-attention layers, each containing a self-attention calculation process and a convolution distillation operation. For the encoded computational data, after processing by the camera behavior trend encoding module, a latent variable is output as the intermediate vector of the global camera trend encoding.

7. The virtual camera behavior generation method as described in claim 6, wherein, The preset camera behavior generation and decoding module, using the global performance scene state features as input, includes: The arrival information of keyframes before and after the global performance scene state features and the actor state features are used as feature data for the input of the frame to be predicted. At the same time, the frame to be predicted is spliced ​​with the data of the known camera behavior of the previous 50 frames at the corresponding frame position to become the prior data in the data; The input data of the stitched camera behavior generation decoding module is subjected to data transformation encoding and position encoding.

8. The virtual camera behavior generation method as described in claim 6, wherein, The camera behavior generation and decoding module predicts the 5D complex surface coordinate camera behavior vector of the corresponding frame and fills it with a 5D placeholder zero vector.

9. The virtual camera behavior generation method as described in claim 8, wherein, The intermediate vector fused with the global camera trend encoding is used to obtain the predicted camera behavior sequence using a cross-self-attention calculation method, including: The single-layer global self-attention is calculated based on the data obtained after performing data transformation encoding and position encoding on the input data of the stitched camera behavior generation decoding module. Cross-self-attention calculation is performed based on the single-layer global self-attention and the latent variable H; The cross-attention calculation results are processed through dropout and norm layers, and then through a linear layer to obtain actor information and camera behavior information of the frame to be predicted after decoding.

10. A virtual camera behavior generation system, based on the global multi-dimensional data fusion features of virtual performances, comprising: The performance scene feature acquisition unit is used to construct key frame state features and key frame arrival information of the performance scene by specifying key frame position information and key frame camera state information through the performance scene visualization and interactive interface of the Unity editor, and extracting actor state features to obtain global performance scene state features and key frame camera state features; wherein, the key frame arrival information includes the arrival distance information of the previous key frame and the smooth departure information of the subsequent key frame. The fusion feature acquisition unit is used to acquire virtual performance global multi-dimensional data fusion features based on the global performance scene state features and keyframe camera state features; wherein, the virtual performance global multi-dimensional data fusion features include global camera behavior trend features and global performance scene state features, the global camera behavior trend features include historical camera motion features, the keyframe camera state features and reference camera movement style features extracted from a preset reference video by a preset global camera behavior trend feature extraction module, and the global performance scene state features include keyframe arrival information state features, actor state features and camera zero vector data of the frame to be predicted; The camera behavior trend encoding unit is used to encode the intermediate vector of the global camera behavior trend based on the preset camera behavior trend encoding module, taking the global camera behavior trend features as input, and introducing the motion trend vector and arrival time signal related to the key frame information. It uses the self-attention and layer-by-layer convolutional distillation calculation method of the Informer model to encode the global camera trend encoding. The camera behavior generation and decoding unit is used to obtain the predicted camera behavior sequence based on the preset camera behavior generation and decoding module, taking the global performance scene state features as input, fusing the intermediate vector of the global camera trend encoding, and using the cross self-attention calculation method. The camera behavior trajectory generation unit is used to obtain the camera behavior trajectory of the predicted camera behavior sequence through the Unity editor's performance scene visualization and interactive interface.

Images