Dance scene camera operation generation method and device based on key frames

Abstract

The application relates to the technical field of dance photography, and provides a dance scene camera motion generation method and device based on key frames, wherein the method comprises the following steps: determining camera key frames on a time axis according to given music and dance; generating key frame camera postures corresponding to the camera key frames according to the given music and dance and a camera motion history record; predicting the in-between function value corresponding to non-key frames between two adjacent camera key frames according to the given music and dance, the camera motion history record and the key frame camera postures; and generating target camera motion according to the in-between function value and the key frame camera postures. The method can realize smoother camera curves and more stable dance scene camera shooting, thereby significantly improving the watching experience of users for the dance. The method also has key frame time controllability and key frame posture controllability, so that more flexible and more delicate use experiences can be provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dance photography, and particularly relates to a dance scene camera movement generation method and device based on key frames. BACKGROUND

[0002] Photography plays a crucial role in dance performances, as it greatly influences the audience's perception and understanding of the dance work. Through multiple camera switching, the producer can capture the subtle movements and facial expressions of the dancers to reveal more dance details. In addition, the combination of creative photography techniques such as camera position switching and slow motion, as well as rich camera movement techniques such as "push, pull, shake, and move", can bring visual impact and novelty to the audience, increasing the appeal of dance performances. Therefore, automatically generating camera movements from music and dance is an attractive but challenging task.

[0003] Among them, the camera movement is composed of several continuous sequences of variable length, and the continuous part and the spliced part represent the camera lens and the switching of the camera lens, respectively.

[0004] In the prior art, DanceCamera3D constructed the first 3D dance-camera movement-music dataset DCM, and demonstrated the rationality of music-dance-driven camera movement synthesis. However, it treats all frames equally and ignores sudden changes between camera shots, which greatly affects the model's generation ability, as the model cannot determine whether to generate continuous or abrupt motion, resulting in severe shaking of the final dance video.

[0005] In this case, DanceCamera3D must perform smoothing post-processing by detecting key frames and filtering frames between them using a total variation denoiser in the camera parameters. However, this post-smoothing may misjudge some key frames and introduce some false smoothing, causing the camera to lose focus on the dancers.

[0006] Therefore, it is necessary to solve the problem that the existing camera movement generation method treats all frames equally and ignores the abrupt motion caused by camera position switching, resulting in severe shaking of the captured dance video and poor camera effect. SUMMARY

[0007] The present application provides a dance scene camera movement generation method and device based on key frames, which overcomes the defects of the existing camera movement generation method that treats all frames equally and ignores the abrupt motion caused by camera position switching, resulting in severe shaking of the captured dance video and poor camera effect, and realizes smoother camera curves, more explicit lens switching, and more stable dance scene camera shooting, significantly improving the user's viewing experience of the dance.

[0008] In one aspect, the present application provides a method for generating dance scene camera motion based on key frames, comprising: determining camera key frames on a time axis according to given music and dance; generating key frame camera poses corresponding to the camera key frames according to the given music and dance, and camera motion history records; predicting the values of in-between functions corresponding to non-key frames between two adjacent camera key frames according to the given music and dance, and the camera motion history records and the key frame camera poses; and generating target camera motion according to the values of the in-between functions and the key frame camera poses.

[0009] Further, the determining of the camera key frames on the time axis according to the given music and dance comprises: obtaining key frame history records; and determining the camera key frames on the time axis according to the given music and dance, and the key frame history records.

[0010] Further, the determining of the camera key frames on the time axis according to the given music and dance, and the key frame history records comprises: encoding a music-dance feature vector according to the given music and dance; obtaining a history key frame feature vector corresponding to the key frame history records; predicting a key frame probability sequence according to the music-dance feature vector and the history key frame feature vector; and determining a key frame corresponding to the key frame probability as a camera key frame when the key frame probability is higher than a set probability threshold.

[0011] Further, the determining of the camera key frames on the time axis according to the given music and dance, and the key frame history records comprises: encoding a music-dance feature vector according to the given music and dance; obtaining a history key frame feature vector corresponding to the key frame history records; predicting a key frame probability sequence according to the music-dance feature vector and the history key frame feature vector; and determining a key frame corresponding to the key frame probability as a camera key frame when the key frame probability is higher than a set probability threshold.

[0012] Further, the generating of the key frame camera poses corresponding to the camera key frames according to the given music and dance, and the camera motion history records comprises: encoding a music-dance feature vector according to the given music and dance; obtaining a history camera motion feature vector corresponding to the camera motion history records; and generating the key frame camera poses corresponding to the camera key frames according to the music-dance feature vector and the history key frame feature vector.

[0013] Further, the predicting of the values of the in-between functions corresponding to the non-key frames between two adjacent camera key frames according to the given music and dance, and the camera motion history records and the key frame camera poses comprises: encoding a music-dance feature vector according to the given music and dance; encoding a history key frame camera motion feature vector according to the camera motion history records and the key frame camera poses; and predicting the values of the in-between functions corresponding to the non-key frames between two adjacent camera key frames according to the music-dance feature vector and the history key frame camera motion feature vector.

[0014] Further, the predicting the in-between function value corresponding to the in-between frame between two adjacent camera key frames according to the music-dance feature vector and the historical key frame camera motion feature vector comprises: decoding an intermediate variable according to the music-dance feature vector and the historical key frame camera motion feature vector; performing mask processing on the intermediate variable to retain variables corresponding to the two adjacent camera key frames to obtain a masked variable; performing non-negative processing on the masked variable to obtain an in-between function increment; performing previous term summation processing on the in-between function increment and performing normalization processing to obtain the in-between function value.

[0015] Further, the generating the target camera motion according to the in-between function value and the camera key frame pose comprises: generating an in-between frame camera pose corresponding to the in-between frame according to the in-between function value and the camera key frame pose; and generating the target camera motion according to the in-between frame camera pose and the key frame camera pose.

[0016] In a second aspect, the present application provides a key frame based dance scene camera motion generation device, comprising: a camera key frame detection module configured to determine camera key frames on a time axis according to given music and dance; a key frame camera pose acquisition module configured to generate key frame camera poses corresponding to the camera key frames according to the given music and dance and a camera motion history record; an in-between function value prediction module configured to predict in-between function values corresponding to in-between frames between two adjacent camera key frames according to the given music and dance, the camera motion history record and the key frame camera poses; and a target camera motion generation module configured to generate a target camera motion according to the in-between function values and the key frame camera poses.

[0017] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the key frame based dance scene camera motion generation method according to any one of the above aspects.

[0018] In a fourth aspect, the present application provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the key frame based dance scene camera motion generation method according to any one of the above aspects.

[0019] The application provides a key frame-based dance scene camera motion generation method, which comprises the following steps: determining camera key frames on a time axis according to given music and dance; generating key frame camera poses corresponding to the camera key frames according to the given music and dance and a camera motion history record; and predicting the interpolation function values corresponding to non-key frames between two adjacent camera key frames according to the given music and dance, the camera motion history record and the key frame camera poses, so as to generate target camera motion according to the interpolation function values and the key frame camera poses. The method can realize smoother camera curves, more explicit lens switching and more stable dance scene camera shooting by designing a three-stage dance scene camera motion generation framework of key frame detection-key frame camera pose generation-interpolation function value prediction, and can significantly improve the viewing experience of users for dance, while the method also has key frame time controllability and key frame pose controllability, so as to provide more flexible and fine use experience. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is an effect diagram of the camera motion generation method provided by the prior art.

[0022] Figure 2 is a flow diagram of the key frame-based dance scene camera motion generation method provided by the embodiment of the application.

[0023] Figure 3 is a camera key frame detection diagram of the key frame-based dance scene camera motion generation method provided by the embodiment of the application.

[0024] Figure 4 is a key frame camera pose generation diagram of the key frame-based dance scene camera motion generation method provided by the embodiment of the application.

[0025] Figure 5 is an interpolation function prediction diagram of the key frame-based dance scene camera motion generation method provided by the embodiment of the application.

[0026] Figure 6 is a whole flow diagram of the key frame-based dance scene camera motion generation method provided by the embodiment of the application.

[0027] Figure 7 is the effect comparison schematic diagram of the key frame based dance scene camera motion generation method provided by the embodiment of the present application and real data and prior art.

[0028] Figure 8 is the structure schematic diagram of the key frame based dance scene camera motion generation device provided by the embodiment of the present application.

[0029] Figure 9 is the structure schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0031] It should be noted that, due to the requirements and complexity of dance photography, it is extremely challenging to synthesize camera motion from music and dance.

[0032] Unlike always continuous human motion, dance camera motion involves both variable length continuous sequences (smooth continuous shots) and sudden drastic changes (abrupt shot changes) to simulate multiple camera cuts.

[0033] However, in the prior art, each camera frame is processed equally, which can cause jitter and non-smoothness in post-processing, which can be seen in detail from Figure 1 , Figure 1 shows the effect schematic diagram of the camera motion generation method provided by the prior art.

[0034] In view of this, the present application provides a key frame based dance scene camera motion generation method, specifically, Figure 2 shows the flow schematic diagram of the key frame based dance scene camera motion generation method provided by the embodiment of the present application.

[0035] As Figure 2As shown, the method comprises: S210, determining camera keyframes on a time axis according to given music and dance; S220, generating keyframe camera poses corresponding to the camera keyframes according to the given music and dance and a camera motion history record; S230, predicting values of an interpolation function corresponding to non-keyframes between two adjacent camera keyframes according to the given music and dance, the camera motion history record and the keyframe camera poses; and S240, generating target camera motion according to the values of the interpolation function and the keyframe camera poses.

[0036] It is worth mentioning that the keyframe-based dance scene camera motion generation method provided in the embodiment of the present application divides the task of synthesizing camera motion from music and dance into three stages, i.e., a keyframe detection stage, a keyframe camera pose generation stage and an interpolation function prediction stage, by combining the professional knowledge of an animator's dance camera operation production (an animator's hierarchical dance camera production process).

[0037] The above steps S210-S240 will be described in detail below with reference to the accompanying drawings.

[0038] S210, determining camera keyframes on a time axis according to given music and dance.

[0039] It should be noted that the keyframe-based dance scene camera motion generation method provided in the embodiment is mainly aimed at dance scenes, and the target is to generate camera motion from given music and dance.

[0040] It can be understood that when generating three-dimensional dance camera motion, music and dance are given in advance, and acoustic features / music features in the given music and dance poses / dance features in the given dance can be extracted by a feature extractor.

[0041] In a specific embodiment, the Librosa library can be used to extract music features in the given music, and the extracted music features include but are not limited to mel-frequency cepstral coefficients, pitch features, chroma features, etc.

[0042] The Librosa library is a Python library dedicated to audio and music signal analysis, which provides a series of functions including audio feature extraction, audio visualization, rhythm analysis, audio processing, etc.

[0043] In still another specific embodiment, the extracted dance features include but are not limited to dance poses of 60 joint positions.

[0044] Then, the extracted music features and dance features are encoded to obtain corresponding music feature vectors and dance feature vectors. Meanwhile, a key frame history record is obtained and encoded to obtain a history key frame feature vector.

[0045] According to the music feature vectors, the dance feature vectors, and the history key frame feature vector, a camera key frame on a time axis can be detected.

[0046] Generally, a camera key frame corresponds to a climax part in music and dance and is a position that needs to be focused on.

[0047] The music and dance can be given according to actual conditions and are not specifically limited herein.

[0048] S220, according to the given music and dance and the camera motion history record, a key frame camera pose corresponding to the camera key frame is generated.

[0049] It can be understood that, on the basis of step S210 of determining the camera key frame on the time axis according to the given music and dance, step S220 further generates the key frame camera pose at the camera key frame.

[0050] The embodiment does not equally treat all frames as in the prior art to generate camera motions corresponding to all frames, but generates corresponding camera poses for key frames and non-key frames respectively to obtain target camera motions corresponding to all frames.

[0051] The ultimate purpose of step S220 is to generate a corresponding camera pose for a key frame, and the ultimate purpose of step S230 is to generate a corresponding camera pose for a non-key frame by predicting an interpolation function value.

[0052] In this step, for the given music and dance, first, music features and dance features are extracted and encoded to obtain music feature vectors and dance feature vectors, as in step S210, which will not be described in detail herein.

[0053] Meanwhile, a camera motion history record is obtained and encoded to obtain a history camera motion feature vector.

[0054] It should be noted that, when predicting an initial frame, zero can be used to fill the camera motion history record.

[0055] Then, according to the music feature vectors, the dance feature vectors, and the history camera motion feature vector, a key frame camera pose at the camera key frame can be generated.

[0056] S230, predicting the in-between frame corresponding interpolation function value between two adjacent keyframes according to the given music and dance, and the camera motion history record and the keyframe camera pose.

[0057] It can be understood that, on the basis of step S220 of generating the keyframe camera pose corresponding to the keyframe of the camera according to the given music and dance, and the obtained camera motion history record, further, step S230 will predict the in-between frame corresponding interpolation function value between two adjacent keyframes.

[0058] For the given music and dance, first, as in step S210, the music features and dance features are extracted, and the music feature vector and dance feature vector are encoded, which will not be described in detail here.

[0059] For the obtained camera motion history record and the generated keyframe camera pose, the blank frame is filled with zeros, and then the history keyframe camera motion feature vector is encoded.

[0060] Then, according to the music feature vector, the dance feature vector and the history keyframe camera motion feature vector, the in-between frame corresponding interpolation function value between any two adjacent keyframes can be predicted.

[0061] It is worth mentioning that, in this embodiment, the in-between frame corresponding interpolation function value between two adjacent keyframes is predicted, and then the camera motion (composed of a plurality of continuous in-between frame corresponding camera poses) between two adjacent keyframes is generated, instead of directly generating the camera pose of the in-between frame between the keyframes as in the prior art, without human intervention and post-smoothing processing, effectively reducing the jitter in the shot dance video.

[0062] The interpolation function value is used to control the moving speed of the camera from one keyframe to the next adjacent keyframe.

[0063] S240, generating target camera motion according to the interpolation function value and the keyframe camera pose.

[0064] It can be understood that, on the basis of step S230 of predicting the in-between frame corresponding interpolation function value between any two adjacent keyframes, further, step S240 will generate the final camera motion corresponding to all frames, that is, the target camera motion.

[0065] Specifically, according to the predicted interpolation function value corresponding to the non-key frame between two adjacent camera key frames, the camera motion (composed of the camera poses corresponding to a plurality of continuous non-key frames) corresponding to the non-key frame between the two adjacent camera key frames can be further generated, and then the camera motion corresponding to all frames (i.e., the target camera motion) can be obtained by combining the generated key frame camera pose.

[0066] In the embodiment, the camera key frames on the time axis are determined according to the given music and dance, the key frame camera poses corresponding to the camera key frames are generated according to the given music and dance and the camera motion history record, and the interpolation function values corresponding to the non-key frames between two adjacent camera key frames are predicted according to the given music and dance, the camera motion history record and the key frame camera poses. Thus, the target camera motion can be generated according to the interpolation function values and the key frame camera poses. The method designs a three-stage end-to-end dance scene camera motion generation framework of key frame detection-key frame camera pose generation-interpolation function value prediction, predicts the interpolation function values between key frames instead of directly predicting the camera motion, can realize smoother camera curves, more explicit shot switching and more stable dance scene camera shooting, and thus significantly improves the viewing experience of the user for the dance. Meanwhile, the method also has key frame time controllability and key frame pose controllability, and thus can provide more flexible and fine use experience.

[0067] On the basis of the above embodiment, the detection process of the camera key frame will be described in detail below.

[0068] According to the given music and dance, the camera key frames on the time axis are determined, including: obtaining a key frame history record; and determining the camera key frames on the time axis according to the given music and dance and the key frame history record.

[0069] According to the given music and dance and the key frame history record, the camera key frames on the time axis are determined, including: encoding the music-dance feature vector according to the given music and dance; obtaining a history key frame feature vector corresponding to the key frame history record; predicting a key frame probability sequence according to the music-dance feature vector and the history key frame feature vector; and determining a key frame corresponding to the key frame probability as the camera key frame in a case where the key frame probability is higher than a set probability threshold.

[0070] It can be understood that, in the embodiment, the ultimate goal is to determine the camera key frames on the time axis according to the given music and dance.

[0071] For the given music and dance, first, the acoustic feature / music feature needs to be extracted from the given music and, extract dance features / dance poses in the given dance .

[0072] Then, select the music context and dance context using the sliding window, specifically, the music context as and the dance context as wherein, denotes the current frame time, denotes the length of the reference history record, denotes the length of the sliding window.

[0073] Next, encode the music context where the music features are located using an encoder to obtain a music feature vector and encode the dance context where the dance features are located to obtain a dance feature vector .

[0074] At the same time, obtain the keyframe history record When predicting the initial frame, the keyframe history record can be padded with zeros to obtain a keyframe context wherein, .

[0075] Next, encode the keyframe context using an encoder to obtain a history keyframe feature vector .

[0076] Then, concatenate the music feature vector and the dance feature vector to obtain a music-dance feature vector, which can be specifically seen in the following formula (1).

[0077] (1).

[0078] Further, according to the music-dance feature vector and the history keyframe feature vector, a transformer decoder and a linear layer are used to obtain a probability sequence of keyframes, i.e., a keyframe probability sequence, which can be specifically seen in the following formula (2).

[0079] (2).

[0080] Thereafter, whether there is a camera keyframe at time can be predicted by comparing the probabilities, which can be specifically seen in the following formula (3).

[0081] (3).

[0082] wherein, denotes the keyframe label at time .

[0083] The meaning of formula (3) is that the key frame probability in the key frame probability sequence is compared with the set probability threshold, and only when the compared key frame probability is higher than the set probability threshold, the key frame corresponding to the compared key frame probability is determined as the camera key frame. Otherwise, the key frame corresponding to the compared key frame probability is not the camera key frame. The value is 0.

[0084] The set probability threshold can be set according to actual needs, and is not specifically limited here.

[0085] According to the above, the camera key frame on the time axis can be determined according to the given music and dance.

[0086] In addition, Figure 3 A camera key frame detection schematic diagram of the camera key frame generation method based on key frames provided by the embodiment of the application is shown.

[0087] As Figure 3 shown, the process of camera key frame detection can be realized by a pre-trained camera key frame detection model.

[0088] Specifically, the camera key frame detection model includes a music encoder, a dance encoder, a key frame encoder, a Transformer decoder and a linear layer.

[0089] The input of the camera key frame detection model is the music features (music context in the above) extracted from the given dance, the dance features (dance context in the above) and the key frame context extracted from the key frame history record.

[0090] The output of the camera key frame detection model is the camera key frame on the time axis.

[0091] In actual reasoning, the music encoder is used to encode the music features to obtain a music feature vector, the dance encoder is used to encode the dance features to obtain a dance feature vector, and the key frame encoder is used to encode the key frame context to obtain a historical key frame feature vector.

[0092] Then, the output of the music encoder and the output of the dance encoder are spliced to obtain a music-dance feature vector.

[0093] Next, the music-dance feature vector and the historical key frame feature vector are input into the Transformer decoder together, and after passing through a linear layer and a comparison probability processing, the final output, i.e., the camera key frame, can be obtained.

[0094] It should be noted that the camera key frame detection model is pre-trained, and the model can be optimized by using a weighted binary cross-entropy loss function during training. The weighted binary cross-entropy loss function can be referred to as formula (4).

[0095] (4).

[0096] wherein, represents the real key frame label at frame represents the weight value corresponding to the key frame.

[0097] In this embodiment, by determining the camera key frames on the time axis according to the given music and dance, generating the key frame camera poses corresponding to the camera key frames according to the given music and dance and the camera motion history record, and predicting the interpolation function values corresponding to the non-key frames between two adjacent camera key frames according to the given music and dance, the camera motion history record and the key frame camera poses, the target camera motion is generated according to the interpolation function values and the key frame camera poses. This method designs a three-stage end-to-end dance scene camera motion generation framework of key frame detection-key frame camera pose generation-interpolation function value prediction to predict the interpolation function values between key frames instead of directly predicting the camera motion, which can realize smoother camera curves, more explicit shot switching and more stable dance scene camera shooting, and thus significantly improve the viewing experience of the user for the dance.

[0098] On the basis of the above embodiment, further, the generation of the key frame camera pose will be described in detail below.

[0099] According to the given music and dance, the music-dance feature vector is encoded, the historical camera motion feature vector corresponding to the camera motion history record is obtained, and the key frame camera pose corresponding to the camera key frame is generated according to the music-dance feature vector and the historical key frame feature vector.

[0100] It can be understood that in this embodiment, the ultimate goal is to generate the key frame camera pose corresponding to the camera key frame according to the given music and dance and the camera motion history record. The reason why the camera motion history record is considered here is that adjacent shots are related in real dance camera motion.

[0101] At each pair of adjacent key frames in and , the corresponding music-dance feature vector is obtained for the given music and dance​ The specific acquisition process can be seen in the above embodiment, which is not expanded here.

[0102] In this embodiment, it is assumed that is less than , the camera motion history record is acquired, and the following frames are padded with zeros, so that the historical camera motion context can be obtained, wherein .

[0103] Then, the historical camera motion context is encoded by using an encoder to obtain the historical camera motion feature vector corresponding to the camera motion history record .

[0104] Further, according to the music-dance feature vector and the historical camera motion feature vector, a Transformer decoder and a linear layer are used to generate the keyframe camera pose at and , which can be seen in the following equations (5)-(6).

[0105] (5).

[0106] (6).

[0107] According to equations (5)-(6), the mask is used in this embodiment to select the keyframe camera poses at and from the generated sequence .

[0108] It should be noted that the keyframe camera poses at and are solved here, rather than the keyframe camera poses at and , which is to avoid repeatedly generating the same keyframe camera pose of the keyframe.

[0109] According to the above, the keyframe camera pose corresponding to the keyframe of the camera can be generated according to the given music and dance, and the camera motion history record.

[0110] In addition, Figure 4 shows a keyframe camera pose generation schematic diagram of the keyframe-based dance scene camera motion generation method provided by the embodiment of the application.

[0111] As shown in Figure 4 , the process of generating the keyframe camera pose can be realized by using a pre-trained keyframe camera pose generation model.

[0112] Specifically, the keyframe camera pose generation model comprises a music encoder, a dance encoder, a camera motion encoder, a Transformer decoder, and a linear layer.

[0113] The input of the keyframe camera pose generation model is music features (music context in the foregoing) extracted from a given dance, dance features (dance context in the foregoing), and historical camera motion context extracted from a camera motion history record.

[0114] The output of the keyframe camera pose generation model is a keyframe camera pose at a keyframe of a camera.

[0115] In actual inference, the music encoder is used to encode the music features to obtain a music feature vector, the dance encoder is used to encode the dance features to obtain a dance feature vector, and the camera motion encoder is used to encode the historical camera motion context to obtain a historical camera motion feature vector.

[0116] Then, the output of the music encoder and the output of the dance encoder are spliced to obtain a music-dance feature vector.

[0117] Next, the music-dance feature vector and the historical camera motion feature vector are input into the Transformer decoder together, and then pass through a linear layer to obtain the final output, i.e., the keyframe camera pose at the keyframe of the camera.

[0118] In this embodiment, by generating the keyframe camera pose corresponding to the keyframe of the camera according to the given music and dance and the camera motion history record, and further predicting the interpolation function value corresponding to the non-keyframe between two adjacent keyframes of the camera according to the given music and dance, the camera motion history record and the keyframe camera pose, the target camera motion is generated according to the interpolation function value and the keyframe camera pose. This method designs a three-stage end-to-end dance scene camera motion generation framework of keyframe detection-keyframe camera pose generation-interpolation function value prediction, predicts the interpolation function value between keyframes, rather than directly predicting the camera motion, can realize smoother camera curve, more explicit lens switching and more stable dance scene camera shooting, and further significantly improves the viewing experience of users on the dance.

[0119] On the basis of the above embodiment, further, the interpolation function prediction process of the non-keyframe between two adjacent keyframes of the camera will be described in detail below.

[0120] According to the given music and dance, and the camera motion history record and the key frame camera pose, the in-between function value corresponding to the non-key frame between two adjacent key frames is predicted, including: according to the given music and dance, the music-dance feature vector is coded; according to the camera motion history record and the key frame camera pose, the history key frame camera motion feature vector is coded; according to the music-dance feature vector and the history key frame camera motion feature vector, the in-between function value corresponding to the non-key frame between two adjacent key frames is predicted.

[0121] It can be understood that the key frame camera pose at the and has been obtained, and the ultimate goal in this embodiment is to predict the in-between function value between and , so as to predict the camera pose of the non-key frame between and .

[0122] At each pair of adjacent key frames at and , for the given music and dance, the corresponding music-dance feature vector is obtained, and the specific obtaining process can be referred to the above embodiment, which will not be expanded here.

[0123] According to the previously generated key frame camera pose and the obtained camera motion history record , the frames without history data are filled with zeros to obtain the history key frame camera motion context . Wherein, or .

[0124] Then, the history key frame camera motion context is encoded using an encoder to obtain the history key frame camera motion feature vector .

[0125] Further, according to the music-dance feature vector and the history key frame camera motion feature vector, an in-between function value corresponding to the non-key frame between and is obtained by using a Transformer decoder and a linear layer.

[0126] Specifically, according to the music-dance feature vector and the historical key frame camera motion feature vector, a tween function value corresponding to a non-key frame between two adjacent key frames of the camera is predicted, including: an intermediate variable is obtained by decoding according to the music-dance feature vector and the historical key frame camera motion feature vector; the intermediate variable is subjected to mask processing to retain variables corresponding to the two adjacent key frames of the camera, to obtain a masked variable; the masked variable is subjected to non-negative processing to obtain an increment of the tween function; the increment of the tween function is subjected to a previous term summation processing and normalization processing to obtain the tween function value.

[0127] In detail, the algorithm for predicting the tween function value corresponding to the non-key frame between the two adjacent key frames of the camera includes the following steps 1-6.

[0128] Step 1, the music-dance feature vector and the historical key frame camera motion feature vector are subjected to a Transformer decoder and a linear layer to obtain an intermediate variable .

[0129] Step 2, the intermediate variable is subjected to mask processing to retain variables corresponding to the two adjacent key frames of the camera, to obtain a masked variable .

[0130] Step 3, the masked variable is subjected to non-negative processing to obtain an increment of the tween function .

[0131] Step 4, the increment of the tween function is subjected to a previous term summation processing to obtain a variable .

[0132] Step 5, the variable is subjected to normalization processing to obtain a tween function value .

[0133] Step 6, the tween function value is returned / output .

[0134] After the tween function value corresponding to the non-key frame between the two adjacent key frames of the camera is calculated, the non-key frame camera pose corresponding to the non-key frame can be further generated according to the tween function value and the camera key frame pose, which can be specifically referred to as formula (7).

[0135] (7).

[0136] In formula (7), represents the motion of the camera from to , that is, the combination of the plurality of non-key frame camera poses, , representing the generated keyframe camera pose, representing the keyframe camera pose corresponding keyframe probability.

[0137] Based on the above, the corresponding in-between function value of the non-keyframe between two adjacent keyframes can be predicted according to the given music and dance, the camera motion history record and the keyframe camera pose, and then the non-keyframe camera pose corresponding to the non-keyframe is generated.

[0138] Then, the complete camera motion, i.e., the target camera motion, can be generated according to the non-keyframe camera pose and the keyframe camera pose.

[0139] In addition, Figure 5 The in-between function prediction schematic diagram of the keyframe-based dance scene camera motion generation method provided by the embodiment of the application is shown.

[0140] As Figure 5 shown, the process of in-between function prediction can be realized by an in-between function prediction model.

[0141] Specifically, the in-between function prediction model includes a music encoder, a dance encoder, a keyframe camera motion encoder, a Transformer decoder and a linear layer.

[0142] The input of the in-between function prediction model is the music feature (music context in the above) extracted from the given dance, the dance feature (dance context in the above) and the historical keyframe camera motion context extracted from the camera motion history record and the keyframe camera pose.

[0143] The output of the in-between function prediction model is the in-between function value corresponding to the non-keyframe between two adjacent keyframes.

[0144] In actual inference, the music encoder is used to encode the music feature to obtain a music feature vector, the dance encoder is used to encode the dance feature to obtain a dance feature vector, and the camera motion encoder is used to encode the historical keyframe camera motion context to obtain a historical keyframe camera motion feature vector.

[0145] Then, the output of the music encoder and the output of the dance encoder are spliced to obtain a music-dance feature vector.

[0146] Next, the music-dance feature vector and the historical keyframe camera motion feature vector are input into the Transformer decoder together, and after passing through a linear layer, the final output, i.e., the in-between function value, is obtained.

[0147] It should be noted that the in-between function prediction model can also directly call the music-dance feature vector coded by the camera key frame detection part or the key frame camera pose generation part.

[0148] It should also be noted that the in-between function prediction model is pre-trained, and during training, the comprehensive loss function described in the following formula (8) can be used for optimization.

[0149] (8).

[0150] In formula (8), , , , are the loss functions , , , corresponding weight coefficients. The loss functions , , , are as follows (9)-(12).

[0151] (9).

[0152] (10).

[0153] (11).

[0154] (12).

[0155] In formula (9)-(12), represents the real camera parameters, represents the generated camera parameters, represents the first-order derivative of the real camera parameters with respect to time, represents the first-order derivative of the generated camera parameters with respect to time, represents the second-order derivative of the real camera parameters with respect to time, represents the second-order derivative of the generated camera parameters with respect to time, represents the real joint mask matrix, represents the generated joint mask matrix. The joint mask matrix represents the state of whether each human body joint is within the camera field of view at any time, 0 represents not within the camera field of view, and 1 represents within the camera field of view.

[0156] In the embodiment, the corresponding interpolation function values of the non-key frames between two adjacent key frames of the camera are predicted according to the given music and dance and the camera motion history record and the key frame camera pose, so as to generate the target camera motion according to the interpolation function values and the key frame camera pose. The method predicts the interpolation function values between the key frames instead of directly predicting the camera motion by designing the end-to-end three-stage dance scene camera motion generation framework of key frame detection-key frame camera pose generation-interpolation function value prediction, which can realize smoother camera curve, more explicit lens switching and more stable dance scene camera shooting, and thus significantly improve the viewing experience of the user for the dance.

[0157] In some embodiments, Figure 6 The overall flowchart of the key frame based dance scene camera motion generation method provided by the embodiment of the application is shown.

[0158] As Figure 6 shown, first, the music and dance are given, and then the key frames of the camera on the time axis are detected according to the given music and dance. In Figure 6 the green color is the non-key frame of the camera, and the yellow color is the key frame of the camera.

[0159] Then, the key frame camera pose corresponding to the key frame of the camera is generated according to the given music and dance and the camera motion history record, so as to capture the dance details or highlights. According to Figure 6 It can be seen that the yellow modules have generated the corresponding camera motion, and the camera motion corresponding to the green modules needs to be further generated by predicting the interpolation function.

[0160] Further, the interpolation function values corresponding to the non-key frames between two adjacent key frames of the camera are predicted according to the given music and dance and the camera motion history record and the key frame camera pose.

[0161] Thus, the final complete target camera motion is generated according to the interpolation function values and the key frame camera pose.

[0162] Finally, the target camera motion and the dance are rendered, and the rendered result is obtained.

[0163] In some embodiments, Figure 7 The effect comparison diagram of the key frame based dance scene camera motion generation method provided by the embodiment of the application and the real data and the prior art is shown.

[0164] In Figure 7In this context, the real data is Ground Truth, and existing technologies include DanceCamera3D and DanceCamera3D (Filtered).

[0165] according to Figure 7 As can be seen, compared with the prior art, the keyframe-based dance scene camera motion generation method provided by the present invention synthesizes dance camera motion with more shot changes in a short time.

[0166] At the same time, it can be seen that the filters used in the existing technology DanceCamera3D are unstable and have the risk of incorrect smoothing, causing the character to deviate from the center of the camera view. This verifies that the post-processing-free framework provided by the present invention is meaningful.

[0167] Corresponding to the keyframe-based camera motion generation method for dance scenes described in the preceding embodiments, the present invention also provides a keyframe-based camera motion generation device for dance scenes, specifically, Figure 8 A schematic diagram of the structure of the keyframe-based dance scene camera motion generation device provided in an embodiment of the present invention is shown.

[0168] like Figure 8 As shown, the device includes: a camera keyframe detection module 810, used to determine camera keyframes on the timeline based on given music and dance; a keyframe camera pose acquisition module 820, used to generate the keyframe camera pose corresponding to the given camera keyframe based on the given music and dance and camera motion history; a tween function value prediction module 830, used to predict the tween function value corresponding to the non-keyframe between two adjacent camera keyframes based on the given music and dance, camera motion history and keyframe camera pose; and a target camera motion generation module 840, used to generate target camera motion based on the tween function value and the keyframe camera pose.

[0169] It should be noted that the keyframe-based dance scene camera motion generation device provided in this embodiment can be referred to in correspondence with the keyframe-based dance scene camera motion generation methods described in the above embodiments, and will not be repeated here.

[0170] In the embodiment, the camera keyframe determination module 810 determines the camera keyframes on the time axis according to the given music and dance, the keyframe camera pose generation module 820 generates the keyframe camera poses corresponding to the camera keyframes according to the given music and dance and the camera motion history, and the interpolation function value prediction module 830 predicts the interpolation function values corresponding to the non-keyframes between two adjacent camera keyframes according to the given music and dance, the camera motion history and the keyframe camera poses. Thus, the target camera motion generation module 840 generates the target camera motion according to the interpolation function values and the keyframe camera poses. The device generates the camera motion in the dance scene by designing the end-to-end three-stage dance scene camera motion generation framework of keyframe detection-keyframe camera pose generation-interpolation function value prediction, predicts the interpolation function values between the keyframes instead of directly predicting the camera motion, can realize smoother camera curve, more explicit lens switching and more stable dance scene camera shooting, and thus significantly improves the viewing experience of the user for the dance.

[0171] In addition, the stage design provided by the present application can give users more flexible use experience by using part of the module combination alone, and realize the user-controllable dance scene camera shooting. For example, the user can give the camera keyframes on the time axis, and generate the camera motion by using the second stage model (keyframe camera pose generation model) and the third stage model (interpolation function prediction model); secondly, the user can also give the camera keyframes on the time axis and the corresponding keyframe camera poses, and generate the complete camera motion by using the third stage model.

[0172] Figure 9 An example of an entity structure diagram of an electronic device is shown in Figure 9 As shown, the electronic device can include a processor 910, a communications interface 920, a memory 930 and a communications bus 940, wherein the processor 910, the communications interface 920 and the memory 930 complete mutual communication through the communications bus 940. The processor 910 can invoke the logic instructions in the memory 930 to execute the keyframe-based dance scene camera motion generation method, which includes: determining the camera keyframes on the time axis according to the given music and dance; generating the keyframe camera poses corresponding to the camera keyframes according to the given music and dance and the camera motion history; predicting the interpolation function values corresponding to the non-keyframes between two adjacent camera keyframes according to the given music and dance, the camera motion history and the keyframe camera poses; and generating the target camera motion according to the interpolation function values and the keyframe camera poses.

[0173] In addition, the logic instructions in the memory 930 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the prior art that contributes essentially or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0174] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement a key frame based dance scene camera motion generation method provided by the above method, and the method comprises: determining camera key frames on a time axis according to given music and dance; generating key frame camera poses corresponding to the camera key frames according to the given music and dance, and a camera motion history record; predicting interpolation function values corresponding to non-key frames between two adjacent camera key frames according to the given music and dance, and the camera motion history record and the key frame camera poses; and generating target camera motion according to the interpolation function values and the key frame camera poses.

[0175] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement it without creative labor.

[0176] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0177] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a dance scene camera motion based on key frames, characterized in that, The method comprises the following steps: determining camera keyframes on a time axis according to given music and dance; generating keyframe camera poses corresponding to the camera keyframes according to the given music and dance and camera motion history records; predicting the values of the interpolation functions corresponding to non-keyframes between two adjacent camera keyframes according to the given music and dance, the camera motion history records and the keyframe camera poses; generating target camera motion according to the values of the interpolation functions and the keyframe camera poses. The method of predicting the values of the interpolation functions corresponding to non-keyframes between two adjacent camera keyframes according to the given music and dance, the camera motion history records and the keyframe camera poses comprises: encoding music-dance feature vectors according to the given music and dance; encoding history keyframe camera motion feature vectors according to the camera motion history records and the keyframe camera poses; predicting the values of the interpolation functions corresponding to non-keyframes between two adjacent camera keyframes according to the music-dance feature vectors and the history keyframe camera motion feature vectors. The method of predicting the values of the interpolation functions corresponding to non-keyframes between two adjacent camera keyframes according to the music-dance feature vectors and the history keyframe camera motion feature vectors comprises: decoding intermediate variables according to the music-dance feature vectors and the history keyframe camera motion feature vectors; masking the intermediate variables to retain variables corresponding to two adjacent camera keyframes, obtaining masked variables; non-negativizing the masked variables to obtain increments of the interpolation functions; performing term summation on the increments of the interpolation functions and performing normalization to obtain the values of the interpolation functions.

2. The keyframe-based dance scene camera motion generation method according to claim 1, wherein, The method of determining camera keyframes on a time axis according to given music and dance comprises: obtaining keyframe history records; determining camera keyframes on a time axis according to the given music and dance and the keyframe history records.

3. The keyframe-based dance scene camera motion generation method of claim 2, wherein, The method of determining camera keyframes on a time axis according to the given music and dance and the keyframe history records comprises: encoding music-dance feature vectors according to the given music and dance; obtaining history keyframe feature vectors corresponding to the keyframe history records; predicting keyframe probability sequences according to the music-dance feature vectors and the history keyframe feature vectors; determining a keyframe corresponding to a keyframe probability as a camera keyframe when the keyframe probability is higher than a set probability threshold.

4. The keyframe-based dance scene camera motion generation method of claim 1, wherein, The method of generating keyframe camera poses corresponding to the camera keyframes according to the given music and dance and camera motion history records comprises: encoding music-dance feature vectors according to the given music and dance; obtaining history camera motion feature vectors corresponding to the camera motion history records; generating keyframe camera poses corresponding to the camera keyframes according to the music-dance feature vectors and the history keyframe feature vectors.

5. The keyframe-based dance scene camera motion generation method according to any one of claims 1-4, characterized in that, The method of generating target camera motion according to the values of the interpolation functions and the keyframe camera poses comprises: generate non-keyframe camera poses corresponding to non-keyframes according to the non-keyframe camera poses and the keyframe camera poses; generate the target camera motion according to the non-keyframe function values and the keyframe camera poses.

6. A keyframe-based motion generation device for a dance scene camera, characterized in that, The method comprises the following steps: a keyframe camera pose detection module configured to determine keyframes of a camera according to a given music and dance; a keyframe camera pose acquisition module configured to generate keyframe camera poses corresponding to the keyframes of the camera according to the given music and dance and a camera motion history record; a non-keyframe function value prediction module configured to predict non-keyframe function values corresponding to non-keyframes between two adjacent keyframes of the camera according to the given music and dance and the camera motion history record and the keyframe camera poses; a target camera motion generation module configured to generate a target camera motion according to the non-keyframe function values and the keyframe camera poses. The method comprises the following steps: encoding a music-dance feature vector according to the given music and dance; encoding a history keyframe camera motion feature vector according to the camera motion history record and the keyframe camera poses; predicting non-keyframe function values corresponding to non-keyframes between two adjacent keyframes of the camera according to the music-dance feature vector and the history keyframe camera motion feature vector; The method comprises the following steps: decoding an intermediate variable according to the music-dance feature vector and the history keyframe camera motion feature vector; masking the intermediate variable to retain variables corresponding to two adjacent keyframes of the camera to obtain a masked variable; non-negativizing the masked variable to obtain an increment of the non-keyframe function; performing a previous term summation on the increment of the non-keyframe function and performing normalization to obtain the non-keyframe function values.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the keyframe-based dance scene camera motion generation method of any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the keyframe-based dance scene camera motion generation method of any one of claims 1 to 5.

Images