A camera control method, device, equipment and storage medium

Abstract

The embodiments of the present disclosure relate to a camera movement control method, device, equipment and storage medium. The method proposed herein includes: obtaining posture information of a target object in a target period and audio information associated with the target period; determining a posture feature representation corresponding to the posture information and an audio feature representation corresponding to the audio information; and determining target camera movement information associated with the target object based on the posture feature representation and the audio feature representation, wherein the target camera movement information indicates a parameter sequence of a virtual camera used to generate a picture of the target object in the target period.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly to a camera control method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the development of computer technology, people can use computer technology to create richer media content. In the process of media content production, camera control is an important task. Traditional camera control usually relies on the designer's design of the picture effect, which has a relatively high technical threshold, making it difficult for ordinary users to produce high-quality media files. Summary of the invention

[0003] In a first aspect of the present disclosure, a camera movement control method is provided. The method comprises: obtaining posture information of a target object in a target period and audio information associated with the target period; determining a posture feature representation corresponding to the posture information and an audio feature representation corresponding to the audio information; and determining target camera movement information associated with the target object based on the posture feature representation and the audio feature representation, wherein the target camera movement information indicates a parameter sequence of a virtual camera used to generate a picture of the target object in the target period.

[0004] In a second aspect of the present disclosure, a device for camera movement control is provided. The device includes: an information acquisition module configured to acquire posture information of a target object within a target period and audio information associated with the target period; a feature determination module configured to determine a posture feature representation corresponding to the posture information and an audio feature representation corresponding to the audio information; and a parameter determination module configured to determine target camera movement information associated with the target object based on the posture feature representation and the audio feature representation, wherein the target camera movement information indicates a parameter sequence of a virtual camera used to generate a picture of the target object within the target period.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment capable of implementing some embodiments of the present disclosure is shown;

[0010] Figure 2 A flowchart of a mirror movement control process according to some embodiments of the present disclosure is shown;

[0011] Figure 3A shows an example model architecture according to some embodiments of the present disclosure;

[0012] Figure 3B A schematic diagram of a model training process according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic structural block diagram showing an example apparatus for controlling mirror movement according to some embodiments of the present disclosure; and

[0014] Figure 5 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0018] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects are subject to the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user knows and confirms. Accordingly, when implementing each embodiment of the present disclosure, the type, scope of use, usage scenario, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0019] In this specification and the embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or it is necessary to perform a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0020] As briefly mentioned above, camera control is an important task in the production process of media content. Traditional camera control usually relies on the design of the picture effect by the designer, which has a relatively high technical threshold, making it difficult for ordinary users to produce high-quality media files.

[0021] The embodiment of the present disclosure proposes a scheme for camera movement control. According to the scheme, the posture information of the target object in the target time period and the audio information associated with the target time period can be obtained. Further, the posture feature representation corresponding to the posture information and the audio feature representation corresponding to the audio information can be determined. Accordingly, the target camera movement information associated with the target object can be determined based on the posture feature representation and the audio feature representation, wherein the target camera movement information indicates the parameter sequence of the virtual camera used to generate the image of the target object in the target time period.

[0022] In this way, the embodiments of the present disclosure can automatically generate adaptive camera movement information based on the posture of the object in the picture to be generated and the corresponding audio information, so that the corresponding picture can better match the posture and audio of the object, thereby improving the quality of camera movement control. Furthermore, the embodiments of the present disclosure can reduce the learning cost of camera movement control and improve the production efficiency of media content.

[0023] Various example implementations of the solution are described in detail below in conjunction with the accompanying drawings.

[0024] Example Environment

[0025] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 .

[0026] In this example environment 100, the electronic device 110 may acquire audio information 120. Taking the dance process involving the target object (eg, a virtual character) in camera control as an example, such audio information 120 may include music corresponding to the dance process.

[0027] Furthermore, the electronic device 110 may also obtain the posture information of the target object. Continuing with the dance process as an example, the posture information may represent the dance posture of the target object.

[0028] As will be described in detail below, the electronic device 110 can determine the camera movement information 130 based on the audio information 120 and the posture information. Such camera movement information can be used to indicate corresponding parameters of the virtual camera, such as position parameters, rotation parameters, field of view parameters, etc.

[0029] In some embodiments, such camera movement information 130 may be further provided for generating media content 140 associated with the target object. As an example, such media content may include different pictures generated based on the camera movement parameters indicated by the camera movement information 130.

[0030] In some examples, the electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable game terminals, VR / AR devices, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDA), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, game devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as "wearable" circuits, etc.).

[0031] In other examples, the electronic device 110 may be, for example, an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The electronic device 110 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0032] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0033] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0034] Example Process

[0035] Figure 2 FIG. 2 is a flowchart showing an example process 200 for camera movement control according to some embodiments of the present disclosure. The process 200 may be implemented, for example, in Figure 1 At the electronic device 110 shown.

[0036] like Figure 2 As shown, in box 210, the electronic device 110 obtains the posture information of the target object within the target time period and the audio information associated with the target time period.

[0037] The following will refer to Figure 3A Describing process 200, Figure 3A An example model architecture 300A is shown according to some embodiments of the present disclosure. Figure 3A As shown, the electronic device 110 may obtain audio information 305 as an audio condition, and may obtain posture information 310 as a posture condition.

[0038] In some embodiments, audio information 305 may include audio of any appropriate content associated with the target object's actions during the target period. For example, such audio information 305 may indicate the background music used when the target object performs a dance move. Alternatively, such audio information 305 may also include, for example, a soundtrack used in the media content to be generated. As another example, such audio information 305 may also include, for example, the target object's singing audio, etc.

[0039] As an example, the audio information 305 may be represented as:

[0040] m={m 1 , m 2 , ..., m N} (1)

[0041] Where N is the number of frames, m i Represents the audio information of the corresponding frame.

[0042] In some embodiments, the gesture information 310 may represent a set of motion gestures of the target object within a target time. In some examples, the motion gestures of the target object may be represented by, for example, position information of its key points.

[0043] Taking the target object as an example, the posture of the target object may include position information of a plurality of preset joint points. As an example, the posture information 310 may be expressed as:

[0044] p={p 1 , p 2 , ..., p N} (2)

[0045] in, represents the spatial positions of multiple joint points (e.g., 60), p i Indicates the posture information of the corresponding frame.

[0046] Continue to refer Figure 2 In box 220, the electronic device 110 determines a gesture feature representation corresponding to the gesture information and an audio feature representation corresponding to the audio information.

[0047] like Figure 3A As shown, the electronic device 110 may, for example, utilize the feature extractor 315 to extract a set of features of the audio information 305, and may utilize the audio encoder 320 to generate a corresponding audio feature representation m emb.

[0048] In addition, the electronic device 110 may utilize the posture encoder 325 to process the position-encoded posture information 310, thereby generating a corresponding posture feature representation p emb .

[0049] Continue to refer Figure 2 In box 230, the electronic device 110 determines the target camera movement information associated with the target object based on the posture feature representation and the audio feature representation, and the target camera movement information indicates a parameter sequence of a virtual camera used to generate a picture of the target object within the target time period.

[0050] In some embodiments, Figure 3A As shown, the electronic device 110 may utilize a camera control model 355 to determine target camera information 365. In some examples, such a camera control model 355 may include, for example, a plurality of decoders 345.

[0051] For the decoder 345, the electronic device 110 may represent m based on the gesture feature emb And the audio feature representation p emb , determining an input feature representation 330 for input to a target model.

[0052] like Figure 3A As shown, the camera control module 355 can be a diffusion model based on a transformer. The input feature representation 335 can also include a diffusion parameter 335 (e.g., T) encoded by position. Such a diffusion parameter T can be used to represent the time step of the diffusion process.

[0053] Furthermore, if Figure 3A As shown, the position-encoded diffusion parameter T may also be provided to a multi-layer perceptron (MLP) 340, such as to be added to the input feature representation 330, as an input to a feature linear modulation layer.

[0054] Further, the camera movement control model 355 can also obtain initial camera movement information 350, which can include, for example, a noise sequence. The noise sequence can include, for example, random camera movement information. Further, the camera movement control model can perform denoising on the noise sequence 350 based on the audio information, the posture information, and the diffusion parameter T to generate a synthetic sequence 360, also called intermediate camera movement information.

[0055] Specifically, the electronic device 110 may utilize the camera movement control model 355 to execute a predetermined round of diffusion process, wherein each round of diffusion process may include a noise addition process and a noise removal process.

[0056] by Figure 3AAs an example, the synthesized sequence 360 ​​may further perform T-1 times of denoising, and a denoising process may be performed based on the audio information, the gesture information, and the noisy synthesized sequence 360 ​​using a camera control model to generate denoised intermediate camera information.

[0057] In some embodiments, such a diffusion process may be performed for a predetermined number of times controlled by the diffusion parameter T until the final target camera movement information 365 is determined.

[0058] In some embodiments, the target mirror movement information 365 may indicate multiple positions of the virtual camera within the target period, which may be represented by three-dimensional coordinates, for example. Additionally or alternatively, the target mirror movement information 365 may also indicate multiple postures of the virtual camera within the target period, which may be represented by three rotation angles, for example. Additionally or alternatively, the target mirror movement information 365 may also indicate multiple field of view (FOV) parameters of the virtual camera within the target period.

[0059] As an example, the camera movement information x can be expressed as:

[0060]

[0061] It can also be expressed in the format of camera center:

[0062]

[0063] Among them, xc includes the coordinates of the camera center position, the 3*3 rotation matrix, and the FOV parameters.

[0064] Based on the process described above, the embodiment of the present disclosure can automatically generate adaptive camera movement information based on the posture of the object in the picture to be generated and the corresponding audio information, so that the corresponding picture can better match the posture and audio of the object, thereby improving the quality of camera movement control.

[0065] The following will further combine Figure 3B The training process of the camera control model 355 is introduced. Such a training process can be performed by any appropriate training device (for example, the electronic device 110 or other devices).

[0066] like Figure 3B As shown, in the process of training the camera movement control model 355, the training device can obtain training data, and such training data may include sample audio information, sample posture information and corresponding sample camera movement information.

[0067] like Figure 3BAs shown, the sample camera movement information may include a true value sequence 370 (ie, ground-truth). In some embodiments, the training device may perform t times of noise addition on the true value sequence 370, thereby obtaining a noise sequence 372 after the noise addition. In some embodiments, the diffusion parameter t may be determined randomly, for example.

[0068] Further, the camera control model 355 can perform a denoising process based on the sample audio information, the sample posture information, the noise sequence 372 and the diffusion parameter t, thereby generating a synthetic sequence 374, i.e., the intermediate camera information. It should be understood that the generation process of the synthetic sequence 374 can refer to the description of the synthetic sequence 360 ​​above, and will not be repeated here.

[0069] Furthermore, the training device may determine the training loss based on a comparison between the intermediate camera movement information and the sample camera movement information.

[0070] In some embodiments, the training device may determine a first portion 376 of the training loss based on a comparison of the intermediate camera information and the sample camera information. In some examples, the training loss may include a position loss, which may represent the difference between a first set of positions of the virtual camera indicated by the intermediate camera information and a second set of positions of the virtual camera indicated by the sample camera information. For example, the position loss may be represented as:

[0071]

[0072] In some examples, the training loss may include a speed loss, which may represent: a difference between a first set of speeds of the virtual camera indicated by the intermediate camera movement information and a second set of speeds of the virtual camera indicated by the sample camera movement information. For example, the speed loss may be represented as:

[0073]

[0074] In some examples, the training loss may include a speed loss, which may represent: a difference between a first set of accelerations of the virtual camera indicated by the intermediate camera movement information and a second set of accelerations of the virtual camera indicated by the sample camera movement information. For example, the acceleration loss may be represented as:

[0075]

[0076] In the above formula, x′ and x″ represent the first-order derivative and the second-order derivative of the position information with respect to time, respectively.

[0077] In some embodiments, in order to better capture the action of the target object through the picture, the loss function may include, for example, a second part 386, namely, the attention loss associated with the target object.

[0078] Specifically, Figure 3B As shown, the training device can use the mask extractor 378 to determine the mask information 382 corresponding to the true value sequence 370 (i.e., the sample camera movement information), and can use the mask extractor 380 to determine the mask information 384 corresponding to the synthetic sequence 370 (i.e., the intermediate camera movement information).

[0079] Further, the training device may determine the attention loss based on the comparison of the mask information 382 and the mask information 384.

[0080] In some embodiments, the mask information 382 or the mask information 384 may indicate whether a set of key points of the target object is in the camera view, which may be expressed as:

[0081]

[0082] Accordingly, attention loss It can be expressed as:

[0083]

[0084] In some embodiments, taking the target object as a human object as an example, such a set of key points may include, for example, a plurality of preset key points of the human object.

[0085] Based on the training process discussed above, the embodiments of the present disclosure can not only ensure the smoothness of the camera movement of the trained camera movement control model, but also ensure that the target object can be presented in the picture in a more appropriate manner.

[0086] Example devices and equipment

[0087] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an example device 400 for camera movement control according to some embodiments of the present disclosure is shown. The device 400 may be implemented as or included in an electronic device. Each module / component in the device 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0088] like Figure 4As shown, the device 400 includes an information acquisition module 410, which is configured to acquire posture information of the target object within a target time period and audio information associated with the target time period; a feature determination module 420, which is configured to determine a posture feature representation corresponding to the posture information and an audio feature representation corresponding to the audio information; and a parameter determination module 430, which is configured to determine target camera movement information associated with the target object based on the posture feature representation and the audio feature representation, the target camera movement information indicating a parameter sequence of a virtual camera used to generate a picture of the target object within the target time period.

[0089] In some embodiments, the parameter determination module 430 is further configured to: determine an input feature representation for input into a target model based on a posture feature representation and an audio feature representation; and determine target camera movement information by the target model based on the input feature representation and the initial camera movement information, wherein the initial camera movement information includes a noise sequence.

[0090] In some embodiments, the target model includes a converter-based diffusion model.

[0091] In some embodiments, the parameter determination module 430 is further configured to: perform a predetermined number of diffusion processes by a diffusion model to determine the target camera movement information, the diffusion process includes a denoising process and a denoising process, the denoising process is used to perform noise processing on the input camera movement information, and the denoising process is used to generate denoised camera movement information based on the input feature representation and the noisy input camera movement information.

[0092] In some embodiments, the input feature representation is further determined based on a diffusion parameter of the diffusion model, the diffusion parameter indicating a time step of the diffusion process.

[0093] In some embodiments, the target model is trained based on the following process: obtaining training data, the training data including sample audio information, sample posture information and corresponding sample camera movement information; performing noise processing on the sample camera movement information; using the model to be trained to process the sample audio information, sample posture information and the noisy sample camera movement information to generate intermediate camera movement information; determining the training loss based on the comparison between the intermediate camera movement information and the sample camera movement information; and adjusting the parameters of the model to be trained based on the training loss to obtain the target model.

[0094] In some embodiments, the training loss indicates at least one of: the difference between a first set of positions of the virtual camera indicated by the intermediate camera movement information and a second set of positions of the virtual camera indicated by the sample camera movement information; the difference between a first set of velocities of the virtual camera indicated by the intermediate camera movement information and a second set of velocities of the virtual camera indicated by the sample camera movement information; the difference between a first set of accelerations of the virtual camera indicated by the intermediate camera movement information and a second set of accelerations of the virtual camera indicated by the sample camera movement information.

[0095] In some embodiments, determining the training loss based on the comparison between the intermediate camera movement information and the sample camera movement information includes: determining a first part of the training loss based on the comparison between the intermediate camera movement information and the sample camera movement information; determining first mask information based on the intermediate camera movement information, the first mask information indicating whether a group of key points of the target object are in the camera view corresponding to the intermediate camera movement information; determining second mask information based on the sample camera movement information, the second mask information indicating whether a group of key points of the target object are in the camera view corresponding to the sample camera movement information; and determining a second part of the training loss based on the comparison between the first mask information and the second mask information.

[0096] In some embodiments, the target object comprises a human body object, and the set of key points comprises a set of joint points of the human body object.

[0097] In some embodiments, the parameter sequence is associated with at least one of the following parameters: the position of the virtual camera; the pose of the virtual camera; the field of view of the virtual camera.

[0098] In some embodiments, the apparatus 400 further includes a media generation module configured to generate multimedia content associated with the target object based on the target camera movement information.

[0099] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The electronic device 500 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown may be used in an electronic device.

[0100] like Figure 5 As shown, the electronic device 500 is in the form of a general electronic device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0101] The electronic device 500 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 500.

[0102] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0103] The communication unit 540 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0104] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 500, or communicate with any device that allows the electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0105] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0106] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0110] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A camera movement control method, comprising: Acquire posture information of a target object within a target period and audio information associated with the target period; Acquire a posture feature representation obtained by encoding the posture information and an audio feature representation obtained by encoding the audio information; as well as Based on the gesture feature representation and the audio feature representation, target camera movement information associated with the target object is determined, where the target camera movement information indicates a parameter sequence of a virtual camera used to generate a picture of the target object within the target time period.

2. The method according to claim 1, wherein determining the camera movement information associated with the target object based on the posture feature representation and the audio feature representation comprises: Determining an input feature representation for inputting into a target model based on the posture feature representation and the audio feature representation; as well as The target camera movement information is determined by the target model based on the input feature representation and initial camera movement information, wherein the initial camera movement information includes a noise sequence. The method of claim 2 , wherein the target model comprises a converter-based diffusion model.

4. The method according to claim 3, wherein determining, by the target model, the target camera movement information based on the input feature representation and the initial camera movement information comprises: A predetermined number of diffusion processes are performed by the diffusion model to determine the target camera movement information, and the diffusion process includes a denoising process and a denoising process. The denoising process is used to perform noise processing on the input camera movement information, and the denoising process is used to generate denoised camera movement information based on the input feature representation and the noisy input camera movement information. The method of claim 3 , wherein the input feature representation is further determined based on a diffusion parameter of the diffusion model, the diffusion parameter indicating a time step of a diffusion process.

6. The method of claim 2, wherein the target model is trained based on the following process: Acquire training data, where the training data includes sample audio information, sample posture information, and corresponding sample camera movement information; Performing noise addition processing on the sample camera movement information; Processing the sample audio information, the sample posture information and the noisy sample camera movement information using the model to be trained to generate intermediate camera movement information; Determining a training loss based on a comparison between the intermediate camera movement information and the sample camera movement information; as well as The parameters of the model to be trained are adjusted based on the training loss to obtain the target model.

7. The method of claim 6, wherein the training loss indicates at least one of: a difference between a first set of positions of the virtual camera indicated by the intermediate camera movement information and a second set of positions of the virtual camera indicated by the sample camera movement information; a difference between a first set of speeds of the virtual camera indicated by the intermediate camera movement information and a second set of speeds of the virtual camera indicated by the sample camera movement information; The difference between a first set of accelerations of the virtual camera indicated by the intermediate camera movement information and a second set of accelerations of the virtual camera indicated by the sample camera movement information.

8. The method according to claim 6, wherein determining the training loss based on the comparison between the intermediate camera movement information and the sample camera movement information comprises: determining a first part of the training loss based on the comparison of the intermediate camera movement information and the sample camera movement information; Based on the intermediate camera movement information, determine first mask information, where the first mask information indicates whether a group of key points of the target object are in a camera view corresponding to the intermediate camera movement information; Based on the sample camera movement information, determine second mask information, where the second mask information indicates whether the set of key points of the target object is in a camera view corresponding to the sample camera movement information; as well as Based on the comparison of the first mask information and the second mask information, a second portion of the training loss is determined. 9 . The method according to claim 8 , wherein the target object comprises a human object, and the set of key points comprises a set of joint points of the human object.

10. The method according to claim 1, wherein the parameter sequence is associated with at least one of the following parameters: The position of the virtual camera; the pose of the virtual camera; The field of view of the virtual camera.

11. The method according to claim 1, further comprising: Based on the target camera movement information, multimedia content associated with the target object is generated.

12. An apparatus for information processing, comprising: An information acquisition module, configured to acquire posture information of a target object within a target period and audio information associated with the target period; a feature determination module configured to obtain a posture feature representation obtained by encoding the posture information and an audio feature representation obtained by encoding the audio information; as well as A parameter determination module is configured to determine target camera movement information associated with the target object based on the posture feature representation and the audio feature representation, wherein the target camera movement information indicates a parameter sequence of a virtual camera used to generate a picture of the target object within the target time period.

13. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 11.

Images