Media generation method and device, equipment and medium

Abstract

The invention provides a media generation method and device, equipment and a medium. A specific embodiment of the method comprises the following steps: acquiring a reference image and a noise image; acquiring control information for guiding generation of media, wherein the control information comprises information of a target subject in the reference image, information of a processing category to which the target subject belongs and motion guidance information of the target subject; and de-noising the noise image by using a target model based on the reference image and the control information to obtain a target medium, and enabling the target subject in the target medium to move according to the motion guidance information. According to the embodiment, the target subject moving in the target media can operate according to the will of the user, the media generation ways are enriched, the requirements of the user can be better met, and the user experience is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of machine learning technology, and more particularly to a method, apparatus, device, and medium for generating media. Background Art

[0002] At present, machine learning can be used to generate a relevant video or animation based on an existing image. In related technologies, a video or animation is generally generated based on only one frame of an existing image, or a video or animation can be generated based on an existing frame of an image and a descriptive text. Because it is impossible to generate a video or animation accurately according to the user's intention, it is difficult to meet the user's needs. At present, a solution for generating media is needed. Summary of the invention

[0003] The embodiments of the present disclosure describe a method, apparatus, device and medium for generating media.

[0004] According to a first aspect, a method for generating media is provided, the method comprising: obtaining a reference image and a noise image; obtaining control information for guiding the generation of media, the control information comprising information of a target subject in the reference image, information of a processing category to which the target subject belongs, and motion guidance information of the target subject; performing denoising processing on the noise image based on the reference image and the control information using a target model to obtain a target media, so that the target subject in the target media moves according to the motion guidance information.

[0005] According to the second aspect, a media generation device is provided, the device comprising: an acquisition unit, configured to acquire a reference image and a noise image; a control unit, configured to acquire control information for guiding the generation of media, the control information comprising information of a target subject in the reference image, information of a processing category to which the target subject belongs, and motion guidance information of the target subject; a denoising unit, configured to perform denoising on the noise image based on the reference image and the control information using a target model to obtain a target media, so that the target subject in the target media moves according to the motion guidance information.

[0006] According to a third aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method as described above in any one of the first aspect is implemented.

[0007] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute any one of the methods in the first aspect.

[0008] According to a fifth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, any one of the methods in the first aspect is implemented.

[0009] According to the media generation scheme provided by the embodiment of the present disclosure, by obtaining a reference image and a noise image, control information for guiding the generation of media is obtained, the control information includes information of the target subject in the reference image, information of the processing category to which the target subject belongs, and motion guidance information of the target subject, and the target model is used to perform denoising on the noise image based on the reference image and the control information to obtain the target media, so that the target subject in the target media moves according to the motion guidance information. Thus, the moving target subject in the target media can operate according to the user's wishes, which not only enriches the media generation path, but also better meets the needs of users and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of a scene of media generation according to an exemplary embodiment of the present disclosure;

[0011] Figure 2 is a flow chart of a method for generating media according to an exemplary embodiment of the present disclosure;

[0012] Figure 3 is a block diagram of a device for generating media according to an exemplary embodiment of the present disclosure;

[0013] Figure 4 It is a schematic block diagram of an electronic device provided in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0014] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0015] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0016] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0017] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0018] The technical solution provided by the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the inventions. It should also be noted that, for ease of description, only the parts related to the relevant inventions are shown in the accompanying drawings. It should be noted that, in the absence of conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0019] At present, machine learning can be used to generate a relevant video or animation based on an existing image. In related technologies, a video or animation is generally generated based on only one frame of an existing image, or based on one frame of an existing image and a descriptive text. Because it is impossible to generate a video or animation accurately according to the user's intention, it is difficult to meet the user's needs.

[0020] The present disclosure provides a media generation scheme, which obtains control information for guiding the generation of media by obtaining a reference image and a noise image, wherein the control information includes information of a target subject in the reference image, information of a processing category to which the target subject belongs, and motion guidance information of the target subject, and uses a target model to perform denoising on the noise image based on the reference image and the control information to obtain a target media, so that the target subject in the target media moves according to the motion guidance information. Thus, the moving target subject in the target media can operate according to the user's wishes, which not only enriches the media generation method, but also better meets the needs of users and improves the user experience.

[0021] See also Figure 1 , is a schematic diagram of a scenario of media generation according to an exemplary embodiment.

[0022] like Figure 1As shown, taking the application stage of model M as an example, model M can be a pre-trained diffusion model. First, the user can select a frame of reference image and operate through the user interface provided by the terminal device to select the target subject in the reference image, set the information of the processing category to which the target subject belongs and the motion guidance information of the target subject. For example, the user can set the mirror movement parameter information for guiding all target subjects to move according to the mirror movement trajectory through the mirror movement setting interface. The user can also set the specified trajectory information for guiding the target subject a to move in a specified manner through the trajectory setting interface. The user can also set the random motion intensity information for guiding the target subject a and the target subject b to move in a random manner through the random motion setting interface. The above-mentioned mirror movement parameter information, specified trajectory information and random motion intensity information can be used as control information.

[0023] Next, a control tensor can be calculated based on the control information. The control tensor may include a tensor 1 for representing the motion trajectory, a tensor 2 for representing the random motion intensity, a tensor 3 for representing the target subject identification, and a tensor 4 for representing the processing category to which the target subject belongs. The control tensor is input into the adapter for adaptation processing to obtain an adaptation result. Finally, the reference image, the adaptation result, and the noisy image can be input into a pre-trained model M, so that the model M can perform denoising processing on the noisy image based on the reference image and the adaptation result to obtain the target media.

[0024] The present disclosure will be described in detail below in conjunction with specific embodiments.

[0025] Figure 2 The flowchart of a method for generating media according to an exemplary embodiment is shown. The method can be applied in a terminal device. In this embodiment, for ease of understanding, an example is given in conjunction with a terminal device that can install a media processing application. Those skilled in the art can understand that the terminal device can include but is not limited to mobile terminal devices such as smart phones, smart wearable devices, tablet computers, and desktop computers. The method can be applied in the application stage of the target model, and can also be applied in the reverse process of the target model training stage, wherein the target model can be a diffusion model. The method can include the following steps:

[0026] like Figure 2 As shown, in step 201, a reference image and a noise image are acquired.

[0027] In this embodiment, the media to be generated may be a dynamic media composed of multiple frames of images, for example, the media may be a video, an animation, or a gif. The noise image may be a random noise image, for example, a noise image randomly generated by Gaussian white noise. This embodiment does not limit the specific generation method of the noise image. The reference image may provide a reference and basis for generating the media. Multiple frames of coherent images may be generated based on the reference image, and the reference image may be used as the first frame image to form the target media. If in the application stage of the target model, the reference image may be any frame image selected or input by the user. If in the reverse direction process in the target model training stage, the sample media may be obtained first, and the first frame image of the sample media may be used as the reference image.

[0028] In step 202, control information for guiding the generation of media is obtained.

[0029] In this embodiment, the control information can be used to guide the generation of media, and the control information may include information about the target subject in the reference image, information about the processing category to which the target subject belongs, and motion guidance information about the target subject. The reference image may include one or more target subjects, each of which may be a semantic unit in the image, and motion-related processing may be performed in units of a semantic unit, so that in the generated media, the target subject can move in a preset manner. For example, the target subject may be a person in the reference image, an animal in the reference image, or an object in the reference image.

[0030] Specifically, the information of the target subject may be unique identification information for indicating the target subject in the reference image, for example, the information of the target subject may be the area information of the target subject in the reference image, or may be the number information of the semantic unit to which the target subject belongs in the reference image, etc. The information of the processing category to which the target subject belongs may be, for example, the number information corresponding to the category to which the target subject belongs, etc. Among them, the processing category may include a first category, a second category and a third category, the first category is a category processed according to a specified camera movement mode, a specified motion mode and a random motion mode, the second category is a category processed according to a specified camera movement mode and a random motion mode, and the third category is a category processed according to a specified camera movement mode. For example, if the processing category to which subject A belongs is the first category, then in the generated media, subject A may move in a specified manner (for example, moving along a specified path) plus a random manner in the specified camera movement effect, that is, the movement of subject A includes a part moving along a specified path and a part moving randomly, and has a camera movement effect. If the processing category to which subject A belongs is the second category, then in the generated media, subject A moves in a random manner in the specified camera movement effect. If the processing category to which subject A belongs is the third category, then in the generated media, subject A moves along with the camera effect in the specified camera effect.

[0031] The motion guidance information of the target subject may be information for guiding the motion processing of the target subject. The motion guidance information may include at least one of the following: camera movement parameter information for guiding the target subject to move according to the camera movement trajectory; specified trajectory information for guiding the target subject to move in a specified manner; random motion intensity information for guiding the target subject to move in a random manner. For example, for the target subject belonging to the first category, the motion guidance information may include camera movement parameter information, specified trajectory information and random motion intensity information. For the target subject belonging to the second category, the motion guidance information may include camera movement parameter information and random motion intensity information. For the target subject belonging to the third category, the motion guidance information may include camera movement parameter information. Among them, the random motion intensity information may be a value for indicating the amount of random motion, which may be represented by a random motion intensity value. The larger the random motion intensity value, the more random motion (i.e., the longer the path of the random motion), and the smaller the random motion intensity value, the less random motion. The camera movement parameter information may be information that can represent the camera movement trajectory and indicate the camera movement processing.

[0032] In this embodiment, during the application stage of the target model, control information for guiding the generation of media can be obtained in the following manner: first, the terminal device can output a reference image to the user through a user interface, and the user performs a first operation and a second operation on the reference image on the user interface. The first operation is an operation of selecting a target subject and setting a processing category to which the target subject belongs, and the second operation is an operation of setting motion guidance information of the target subject. Finally, the target subject and information on the processing category to which the target subject belongs can be determined based on the user's first operation, and the motion guidance information of the target subject can be determined based on the user's second operation.

[0033] For example, for the user's first operation, in one implementation, the terminal device can display a reference image through an operation interface of an application for generating media, and the user can select the target subject separately on the operation interface. The terminal device can separate the target subject from the area selected by the user through semantic segmentation and number it, so as to obtain the information of the target subject. In addition, the user sets a processing category for each target subject. For example, after the user selects the target subject, the processing category can be set for the selected target subject by triggering buttons of different processing categories. Specifically, after the user selects subject 1, the button corresponding to the first category is triggered, and the processing category of subject 1 can be set to the first category. For another example, the user can also trigger buttons of different processing categories before selecting the target subject, and then select the target subject to set the processing category for the target subject. Specifically, the user can trigger the button corresponding to the second category, and then after selecting subject 2, the processing category of subject 2 can be set to the second category.

[0034] In another implementation, the terminal device may also first perform semantic segmentation on the reference image to obtain multiple semantic regions, and number each semantic region. Then, the reference image after semantic segmentation is displayed through the operation interface of the application used to generate the media, and the user can select the semantic region as the target subject by triggering on the operation interface, and the number of the semantic region can be obtained as the information of the target subject. In addition, the user sets the processing category for each target subject. For example, after the user selects the target subject, the processing category can be set for the selected target subject by triggering buttons of different processing categories. For another example, the user can also trigger buttons of different processing categories before selecting the target subject, and then select the target subject to set the processing category for the target subject.

[0035] For the second operation of the user, after selecting the target subject and setting the processing category to which the target subject belongs, the user can then set the motion guidance information of the target subject. For example, for the target subject belonging to the first category, the user can draw the trajectory information of the target subject by dragging on the screen (for example, translating or rotating the target subject), and set the intensity value of random motion as random motion intensity information by selecting or inputting, and set the motion parameters by the motion parameter setting button. For the target subject belonging to the second category, the user can set the intensity value of random motion as random motion intensity information by selecting or inputting, and set the motion parameters by the motion parameter setting button. For the target subject belonging to the third category, the user can only set the motion parameters by the motion parameter setting button. It should be noted that if the reference image includes multiple target subjects, a unified motion parameter can be set for the multiple target subjects, and different motion parameters do not need to be set for each target subject.

[0036] In this embodiment, in the reverse process of the target model training stage, the control information for guiding the generation of media can be obtained in the following manner: First, before obtaining the reference image, the sample media can be obtained first. The sample media can be a pre-recorded or produced video, animation or animated image, and the sample media can include at least one movable subject. The first frame image of the sample media can be obtained as a reference image, and the reference image can be semantically segmented to obtain multiple semantic areas, and each semantic area is numbered. Then, based on the results of the semantic segmentation, multiple target subjects are determined, and information on the processing categories to which the multiple target subjects belong is set. For example, the processing category to which the target subject belongs can be specified by the user, or the processing category to which the target subject belongs can be randomly set. Finally, the motion guidance information of the target subject can be determined based on the image frame after the reference image in the sample media and the information on the processing category to which the target subject belongs.

[0037] Specifically, the pixel points corresponding to the target subject in the image can be determined based on the reference image. Each frame of the sample video is obtained, and the spatial coordinates corresponding to each pixel point in each frame are calculated by analyzing these image frames. Then, the pixel points corresponding to the target subject in each frame of the image and the spatial coordinates corresponding to each pixel point are determined, so as to obtain the trajectory data corresponding to the target subject.

[0038] In addition, different trajectory functions can be set in advance for different processing categories, and each trajectory function includes a specified motion function item and a random motion function item. At least one function item in the trajectory functions corresponding to the target subjects belonging to different processing categories is different. For example, the specified motion function item and the random motion function item corresponding to the target subject of the first category are different from those of the target subject of the third category. The specified motion function item corresponding to the target subject of the first category is different from that of the target subject of the second category. The random motion function item corresponding to the target subject of the third category is different from that of the target subject of the second category. Among them, the specified motion function item corresponding to the target subject of the third category and the target subject of the second category can be a unit matrix. The random motion function item corresponding to the target subject of the third category can be a 0 matrix. Therefore, the trajectory function corresponding to the target subject can be determined according to the processing category to which the target subject belongs.

[0039] It should be noted that the designated motion function item included in the trajectory function is related to the designated trajectory information in the motion guidance information, and the random motion function item included in the trajectory function is related to the random motion intensity information in the motion guidance information.

[0040] Finally, the motion guidance information of the target subject can be determined based on the trajectory data corresponding to the target subject and the trajectory function corresponding to the target subject. For example, for the target subject of the first category, multiple simulated trajectories input by the user can be randomly simulated first, and each simulated trajectory can be tested using the trajectory data corresponding to the target subject, and the simulated trajectory closest to the trajectory data corresponding to the target subject is selected as the designated trajectory information in the motion guidance information of the target subject for guiding the target subject to move in a designated manner. Then, the designated motion function term in the trajectory function corresponding to the target subject can be subtracted from the trajectory data corresponding to the target subject to obtain the random motion function term in the trajectory function corresponding to the target subject, and based on the random motion function term, the random motion intensity information in the motion guidance information of the target subject for guiding the target subject to move in a random manner can be obtained.

[0041] For another example, for the target subject of the second category, the random motion function item in the trajectory function corresponding to the target subject can be obtained directly according to the trajectory data corresponding to the above-mentioned target subject, and based on the random motion function item, the random motion intensity information in the motion guidance information of the target subject for guiding the target subject to move in a random manner can be obtained, and based on the corresponding specified motion function item, a unit matrix can be used to represent the specified trajectory information for guiding the target subject to move in a specified manner.

[0042] For the target subject of the third category, based on its corresponding designated motion function item and random motion function item, a unit matrix can be used to represent the designated trajectory information for guiding the target subject to move in a designated manner, and a 0 matrix can be used to represent the random motion intensity information for guiding the target subject to move in a random manner.

[0043] It should be noted that for any frame of reference image, no matter how many target subjects are included in the reference image, each frame of the sample media can be analyzed using a preset algorithm to parse out the camera movement parameter information, which can be used as the camera movement parameter information corresponding to each target subject.

[0044] In step 203, the target model is used to perform denoising on the noisy image based on the reference image and the control information to obtain the target medium.

[0045] In this embodiment, a control tensor can be obtained based on the above control information. The control tensor may include a first tensor for representing the motion trajectory, a second tensor for representing the random motion intensity, a third tensor for representing the target subject identifier, and a fourth tensor for representing the processing category to which the target subject belongs. Specifically, for example, after determining the designated trajectory information and the camera movement parameter information included in the control information, motion fusion processing can be performed based on the designated trajectory information and the camera movement parameter information to obtain a first tensor that fuses the designated trajectory and the camera movement trajectory. Then, a second tensor is generated based on the random motion intensity information, a third tensor is generated based on the number corresponding to the target subject, and a fourth tensor is generated based on the processing category to which the target subject belongs. It should be noted that if the control information only includes the designated trajectory information, the first tensor can be generated directly based on the designated trajectory information. If the control information only includes the camera movement parameter information, the first tensor is generated directly based on the camera movement parameter information.

[0046] Then, the control tensor is input into the target adapter to obtain the target result output by the target adapter. In the application phase of the target model, the target adapter can be a pre-trained adapter. In the reverse process of the target model training phase, the target adapter can be an adapter to be trained. In the parameter adjustment process, only the parameters of the target adapter can be adjusted without adjusting the parameters of the target model.

[0047] Finally, the reference image and the target result can be used to guide the target model to perform denoising on the noisy image. Specifically, the reference image, the target result and the noisy image can be input into the target model respectively, so that the target model can perform denoising on the noisy image based on the reference image and the target result to obtain the target media, in which the target subject moves according to the motion guidance information.

[0048] The present disclosure provides a method for generating media, which obtains control information for guiding the generation of media by obtaining a reference image and a noise image, wherein the control information includes information of a target subject in the reference image, information of a processing category to which the target subject belongs, and motion guidance information of the target subject, and uses a target model to perform denoising processing on the noise image based on the reference image and the control information to obtain a target media, so that the target subject in the target media moves according to the motion guidance information. Thus, the moving target subject in the target media can operate according to the user's wishes, which not only enriches the way of media generation, but also better meets the needs of users and improves the user experience.

[0049] It should be noted that although the operations of the method of the disclosed embodiment are described in a specific order in the above embodiments, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flowchart can change the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution.

[0050] Corresponding to the aforementioned embodiment of the method for generating media, the present disclosure also provides an embodiment of an apparatus for generating media.

[0051] like Figure 3 As shown, Figure 3 It is a block diagram of a media generation device according to an exemplary embodiment of the present disclosure. The device may include: an acquisition unit 301, a control unit 302 and a denoising unit 303.

[0052] The acquisition unit 301 is configured to acquire a reference image and a noise image.

[0053] The control unit 302 is configured to obtain control information for guiding the generation of media, wherein the control information includes information of a target subject in a reference image, information of a processing category to which the target subject belongs, and motion guidance information of the target subject.

[0054] The denoising unit 303 is configured to perform denoising processing on the noisy image based on the reference image and the control information using the target model to obtain the target medium, so that the target subject in the target medium moves according to the motion guidance information.

[0055] In some embodiments, the processing categories include a first category processed according to a specified camera movement mode, a specified motion mode, and a random motion mode, a second category processed according to a specified camera movement mode and a random motion mode, and a third category processed according to a specified camera movement mode.

[0056] In other embodiments, the motion guidance information includes at least one of the following: camera movement parameter information for guiding the target subject to move along the camera movement trajectory; specified trajectory information for guiding the target subject to move in a specified manner; random motion intensity information for guiding the target subject to move in a random manner.

[0057] In other embodiments, during the application stage of the target model, the control unit is configured to: determine a first operation and a second operation performed by a user on a reference image; based on the first operation, determine information about the target subject and a processing category to which the target subject belongs; and based on the second operation, determine motion guidance information about the target subject.

[0058] In some other implementations, during the training phase of the target model, before acquiring the reference image, it further includes: a sample acquisition unit configured to acquire sample media.

[0059] The acquisition unit is configured to: acquire a first frame image of the sample media as a reference image.

[0060] The control unit is configured to: perform semantic segmentation on a reference image and determine a plurality of target subjects based on the result of the semantic segmentation; set information on processing categories to which the plurality of target subjects belong; and determine motion guidance information of the target subject based on image frames following the reference image in the sample media and information on the processing categories to which the target subject belongs.

[0061] In other embodiments, motion guidance information of the target subject is determined based on the image frame after the reference image in the sample video and the processing category to which the target subject belongs in the following manner: trajectory data corresponding to the target subject is determined based on the image frame after the reference image in the sample video; trajectory function corresponding to the target subject is determined based on the processing category to which the target subject belongs; motion guidance information of the target subject is determined based on the trajectory data and the trajectory function.

[0062] In other embodiments, the trajectory function includes a specified motion function term and a random motion function term; and at least one function term in the trajectory functions corresponding to target subjects belonging to different processing categories is different.

[0063] In other embodiments, the denoising unit is configured to: obtain a control tensor based on control information; the control tensor includes a first tensor for representing the motion trajectory, a second tensor for representing the random motion intensity, a third tensor for representing the target subject identification and a fourth tensor for representing the processing category to which the target subject belongs; input the control tensor into the target adapter to obtain the target result output by the target adapter; use the reference image and the target result to guide the target model to denoise the noisy image.

[0064] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment scheme of the present disclosure. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0065] Reference below Figure 4 , Figure 4 A schematic block diagram of an electronic device provided for some embodiments of the present disclosure. The electronic device 920 is suitable for implementing the media generation method provided in the embodiments of the present disclosure, for example. The electronic device 920 may be a terminal device, etc., and may be used to implement a client or a server. The electronic device 920 may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that Figure 4 The electronic device 920 shown is only an example and does not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0066] like Figure 4 As shown, the electronic device 920 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 921, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 922 or a program loaded from a storage device 928 into a random access memory (RAM) 923. In the RAM 923, various programs and data required for the operation of the electronic device 920 are also stored. The processing device 921, the ROM 922, and the RAM 923 are connected to each other via a bus 924. An input / output (I / O) interface 925 is also connected to the bus 924.

[0067] Typically, the following devices may be connected to the I / O interface 925: an input device 926 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 927 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 928 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 929. The communication device 929 may allow the electronic device 920 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 920 is shown to have various devices, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 920 may alternatively implement or possess more or fewer devices. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0068] According to an embodiment of the present disclosure, the method for generating the above-mentioned media can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method for generating the above-mentioned media. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 929, or installed from the storage device 928, or installed from the ROM 922. When the computer program is executed by the processing device 921, the functions defined in the method for generating the media provided in the embodiment of the present disclosure can be implemented.

[0069] The embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method provided by the present disclosure.

[0070] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0071] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0072] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0073] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present disclosure can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0074] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present disclosure in detail. It should be understood that the above description is only the specific implementation method of the embodiments of the present disclosure, and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present disclosure shall be included in the scope of protection of the present invention.

Claims

1. A method for generating a medium, the method comprising: Obtain a reference image and a noise image; Acquire control information for guiding the generation of media, the control information including information of a target subject in the reference image, information of a processing category to which the target subject belongs, and motion guidance information of the target subject; The noise image is denoised using a target model based on the reference image and the control information to obtain a target medium, so that the target subject in the target medium moves according to the motion guidance information.

2. The method according to claim 1, wherein: The processing categories include a first category processed according to a specified camera movement mode, a specified motion mode and a random motion mode, a second category processed according to a specified camera movement mode and a random motion mode and a third category processed according to a specified camera movement mode.

3. The method according to claim 1, wherein: The exercise guidance information includes at least one of the following: Camera movement parameter information for guiding the target subject to move along the camera movement trajectory; Specified trajectory information for guiding the target subject to move in a specified manner; Random motion intensity information for directing the target subject to move in a random manner.

4. The method according to claim 1, wherein: In the application phase of the target model, the acquiring of control information for guiding the generation of media includes: determining a first operation and a second operation performed by a user on the reference image; Based on the first operation, determining information about the target subject and the processing category to which the target subject belongs; Based on the second operation, motion guidance information of the target subject is determined.

5. The method according to claim 1, wherein: In the training phase of the target model, before acquiring the reference image, the method further includes: acquiring sample media; Wherein, obtaining a reference image includes: Acquire the first frame image of the sample media as the reference image; The obtaining of control information for guiding the generation of media includes: Performing semantic segmentation on the reference image, and determining a plurality of target subjects based on the result of the semantic segmentation; Setting information of the processing categories to which the multiple target entities belong; Based on the image frame following the reference image in the sample media and information about the processing category to which the target subject belongs, the motion guidance information of the target subject is determined.

6. The method according to claim 5, wherein: The determining the motion guidance information of the target subject based on the image frame after the reference image in the sample video and the processing category to which the target subject belongs includes: Determining trajectory data corresponding to the target subject according to an image frame following the reference image in the sample video; Determining a trajectory function corresponding to the target subject according to the processing category to which the target subject belongs; Based on the trajectory data and the trajectory function, motion guidance information of the target subject is determined.

7. The method according to claim 6, wherein: The trajectory function includes a specified motion function term and a random motion function term; at least one function term in the trajectory functions corresponding to target subjects belonging to different processing categories is different.

8. The method according to claim 1, wherein: The method of using the target model to perform denoising on the noisy image based on the reference image and the control information includes: A control tensor is obtained based on the control information; the control tensor includes a first tensor for representing a motion trajectory, a second tensor for representing a random motion intensity, a third tensor for representing a target subject identifier, and a fourth tensor for representing a processing category to which the target subject belongs; Inputting the control tensor into a target adapter to obtain a target result output by the target adapter; The target model is guided to perform denoising on the noisy image using the reference image and the target result.

9. A device for generating media, the device comprising: An acquisition unit, configured to acquire a reference image and a noise image; A control unit configured to obtain control information for guiding the generation of media, wherein the control information includes information of a target subject in the reference image, information of a processing category to which the target subject belongs, and movement guidance information of the target subject; The denoising unit is configured to use a target model to denoise the noise image based on the reference image and the control information to obtain a target medium, so that the target subject in the target medium moves according to the motion guidance information.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

12. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 8 is implemented.

Images