Method and apparatus for generating video

By using audio or audio processing effects as reference material to generate videos, the cumbersome sound editing problem in existing technologies is solved, and a significant improvement in video creation efficiency is achieved.

CN123123668APending Publication Date: 2026-11-17BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611239901.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, when users are not satisfied with the sound in a video, they need to re-edit the audio in post-production, which is cumbersome and inefficient.

Method used

An interactive method is provided that allows users to add audio or audio processing effects as reference materials to the video generation area, and input relevant information to generate a video, using audio-related materials as reference materials to generate a matching video.

Benefits of technology

It significantly improves video creation efficiency and meets users' creative needs in the audio dimension.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123668A_ABST
    Figure CN123123668A_ABST
Patent Text Reader

Abstract

An embodiment provides a method and apparatus for generating video. The method includes displaying a second region in response to a first operation on a first region, the second region being used to provide audio or audio processing effects. The method further includes displaying a first component corresponding to first material in the first region in response to a second operation on the second region, the first material indicating first audio or first audio processing effect. The method further includes displaying first information in the first region in response to a third operation on the first region, the first information indicating that a video is generated using the first material as reference material, the content of the video being associated with the first information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The technical field generally relates to the field of computers, and more specifically to methods and apparatus for generating video. Background Technology

[0002] With the continuous development of information technology and the widespread use of mobile terminals, video has become an important medium for information transmission and communication. Video creation methods based on natural language processing and generative multimodal models have been widely applied.

[0003] In related technologies, users can input descriptive text and reference images of video footage in an application to express their creative intent. The application then generates a video that matches the user's creative intent based on the descriptive text and reference images. Summary of the Invention

[0004] In an example embodiment, a method, apparatus, device, medium, and product for generating video are provided.

[0005] In a first aspect, a method for generating a video is provided. The method includes displaying a second region in response to a first operation on a first region, the second region being used to provide audio or audio processing effects. The method also includes displaying a first component corresponding to first material in the first region in response to a second operation on the second region, the first material indicating first audio or first audio processing effect. The method further includes displaying first information in the first region in response to a third operation on the first region, the first information indicating that a video is generated using the first material as reference material, the content of the video being associated with the first information. This method generates video by using audio-related material as reference material, enabling the generation of video to reference relevant features of audio or audio processing effects, generating a video that matches audio-related features, satisfying users' creative needs regarding the sound dimension during video creation, and significantly improving video creation efficiency.

[0006] In a second aspect, an apparatus for generating video is provided, comprising a first display module configured to display a second area in response to a first operation on a first area, the second area being used to provide audio or audio processing effects. A second display module is configured to display a first component corresponding to first material in the first area in response to a second operation on the second area, the first material indicating first audio or first audio processing effect. A third display module is configured to display first information in the first area in response to a third operation on the first area, the first information indicating that a video is generated using the first material as reference material, the content of the video being associated with the first information. This apparatus generates video by using audio-related material as reference material, enabling the generation of video to reference relevant features of audio or audio processing effects, generating a video that matches audio-related features, satisfying users' creative needs regarding the sound dimension during video creation, and significantly improving video creation efficiency.

[0007] In a third aspect, an electronic device is provided, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform a method for generating video when executed by the at least one processing unit. The method includes displaying a second region in response to a first operation on a first region, the second region being used to provide audio or audio processing effects. The method further includes displaying a first component corresponding to first material in the first region in response to a second operation on the second region, the first material indicating first audio or first audio processing effect. The method further includes displaying first information in the first region in response to a third operation on the first region, the first information indicating that a video is generated using the first material as reference material, the content of the video being associated with the first information. This method generates video by using audio-related material as reference material, enabling the generation of video to reference relevant features of audio or audio processing effects, generating a video that matches audio-related features, meeting users' creative needs regarding the sound dimension during video creation, and significantly improving video creation efficiency.

[0008] In a fourth aspect, a computer-readable storage medium is provided having machine-executable instructions stored thereon, which, when executed by a device, cause the device to perform a method for generating video. The method includes displaying a second region in response to a first operation on a first region, the second region being used to provide audio or audio processing effects. The method also includes displaying a first component corresponding to first material in the first region in response to a second operation on the second region, the first material indicating first audio or first audio processing effect. The method further includes displaying first information in the first region in response to a third operation on the first region, the first information indicating that a video is generated using the first material as reference material, the content of the video being associated with the first information. This method generates video by using audio-related material as reference material, enabling the generation of video to reference relevant features of audio or audio processing effects, generating a video that matches audio-related features, meeting users' creative needs regarding the sound dimension during video creation, and significantly improving video creation efficiency.

[0009] In a fifth aspect, a computer program product is provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement a method for generating video. The method includes displaying a second region in response to a first operation on a first region, the second region being used to provide audio or audio processing effects. The method also includes displaying a first component corresponding to first material in the first region in response to a second operation on the second region, the first material indicating first audio or first audio processing effect. The method further includes displaying first information in the first region in response to a third operation on the first region, the first information indicating that a video is generated using the first material as reference material, the content of the video being associated with the first information. This method generates video by using audio-related material as reference material, enabling the generation of video to reference relevant features of audio or audio processing effects, generating a video that matches audio-related features, meeting users' creative needs regarding the sound dimension during video creation, and significantly improving video creation efficiency.

[0010] The summary section is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features, nor is it intended to limit the scope. Other features will become readily apparent from the following description. Attached Figure Description

[0011] Figure 1 A schematic diagram of an example environment in which the embodiments can be implemented is shown; Figures 2A to 2D A schematic diagram showing a first region is shown; Figure 3A flowchart of a method 300 for generating video is shown; Figures 4A to 4E A schematic diagram showing a second region is shown; Figure 5 Another schematic diagram showing the third area is shown; Figure 6A and Figure 6B A schematic diagram showing a third region is shown; Figure 7 A schematic diagram showing an audio identifier is provided. Figure 8 A schematic diagram showing a first region is shown; Figure 9A and Figure 9B A schematic diagram showing a fourth region is shown; Figure 10A and Figure 10B A schematic diagram showing a prompt message is provided. Figure 11 Another schematic diagram showing the first area is shown; Figure 12 A schematic block diagram of an example device according to some embodiments is shown; and Figure 13 A block diagram of an example device that can be used to implement the embodiments is shown.

[0012] In all the accompanying figures, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0013] The names of messages or information exchanged between multiple devices in the embodiments are for illustrative purposes only and are not intended to limit the scope of these messages or information. It is understood that before using the technical solutions disclosed in the embodiments, users should be informed of the type, scope of use, and usage scenarios of the personal information involved, and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0014] For example, upon receiving a user's active request, a prompt message can be sent to the user, explicitly informing them that the requested operation will require the acquisition and use of their personal information. This allows the user to choose whether to provide personal information to the software or hardware (electronic device, application, server, or storage medium) performing the operations described herein, based on the prompt message. As an optional but non-limiting implementation, the prompt message can be sent to the user in the form of a pop-up window, which can present the prompt message in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0015] It is understood that the above notification and user authorization process is merely illustrative and does not constitute a limitation on the implementation method. Other methods that comply with relevant laws and regulations may also be applied to the embodiments.

[0016] The various embodiments will now be described in more detail with reference to the accompanying drawings. While some embodiments are shown in the drawings, it should be understood that implementations can be achieved in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the technical content of this document. It should be understood that the drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection herein.

[0017] In the description of the embodiments, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0018] In relevant technical solutions, video creation methods based on natural language processing and generative multimodal models have been widely used. For example, users can input descriptive text and reference images to express their creative intent in an application, and the application can then generate a video that matches the user's creative intent based on the descriptive text and reference images. However, if the user is not satisfied with the sound in the video (such as background music, narration, etc.) after it is generated in this way, the only way to edit the video is to re-dub it in post-production, which is cumbersome and inefficient.

[0019] To address this, a method for generating videos is provided, offering an interactive approach that uses audio-related materials as reference material. This allows users to express their video creation needs through audio or audio processing effects. In this method, users can add audio-related materials (such as audio or audio processing effects) to the area used for video generation and input information (such as descriptive text) in this area to instruct the application to generate the video using the audio-related materials as reference. By using audio-related materials as reference material to generate videos, the application can use the relevant features of the audio or audio processing effects as a reference to generate videos that match the audio-related features, satisfying users' creative needs regarding the sound dimension during video creation and significantly improving video creation efficiency.

[0020] The embodiments will now be described in further detail with reference to the accompanying drawings, wherein... Figure 1A schematic diagram of an example environment 100 in which embodiments can be implemented is shown. This example environment 100 includes a user device on which applications 110 (such as video generation applications, social applications supporting video generation, media applications supporting video generation, media content editing applications supporting video generation, media content generation applications, etc., not limited to these) can be installed. Application 110 serves as the core execution carrier of the methods in this application embodiment, undertaking tasks such as receiving multimodal input (such as text, audio, images, or video), video generation, video editing, audio editing, display, and user interaction responses. In some embodiments, the user device is communicatively connected to a server, which provides background services for the application 110 running on the user device. For example, the server deploys models (such as machine learning models, generative multimodal models, etc., not limited to these). The application 110 on the user device can communicate with the server to invoke the models on the server to generate videos, receive videos from the server, and display the videos for the user to view. Alternatively, the application 110 can set up an agent that can invoke the models on the server to generate videos. User equipment can be any type of mobile computing device, including mobile computers (e.g., personal digital assistants, laptops, notebooks, tablets, netbooks, etc.), mobile phones (e.g., cellular phones, smartphones, etc.), wearable computing devices (e.g., smartwatches, head-mounted devices, including smart glasses, etc.) or other types of mobile devices. In some embodiments, user equipment can also be a fixed computing device, such as a desktop computer, game console, smart TV, etc.

[0021] like Figure 1As shown, the user device displays the interface of application 110. Users can interact with this interface to perform functions such as video generation, video editing, audio generation, and audio editing, and these are not limited to these. For example, the interface of application 110 includes a first area 111, which users can interact with to express their video creation needs (the first area 111 can also be understood as an interactive area providing video generation functions, or referred to as a generator, video creation area, video generation area, etc., and is not limited to these). For example, users can click the upload control 112 in the first area 111 to add reference materials for video generation, such as images, videos, virtual avatars, etc. Another example is that users can click on a portion of the first area 111 (such as the area below the upload control 112) to input descriptive text through keyboard input or voice input to express their video creation needs. Yet another example is that users can interact with the model-related component 113 in the first area 111 to select a model for video generation. It should be understood that application 110 can be associated with one or more models, and the embodiments of this application do not limit the number and type of models associated with application 110. After the user expresses their video creation needs through the first area 111, they can click the generation control 114 to trigger the application 110 to use the model to generate a video that matches their video creation needs.

[0022] In some embodiments, the interface of application 110 also includes other interactive areas to provide more functions for video generation scenarios. For example, the interface of application 110 also includes a playback area 115, through which users can view videos (such as generated videos, reference videos used to generate videos, etc.). As another example, the interface of application 110 also includes a display area 116 for the generated results, through which users can view the generated video, audio, and other related results. Furthermore, the interface of application 110 also includes an editing area 117, which may include a timeline, editing tracks, function controls, etc. Editing tracks include, but are not limited to: video editing tracks, text editing tracks, audio editing tracks, etc. Function controls include, but are not limited to: splitting, deleting, undoing, recording, etc. Exemplarily, a positioning component on the editing track can indicate the position of the video frame displayed in the playback area 115 within the video. Users can browse the video in the playback area 115 by clicking the corresponding position on the editing track or by dragging the positioning component.

[0023] It should be understood that Figure 1The interface layout of application 110 is shown for illustrative purposes only and does not constitute a limitation of this application. In the interface of application 110, controls, components, or areas may employ other display styles and positional distributions. For example, when application 110 runs on a mobile device with a small screen, such as a smartphone, the above-mentioned... Figure 1 The interface layout shown can be adaptively adjusted according to the device's screen size and interaction method. For example, on a mobile device, the first area 111 can occupy the main display area of ​​the screen, while other areas can be presented through bottom, side slide-out panels, or collapsible menus, thereby achieving complete video generation functionality within a limited screen space. In this way, the video generation method provided in this application embodiment can adapt to various types of user devices, exhibiting good cross-platform applicability and interactive flexibility. It should be understood that the architecture and functionality in the example environment are described for illustrative purposes only and do not imply any limitation on the scope. The embodiments can also be applied to other environments with different structures and / or functionalities.

[0024] In the embodiments of this application, for Figure 1 The video generation scenario shown provides an interactive method for generating videos using audio as reference material, allowing users to express their video creation needs through audio. For example, a user can add audio as reference material to the first area 111 and input information (such as descriptive text) in that area 111 to instruct the model to generate a video using the audio as reference material. See below for reference... Figures 2A to 2D This process will be explained. Figures 2A to 2D A schematic diagram showing the first region is shown.

[0025] like Figure 2AAs shown, taking the example of a user adding reference material via the upload control 112 in the first area 111, application 110, in response to the user's interactive operation on the upload control 112 (e.g., clicking the upload control 112), displays at least one material prompt item 201 to indicate the upload method of the reference material. The reference material can serve as a constraint condition when the model generates video, driving the model to generate a video matching the reference material by referencing its relevant features. For example, the first material prompt item 201-1 (e.g., "Local Device") indicates uploading audio-related material (e.g., audio or audio processing effects) from the local device as reference material. As another example, the second material prompt item 201-2 (e.g., "Add Audio") indicates uploading audio-related material from application 110's audio material library as reference material. Exemplarily, the user can interact with the material prompt item 201 to add audio-related material as reference material to the first area 111. For example, in response to the interactive operation on the first material prompt item 201-1 (e.g., clicking), application 110 displays the audio identifier of the audio on the local device for the user to select. For example, in response to an interactive action (such as clicking) on ​​the second material prompt item 201-2, application 110 displays the audio material library of application 110 for the user to select.

[0026] like Figure 2B As shown, taking the example of a user adding a first material (in the figure, music A is used as an example; the first material can also be an audio processing effect, such as reverb, echo, etc., but is not limited to this) as a reference material to the first area 111 via the upload control 112, the application 110 displays the first component 202 corresponding to the first material in the first area 111. That is, the application 110 indicates that the first material has been added to the first area 111 as a reference material by displaying the first component 202. For example, the first component 202 may include information such as the identifier, duration, or source of the first material, but the embodiments of this application do not limit this.

[0027] like Figure 2CAs shown, when a user adds the first material as a reference material to the first area 111, the user can input information in the first area 111 to instruct the application 110 to generate a video using the first material as a reference material. For example, the application 110 displays the first information 203 in the first area 111. Taking the model associated with component 113 as the first model (such as model 1) as an example, in this scenario, the first area 111 can be understood as an interactive area associated with the first model, used to provide video generation functionality. It should be understood that the first model has the ability to generate a video using audio-related materials as reference materials. Correspondingly, the first information 203 instructs the generation of a video using the first material as a reference material. For example, the first information 203 includes "Generate a video that matches the content of 'Music A'". The embodiments of this application do not limit the content and style of the first information 203. The first information 203 may include text-type information (such as descriptive text), for example, through input operations such as text input and voice input. The first information 203 may also include identification-type information (such as the identification of "Music A"). For example, referencing Figure 2D The first area 111 includes a mention control 204 (such as "@"). For example, when a user clicks the mention control 204, the application 110 displays at least one prompt to indicate a component in the first area 111. When the user clicks the prompt 205 associated with the first component 202 (such as "@MusicA"), identification information 206 (such as the identifier of "MusicA") is entered into the first area 111.

[0028] pass Figures 2A to 2D The interactive method shown allows users to add a first source material as a reference to the first area 111, and input first information 203 in this area 111 to instruct the application 110 to generate a video using the first source material as a reference. This process involves expressing video creation needs through the first area 111. Afterwards, the user can click the generation control 114 to trigger the application 110 to generate a video matching the first information 203, the content of which is associated with the first information 203. In this way, audio-related materials are used as references to drive the model to generate video, enabling the model to generate a matching video based on the relevant features of the audio-related materials. This satisfies the user's creative needs regarding the sound dimension during video creation, significantly improving video creation efficiency. It should be understood that... Figures 2A to 2DThis illustration is for demonstrative purposes only and does not constitute a limitation of this application. In some embodiments, application 110 supports more interactive methods, such as multiple ways of uploading audio materials, uploading multiple types of audio, and uploading multiple audio files as reference materials. In other embodiments, application 110 also supports adding other modal materials such as videos, images, or virtual avatars as reference materials to the first area 111, so that the first model can generate a video that matches the multimodal materials by referencing the relevant features of the multimodal materials, thus meeting the user's creative needs in multiple dimensions during video creation.

[0029] The process of the embodiments will be described in detail below with reference to other accompanying drawings. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and their scope is not limited in this respect.

[0030] Figure 3 A flowchart of a method 300 for generating video is shown. In this embodiment, it can be generated by... Figure 1 The method is performed by application 110 on a user device in an embodiment. At block 302, in response to a first operation on a first region, a second region is displayed, the second region being used to provide audio or audio processing effects. Exemplarily, application 110 indicates its audio library by displaying the second region. Exemplarily, refer to... Figure 2A The first operation for the first area 111 is, for example, the interactive operation for the second material prompt item 201-2 in the first area 111 (such as clicking "Add Audio"). Application 110 also supports triggering the display of the second area in other ways, such as the user triggering application 110 to display the second area by using the voice command "Show Audio Material Library".

[0031] In some embodiments, the second area is used to provide at least one type of material, which includes a first type and a second type. The first type is associated with at least one of the following audio types: recorded audio and audio already published in the application. Audio already published in the application includes, for example, music, narration, voice-over, or sound effects. Sound effects refer to audio materials that can be used for audio-visual editing, such as applause, cheers, ambient noise (such as rain or wind), or transition sound effects, but are not limited to these. It should be understood that music generally refers to audio materials with a coherent melody and rhythm, and may include vocals or instrumental performances. Sound effects do not necessarily have a melody, nor do they necessarily include vocals. In some cases, sound effects may be a set of fixed background sounds (such as continuous rain or machine noise) that lack melody but can be used to present a specific atmosphere. By providing sound effects as an audio material to users, the needs of users for sound expression of specific situations, atmospheres, or action details in video creation can be met, further enriching the diversity of audio reference materials.

[0032] The second type is associated with at least one of the following audio processing effects: processing effects targeting audio acoustic features, and processing effects targeting audio spatiotemporal features. Audio acoustic features include, for example, features such as volume, frequency, timbre, and pitch; audio spatiotemporal features include, for example, features such as rhythm, reverberation, and panning, which are not limited in the embodiments of this application. That is, application 110 supports uploading various types of audio-related materials as reference materials, which can meet users' diverse needs for video creation in the sound dimension.

[0033] In box 304, in response to a second operation on the second area, a first component corresponding to the first material is displayed in the first area. This first material indicates the first audio or a first audio processing effect. The second operation refers to an interactive operation that adds audio-related material provided in the second area as a reference material to the first area. For example, the second operation could be an interactive operation on material information in the second area. Through the second operation, the user can select desired audio-related material from the audio material library of application 110 as a reference material and add it to the first area, triggering application 110 to display the first component corresponding to the first material (such as...) in the first area. Figure 2B The first component 202 shown.

[0034] In box 306, in response to a third operation on the first area, first information is displayed in the first area. The first information indicates that a video will be generated using the first material as reference material, and the content of the video is associated with the first information. The third operation refers to an interactive operation involving inputting information in the first area. For example, the third operation could be a confirmation operation for a prompt, an input operation for text-based information (such as text input or voice input), etc., and is not limited to these. Through the third operation, the user can flexibly express their video creation needs, triggering application 110 to display the first information (such as text input or voice input) in the first area. Figure 2C The first information 203 shown. Based on this, application 110 can respond to interactive operations on the generated control (such as clicking). Figure 2C The generated control 114 shown generates a video that matches the first information.

[0035] In method 300, by using audio-related materials as reference materials to generate videos, the application can generate videos that match the audio-related features by referencing the relevant features of the audio or audio processing effects, thus meeting the user's creative needs regarding the sound dimension when creating videos and greatly improving the efficiency of video creation.

[0036] In some embodiments of method 300, application 110 supports more interaction methods, such as multiple interaction methods for the second area, multiple ways to upload audio materials, uploading multiple types of audio, uploading multiple audio files as reference materials, and adding other modal materials such as videos, images, or virtual avatars as reference materials to the first area 111, etc. See below for reference. Figures 4A to 11 This section will introduce these implementation methods.

[0037] In some embodiments, the second operation on the second region is an interactive operation on the material information in the second operation. For example, application 110 can display material information of audio-related materials in the second region, where the material information includes material type, material attributes, material duration, material name, material identifier, etc., and is not limited thereto. Taking the first material in method 300 as an example, application 110 displays the first material information of the first material in the second region, and in response to the first interactive operation on the first material information, displays the first component corresponding to the first material in the first region. The first material information indicates that the first material is the first audio, or that the first material information indicates that the first material is the first audio processing effect. That is, the user can select multiple types of audio-related materials as reference materials from the audio material library.

[0038] Figures 4A to 4E A schematic diagram showing the second region is illustrated. For example... Figure 4AAs shown, application 110 displays a second area 401 on the interface and displays the material information of the material in the second area 401. For example, the user can click... Figure 2A The second material prompt item 201-2 shown triggers application 110 to display the second area 401. Application 110 also supports triggering the display of the second area 401 through other means, such as the user triggering the display of the second area 401 through the voice command "Show audio material library". Exemplarily, the second area 401 is used to provide audio or audio processing effects. Application 110 can display a tab bar 402 in the second area 401, such as including "All", "Sound Effects", "Music", "Recording", etc., so that the user can quickly view the audio-related materials under the corresponding tab. Taking the first material information 403 of the first material in the second area 401 as an example, in response to the first interactive operation on the first material information 403 (such as clicking the first material information 403 and then clicking the confirmation control 404), application 110 displays the first component 202 corresponding to the first material in the first area 111. In some embodiments, application 110 supports displaying a first prompt message in the second area 401, which indicates that the second operation must meet a first condition. The first condition is associated with at least one of the following: the number of audio-related materials and the total duration of the audio-related materials. For example, application 110 can display the number of currently selected materials and the total duration, as well as the number of materials and the total duration of materials indicated by the first condition, in real time at the bottom of the second area 401, so that the user can make selections within the limitations of application 110.

[0039] In some embodiments, such as Figure 4B As shown, application 110 supports previewing and selecting audio-related materials. For example, in response to an interactive operation on material information 405, application 110 plays the material corresponding to material information 405 (such as sound effect xxx), and selects the material; clicking again deselects it. For example, a playback progress bar is displayed in component 406, allowing users to flexibly adjust the previewed segment. It should be understood that application 110 can play the first material in the same way. For example, in response to a second interactive operation on first material information 403 (such as clicking first material information 403, after which application 110 displays component 406 in the second area 401 to play the first material), application 110 plays the first material.

[0040] In some embodiments, application 110 provides audio recording functionality through a second region 401. For example... Figure 4CAs shown, application 110 displays a first control 407 (such as "Start Recording") in the second area 401, and receives the recorded second audio in response to an interactive operation on the first control 407. For example, the recording process for the second audio is displayed in the audio recording area 408. In some embodiments, application 110 may display a prompt message to indicate the duration of the recorded audio. For example, the duration of the recorded audio is related to the first model currently associated with the first area 111. In some embodiments, application 110 supports verification of the duration of the recorded audio. For example, when the user stops recording, if the recorded audio duration is less than 2 seconds, application 110 does not save the recording file and displays a prompt message "Recording duration cannot be less than 2 seconds". If the recording duration reaches the upper limit (e.g., 15 seconds or 30 seconds, the specific value is related to the currently associated model), application 110 can automatically cut off the recording and save the file, while displaying a prompt message "Recording duration cannot exceed XX seconds". In this way, invalid or excessively long audio can be avoided from consuming computing resources, improving the stability of system processing. In some embodiments, the audio recording area 408 can provide one or more sub-functions related to the audio recording function. Examples include noise reduction, volume adjustment, echo cancellation, intelligent cropping, and more, but are not limited to these.

[0041] In some embodiments, application 110 also provides editing functions for the recorded audio through a second region 401. For example... Figure 4D As shown, taking a second audio file obtained through recording as an example, application 110 displays the second audio source information 409 in the second area 401 (the illustration uses the display of the second source information 409 under the "Recording" tab as an example, and does not constitute a limitation of this application). Exemplarily, in response to an interactive operation on the second source information 409, application 110 edits the second audio file, thereby triggering application 110 to display the edited second source information, which indicates the edited second audio file. For example, if a user clicks the editing control 410 in the second source information 409, application 110 displays an editing area 411 for the second audio file. This editing area 411 can provide one or more editing functions, such as timbre switching, speech rate adjustment, language switching, etc. In this way, a lightweight audio editing method is provided, which can significantly improve the user's freedom in post-processing audio.

[0042] In some embodiments, application 110 provides audio search functionality through a second area 401. Exemplarily, application 110 may display a search component in the second area 401, and in response to interactive operations on the search component, display audio search results in the second area 401. The search component may be a search input box or a search panel; this embodiment of the application does not limit this. Exemplarily, search results may be displayed in a list or grid format, showing matching audio information, including audio name, duration, type tags, etc., facilitating quick location and selection of target audio by the user. Taking a search panel as an example, for instance... Figure 4E As shown, application 110 displays a search control 412 in the second area 401 (the figure uses a search button as an example for illustration and does not constitute a limitation of this application). In response to a user's interactive operation on the search control 412 (such as clicking the search button), application 110 displays a search component 413 in the second area 401. This search component 413 may include a search input box, historical search results, recommended search results, etc. Users can use this search component 413 to search for audio, for example, by entering keywords in the search input box and pressing the Enter key, etc., and is not limited to this. Through these multiple search methods, the limitations of traditional keyword matching are broken, significantly reducing the cognitive and operational costs of material retrieval.

[0043] In some embodiments, application 110 supports multiple search methods to improve the efficiency of finding audio materials. For example, in addition to conventional text keyword matching search, application 110 also supports semantic fuzzy search. For example, if a user enters text (such as "light and cheerful music suitable for travel backgrounds"), application 110 can search based on the semantic features of the text and return audio that matches those semantic features. Application 110 also supports similar audio search. For example, if a user selects an existing audio as a benchmark, application 110 can search based on the audio features of that audio and return other audio similar to it. In addition, application 110 also supports image-based music search. For example, if a user uploads an image (such as a sunset seascape photo), application 110 can search based on the image features of the image and return audio that matches the information presented in the image. Through these multiple search methods, users can quickly and accurately locate reference materials that meet their creative needs from the audio material library, significantly improving material retrieval efficiency.

[0044] In some embodiments, application 110 also supports uploading audio-related materials through the editing area 117 of the interface (hereinafter referred to as the third area for easy distinction from other areas in the interface). That is, users can select audio-related materials as reference materials through the editing track displayed in the third area. For example, application 110 can display a first editing track in the third area, which is associated with a third audio (for example, the user has imported an audio clip as background music during the project editing process, and the audio has been displayed on the editing track). In response to an interactive operation on the first editing track, application 110 displays a second component corresponding to the second material in the first area 111, which indicates the third audio. Figure 5 A schematic diagram showing a third region is illustrated. For example... Figure 5 As shown, application 110 displays a first editing track 501 in editing area 117 (i.e., the third area), which is associated with a third audio (such as music B). Users can interact with this first editing track 501 to add the third audio to the first area 111. For example, a user can right-click on the first editing track 501 and select "Add to Generator" from the menu; or, a user can directly drag and drop the audio block corresponding to the third audio to the first area 111 (this is illustrated in the diagram). In response to these interactions, application 110 displays a second component 502 corresponding to the second material in the first area 111. In other words, for materials on the editing track (such as music), application 110 allows users to directly drag and drop existing audio from the editing track to the first area 111 as reference material. This implementation allows users to easily generate new video content based on the audio on the current editing track during the editing process by simply dragging or clicking, greatly improving the continuity of creation.

[0045] In some embodiments, application 110 also supports recording audio through the editing area 117 of the interface and synchronously displaying the recorded audio in the second area 401, achieving seamless integration of recording and material management. For example, application 110 can display a second control in the third area to implement the audio recording function. The second control is, for example, a "Global Record" button. In response to an interaction with this second control, application 110 receives the recorded fourth audio and displays a second editing track associated with the fourth audio in the third area. Based on this, application 110 can display third material information in the second area 401 to indicate the fourth audio, thereby synchronizing the fourth audio to the audio material library.

[0046] Figure 6A Another schematic diagram showing the third region is shown. (For example...) Figure 6AAs shown, the interface of application 110 includes an editing area 117 (i.e., the third area), which includes an editing track. A second control 601 (e.g., a "Global Recording" button) is displayed in the editing area 117. In response to a user interaction with the second control 601 (e.g., clicking the button), application 110 begins receiving the recorded fourth audio. After recording, application 110 displays a second editing track 602 in the editing area 117, which is associated with the fourth audio (e.g., audio YY in the figure). Based on this, to facilitate user management of the audio from the audio library in subsequent creations, application 110 can automatically display the third material information corresponding to the fourth audio in the second area 401 (e.g., adding a new recording file entry under the "Recording" category), facilitating subsequent reuse by the user. Furthermore, in some embodiments, application 110 also supports adding reference materials from the editing track to the first area 111. Figure 6B As shown, application 110 responds to user interactions with the second editing track 602 (e.g., the user can right-click the second editing track 602 and select "Add to Generator" from the menu; or the user can drag an audio block from the second editing track 602 to the first area 111), displaying the component 603 corresponding to the fourth audio in the first area 111. For example, during dragging, the dragged audio block follows the movement trajectory, and when the audio block enters the range of the first area 111, the audio block changes to component 603 to prompt the user to release it to complete the addition. It should be understood that this process can preserve the original track material in the second editing track 602. In this way, after the user records audio through the global audio recording entry, it is automatically placed on the track. The user can directly drag audio from the editing track to the first area as reference material without repeatedly searching in the audio material library, greatly shortening the operation path.

[0047] In some embodiments, application 110 also supports uploading audio-related materials from a local device. For example, application 110 can display the audio identifier of a fifth audio file on the local device in the interface (e.g., presenting the audio identifier of the audio file on the local device as a file list or icon). In response to a user's interactive operation on the audio identifier corresponding to the fifth audio file, a third component corresponding to the third material is displayed in a first area, the third material indicating the fifth audio file. The embodiments of this application do not limit the display format and interaction method of the audio identifier. Figure 7 A schematic diagram showing an audio identifier is illustrated. (For example...) Figure 7 As shown, application 110 displays a media selection area 701 associated with the local device. This media selection area 701 can be, for example, a file selection window or an audio file list, and can be located anywhere on the application 110 interface without limitation. Users can click... Figure 2AThe first material prompt item 201-1 shown triggers application 110 to display the material selection area 701. Application 110 also supports triggering the display of the material selection area 701 in other ways, such as the user triggering the display of the material selection area 701 by using the voice command "local upload". For example, application 110 displays the audio identifier 702 of the fifth audio in the material selection area 701 (e.g., including the audio icon, file name, file size, duration, etc., but not limited thereto), and in response to the user's interactive operation on the audio identifier 702 (e.g., double-clicking, or directly dragging to the first area 111, but not limited thereto), displays the third component 703 (such as music C) in the first area 111. In this way, application 110 opens up the ability to import audio from the local device at the local import entry, allowing users to make full use of the audio files already stored on the local device as reference material for video generation.

[0048] As can be seen, Application 110 supports multiple ways to upload audio-related materials. This provides a multi-dimensional and all-scenario source of audio-related materials, lowering the barrier for users to obtain reference materials and adapting to the needs of different creative stages. For example, recording audio can meet the needs of users for personalized expression (such as specific narration), the audio material library can meet the need for convenient access, and local import can meet the need for reusing local materials.

[0049] In some embodiments, users can upload multiple audio-related materials as reference materials to express richer video creation needs. For example, after the user has added the first component 202 to the first area 111, the user can continue to add other audio-related materials, so that the first area 111 also includes a fourth component corresponding to the fourth material. The number of fourth materials can be one or more, and correspondingly, the number of fourth components can also be one or more. For any fourth material, the fourth material indicates a sixth audio or a second audio processing effect. The type of the fourth material can be the same as or different from the first material. When the types of the fourth material and the first material are different, for example, the first material indicates a first audio and the fourth material indicates a second audio processing effect, this achieves a unified inspiration across multiple sound dimensions, significantly improving the video generation effect. For example, the user can select multiple audio-related materials from the audio material library at once as reference materials, or they can select them multiple times; this embodiment does not limit this. Correspondingly, the first information entered by the user indicates that the video is generated using both the first and fourth materials as reference materials. This embodiment does not limit the method of adding and displaying multiple audio-related materials.

[0050] Figure 8 A schematic diagram showing the first region is shown. (Example) Figure 8As shown, the first region 111 includes a first component 202 corresponding to a first material (such as music A) and a fourth component 801 corresponding to a fourth material (such as "sound effect B" and "timbre C"). Accordingly, the first information 203 indicates that the first material and the fourth material are used together as reference materials to generate a video. In some embodiments, the application 110 can use different visual styles or labels to distinguish audio with different audio attributes, such as music, sound effects, timbre, and recordings, in the first region 111 (e.g., using different colored badges or icons; the figure is only for illustration and does not constitute a limitation of this application), so that users can quickly identify the type of each material. This mechanism, which supports multiple audio-related materials working together as reference materials, enables the model to integrate the composite features of different audio-related materials to generate video. For example, the model can combine the rhythmic features of a "background music" and the features of a "narration" to generate a matching video. This allows users to create more complex and refined audiovisual content, achieving a unified inspiration across multiple sound dimensions and significantly improving the video generation effect.

[0051] In some embodiments, a user can edit the audio material in the first area 111 to precisely control the available segments of the reference material. Exemplarily, application 110 displays a fourth area in response to an interactive operation on the first component. This fourth area is associated with editing functions for the first material. These editing functions include, for example, cropping. The user can edit the first material through the fourth area. Accordingly, application 110 displays the edited first material in response to an interactive operation on the fourth area. The embodiments of this application do not limit the interface layout, editing function types, or interaction methods of the fourth area. Figure 9A A schematic diagram showing the fourth region is illustrated. For example... Figure 9AAs shown, in response to a user's interactive operation on the first component 202 (e.g., clicking the first component 202, or selecting "Edit" in the menu after clicking the component, etc., but not limited to these), application 110 displays a fourth region 901, which is associated with the editing function for the first material. Exemplarily, the fourth region 901 can be displayed superimposed on the first region 111. For example, the fourth region 901 includes an audio waveform area, a trimming start handle, and a trimming end handle. The user determines the audio range to be retained by dragging the start and end handles, and the trimmed and retained portions in the audio waveform area can be displayed in different styles. After the user edits the first material, application 110 can display the edited first material and update the display style of the first component 202 (e.g., update the total audio duration value marked on the component). In some embodiments, application 110 can display a tooltip near the first component 202 (e.g., a tooltip appears when the mouse hovers over the first component 202), such as "Click to edit and trim audio," reducing the user's learning curve.

[0052] In some embodiments, the fourth region 901 may also be associated with a replacement function for the first material. For example, continuing to refer to Figure 9A The fourth area 901 includes a replacement control 902. After the user clicks this replacement control 902, the application 110 can display the second area 401 for the user to select a new material to replace the currently edited material. After the user selects a new material, the application 110 can remove the first component 202 and replace it with a component corresponding to the new material. The fourth area 901 can also be associated with a deletion function for the first material. For example, see [reference to previous section]. Figure 9A The fourth area 901 includes a delete control 903. After the user clicks the delete control 903, the application 110 can remove the first component 202.

[0053] In some embodiments, where the first region 111 includes multiple components corresponding to audio-related materials, application 110 can provide editing functions for multiple materials through the fourth region 901. Exemplarily, the first region 111 may also include a fifth component associated with a fifth material, and the editing functions of the fourth region 901 are also applicable to this fifth material. (See reference...) Figure 9BWhen the first area 111 includes multiple components corresponding to audio-related materials, application 110, in response to an interaction with the first component 202, displays the fourth area 901, where the user can switch between materials for editing. For example, in response to an interaction with component 904, application 110 switches the content of the fourth area 901 to edit sound effect B. This method achieves a shared editing panel for multiple components, avoiding cluttered interfaces caused by pop-up windows for each component. This unified editing entry point reduces system rendering overhead and the user's visual burden.

[0054] In some embodiments, application 110 can dynamically display relevant prompts based on conditions associated with the model to guide the user in configuring reference materials appropriately. For example, when the user performs a fourth operation (i.e., adding a component corresponding to audio-related materials in the first region 111) and the component corresponding to the audio-related materials in the current first region 111 meets a second condition, application 110 displays a second prompt indicating adjustments to the materials associated with the first region 111. The second condition is associated with the first model and indicates one or more of the following: the number of audio-related materials in the first region reaches a first threshold, or the total duration of audio-related materials in the first region reaches a second threshold. Embodiments of this application do not limit the display format and specific content of the prompts. Figure 10A and Figure 10B A schematic diagram illustrating the display of prompt information is shown. For example... Figure 10A As shown, when a user performs a fourth operation, such as clicking the second material prompt item 201-2 (e.g., "Add Audio") in the first area 111, if the current first area 111 meets the second condition, the application 110 displays the second prompt message 1001 (the prompt box in the figure is used as an example for illustration and does not constitute a limitation of this application). For example, when the number of audio-related materials in the first area 111 reaches the first threshold (e.g., 3), the second prompt message 1001 is "The current model only supports 3 reference audios. Please remove some and try again." Another example is when the total duration of audio-related materials in the first area 111 reaches the second threshold (e.g., 30 seconds), the second prompt message 1001 is "The total duration of audio materials (currently 35 seconds) exceeds the model limit (30 seconds). Please trim the longer audio." Furthermore, the application 110 can also combine the model-related conditions to display corresponding prompt messages in the fourth area 901 to remind the user of the audio trimming duration. For example, the model association condition indicates that the total duration of the audio-related material must be less than or equal to a second threshold (e.g., 30 seconds). Based on this, application 110 can display corresponding text prompts in the fourth area 901, such as "Total duration ≤ 30 seconds," to remind the user to complete the editing of audio materials within the model's capabilities. In this way, it can be ensured that the reference materials conform to the model input specifications.

[0055] In some embodiments, in response to switching from a first model to a second model and the current first region 111 meeting a third condition, application 110 displays a third prompt message, which instructs adjustment of the materials associated with the first region 111. The third condition is associated with the second model and indicates one or more of the following: the number of audio-related materials in the first region reaches a third threshold (which may be different from the first threshold), or the total duration of audio-related materials in the first region reaches a fourth threshold (which may be different from the second threshold). For example, taking model 1 (supporting 10 audio files) as the first model and model 2 (supporting 3 audio files) as the second model, when the user switches from model 1 to model 2 and has already uploaded 10 audio files, application 110 may display the third prompt message: "Switching to model 2 only supports 3 reference audio files. 10 have already been selected. Confirm discarding the excess 7 audio files?". After user confirmation, application 110 can delete excess audio, retaining either the first three or the last three added audio (depending on the preset strategy, which is not limited), ensuring that the number of audio-related materials in the first region 111 meets the input requirements of the second model. In this way, application 110 can adapt to heterogeneous input constraints between different models, ensuring that the video generation task can be submitted normally after switching models.

[0056] In some embodiments, application 110 can also dynamically switch the prompt information according to the model switching. For example, refer to Figure 10B When the model associated with the first region 111 is the first model (e.g., model 1), application 110 can display a fourth prompt message 1002, "x audio files can be uploaded...", to remind the user if no materials have been uploaded. When the user switches the first model to the second model (e.g., model 2) via component 113, application 110 can display a fifth prompt message 1003, "y audio files can be uploaded." This effectively avoids generation failures caused by reference materials not meeting the model input conditions, improving the success rate of video generation and the user experience.

[0057] In some embodiments, a user can instruct a model to generate video by inputting various types of information through a first area 111. In this scenario, a third operation (i.e., the operation of inputting information in the first area) may include an interactive operation on a first prompt item, and correspondingly, the first information may include second information associated with the first prompt item. Exemplarily, application 110 responds to a third control (such as...) in the first area 111. Figure 2D The interactive operation of the mentioned control 204 (shown) displays at least one prompt item, which is associated with a component in the first area 111 (e.g., including a first prompt item associated with a first component, such as...). Figure 2DThe prompt item 205 is shown. In response to an interactive action on the first prompt item, application 110 displays second information (such as...) in the first area 111. Figure 2D The identification information 206 shown. The second information may have a specific visual style to distinguish it from ordinary text entered by the user. In some embodiments, the third operation may further include an input operation (e.g., text input or voice input) targeting the first area 111, and accordingly, the first information may include the third information associated with the input operation. For example, in response to the input operation, application 110 displays the third information (e.g., ...) in the first area 111. Figure 2D (The text information shown). Using the above methods, users can quickly reference added materials by clicking on prompts, and combine this with text input to efficiently and accurately express their video creation needs.

[0058] In some embodiments, users can add other modal materials such as videos, images, or virtual avatars as reference materials to the first area 111, achieving multimodal reference. Exemplarily, the first area 111 may also include one or more of the following components: a sixth component corresponding to the sixth material (for indicating video-related materials), a seventh component corresponding to the seventh material (for indicating image-related materials), or an eighth component corresponding to the eighth material (for indicating virtual object-related materials). Correspondingly, the aforementioned at least one prompt item triggered by the third control may also include one or more of the following prompt items: a second prompt item associated with the sixth component, a third prompt item associated with the seventh component, or a fourth prompt item associated with the eighth component. The embodiments of this application do not limit the type of multimodal components, the method of adding them, or the display format of the prompt items. Figure 11 For example, let's illustrate this. Figure 11 Another schematic diagram showing the first region is shown. (See diagram below.) Figure 11 As shown, the first area 111 not only includes the first component 202 corresponding to the audio-related materials, but may also include the sixth component 1101 corresponding to the sixth material (e.g., a reference video added by the user). It should be understood that the method of adding the sixth component is similar to the method of adding the component corresponding to the audio-related materials; for example, the user can add it through... Figure 1The upload control 112 shown allows users to select different modalities for uploading. Correspondingly, when a user clicks a third control (such as mention control 204), the application 110 displays at least one prompt item associated with a second prompt item (e.g., "@Video B") in the sixth component 1101. Users can click these prompt items sequentially to generate structured multimodal information in the input box, thereby constructing first information 203, such as "Generate a video based on the footage in 'Video B' and add music similar to 'Music A'". In some scenarios, the application 110 can display a sixth prompt information to indicate the types of materials that can be uploaded when no materials are added in the first area 111. For example, a bubble prompt "Add image, video, or audio" can be displayed near the upload control 112 to clearly inform the user of the currently supported material types. In this way, the application 110 can generate videos that match the multimodal materials by referencing their relevant characteristics, fully satisfying users' comprehensive creative needs regarding visuals, sound, and character portrayal during video creation.

[0059] Figure 12 A schematic block diagram of an example device 1200 according to some embodiments is shown. Device 1200 can be implemented by software, hardware, or a combination of both. Figure 12 As shown, the device 1200 includes a first display module 1201, a second display module 1202, and a third display module 1203. The first display module 1201 is configured to display a second area in response to a first operation on a first area, the second area being used to provide audio or audio processing effects. The second display module 1202 is configured to display a first component corresponding to a first material in the first area in response to a second operation on the second area, the first material indicating first audio or first audio processing effect. The third display module 1203 is configured to display first information in the first area in response to a third operation on the first area, the first information indicating that a video is generated using the first material as reference material, the content of the video being associated with the first information.

[0060] In some embodiments, the second region is used to provide at least one type of material, the at least one type including a first type and a second type; the first type is associated with at least one of the following audio: audio obtained through recording, audio already published in the application; and the second type is associated with at least one of the following audio processing effects: processing effects for audio acoustic features, processing effects for audio spatiotemporal features.

[0061] In some embodiments, the second area includes first material information, which indicates first material, and the second operation includes a first interactive operation on the first material information.

[0062] In some embodiments, the device further includes a playback module configured to play the first material in response to a second interactive operation on the first material information.

[0063] In some embodiments, the device further includes a first information display module configured to display a first prompt message in a second area, the first prompt message indicating that a second operation must meet a first condition, the first condition being associated with at least one of the following: the number of audio-related materials and the total duration of the audio-related materials.

[0064] In some embodiments, the device further includes a first receiving module configured to receive recorded second audio in response to an interactive operation on a first control in a second area.

[0065] In some embodiments, the device further includes a fourth display module configured to display second material information in a second area, the second material information indicating a second audio; and to display edited second material information in response to an interactive operation on the second material information, the edited second material information indicating an edited second audio.

[0066] In some embodiments, the device further includes a fifth display module configured to display a first editing track in a third region, the first editing track being associated with a third audio; and to display a second component corresponding to a second material in the first region in response to an interactive operation on the first editing track, the second material indicating the third audio.

[0067] In some embodiments, the device further includes a second receiving module configured to receive a recorded fourth audio in response to an interactive operation of a second control in a third region; and a sixth display module configured to display a second editing track in the third region, the second editing track being associated with the fourth audio; wherein the second region is also used to display third material information indicating the fourth audio.

[0068] In some embodiments, the device further includes a seventh display module configured to display an audio identifier of a fifth audio in a local device; and in response to an interactive operation on the audio identifier, to display a third component corresponding to a third material in a first area, the third material indicating the fifth audio.

[0069] In some embodiments, the first region further includes a fourth component corresponding to the fourth material, the fourth material indicating a sixth audio or second audio processing effect; and the first information indicating that a video is generated using the first material and the fourth material as reference materials.

[0070] In some embodiments, the device further includes an eighth display module configured to display a fourth area associated with an editing function for the first material in response to an interactive operation on the first component; and to display the edited first material in response to an interactive operation on the fourth area.

[0071] In some embodiments, the fourth region is also associated with a replacement function for the first material.

[0072] In some embodiments, the first region further includes a fifth component corresponding to the fifth material, and the fourth region is also associated with editing functions for the fifth material.

[0073] In some embodiments, the device further includes a second information display module configured to display a second prompt message in response to a fourth operation and when the first region meets a second condition. The second prompt message indicates that the material associated with the first region should be adjusted. The fourth operation indicates that audio-associated material should be added to the first region. The second condition is associated with a first model and indicates that one or more of the following conditions are met: the number of audio-associated materials in the first region reaches a first threshold, or the total duration of audio-associated materials in the first region reaches a second threshold.

[0074] In some embodiments, the device further includes a third information display module configured to display a third prompt message in response to a first model switching to a second model and a first region meeting a third condition. The third prompt message indicates that the material associated with the first region should be adjusted. The third condition is associated with the second model and indicates one or more of the following: the number of audio-associated materials in the first region reaches a third threshold, or the total duration of audio-associated materials in the first region reaches a fourth threshold.

[0075] In some embodiments, the device further includes a prompt display module configured to display at least one prompt in response to an interaction with a third control in a first area, the at least one prompt being associated with a component in the first area, the at least one prompt including a first prompt associated with a first component, and the third operation including an interaction with the first prompt; a third display module 1203 configured to display second information in the first area in response to an interaction with the first prompt, the first information including the second information and the second information being associated with the first prompt.

[0076] In some embodiments, the third operation further includes an input operation for the first region, and the third display module 1203 is configured to display third information in the first region in response to the input operation, the first information including the third information and the third information being associated with the input operation, the input operation indicating one or more of text input and voice input.

[0077] In some embodiments, the first region further includes one or more of the following components: a sixth component corresponding to the sixth material, a seventh component corresponding to the seventh material, or an eighth component corresponding to the eighth material; wherein the sixth material indicates a video, the seventh material indicates an image, and the eighth material indicates a virtual object; at least one prompt item further includes one or more of the following prompt items: a second prompt item associated with the sixth component, a third prompt item associated with the seventh component, or a fourth prompt item associated with the eighth component.

[0078] In some embodiments, the device further includes a search module configured to display a search component in a second area; and to display audio search results in the second area in response to an interactive operation on the search component.

[0079] This device provides an interactive method for generating videos using audio-related materials as reference, allowing users to express their video creation needs through audio or audio processing effects. Users can add audio-related materials (such as audio or audio processing effects) to the area used for video generation and input information (such as descriptive text) in this area to instruct the application to generate the video using the audio-related materials as reference. This method of generating videos using audio-related materials as reference allows the application to generate videos that match the relevant characteristics of the audio or audio processing effects, satisfying users' creative needs regarding the sound dimension during video creation and significantly improving video creation efficiency.

[0080] The division of modules or units in the embodiments of this article is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. In addition, the functional units in the embodiments of this article can be integrated into one unit, exist as separate physical entities, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0081] Figure 13 A block diagram of an example device 1300 that can be used to implement the embodiments is shown. It should be understood that... Figure 13 The device 1300 shown is merely an example and should not be construed as limiting the functionality and scope of the implementation described herein. For example, device 1300 may correspond to the implementation described herein. Figure 1 The user equipment described above can be used to perform the above-described... Figures 1 to 11 The process. For example, device 1300 may correspond to the electronic device of the third aspect of the invention.

[0082] like Figure 13As shown, device 1300 is in the form of a general-purpose computing device. Components of device 1300 may include, but are not limited to, one or more processors or processing units 1310, memory 1320, storage device 1330, one or more communication units 1340, one or more input devices 1350, and one or more output devices 1360. Processing unit 1310 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1320. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of device 1300.

[0083] Device 1300 typically includes multiple computer storage media. Such media can be any available media accessible to device 1300, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1320 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 1330 can be removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within device 1300.

[0084] Device 1300 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 13 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1320 may include computer program product 1325 having one or more program modules configured to perform various methods or actions in various implementations.

[0085] Communication unit 1340 enables communication with other computing devices via a communication medium. Additionally, the functionality of components of device 1300 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, device 1300 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0086] Input device 1350 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1360 can be one or more output devices, such as a monitor, speaker, printer, etc. Device 1300 can also communicate as needed with one or more external devices (not shown) via communication unit 1340. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with device 1300, or with any device that enables device 1300 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0087] Furthermore, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the method described above. This computer-readable storage medium may be a non-transitory computer-readable storage medium. Furthermore, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the method described above. Additionally, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the method described above.

[0088] Various aspects of the situation are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to various embodiments. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0090] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0091] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products based on multiple implementations. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0092] The implementations have been described above. The foregoing descriptions are illustrative and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating video, comprising: In response to a first operation on the first region, a second region is displayed, the second region being used to provide audio or audio processing effects; In response to a second operation targeting the second region, a first component corresponding to a first material is displayed in the first region, the first material indicating a first audio or a first audio processing effect; as well as In response to a third operation targeting the first region, first information is displayed in the first region, the first information indicating that a video is generated using the first material as reference material, the content of the video being associated with the first information.

2. The method of claim 1, wherein the second region is used to provide at least one type of material, the at least one type including a first type and a second type; The first type is associated with at least one of the following audio types: audio obtained through recording, audio already published in the application; and The second type is associated with at least one of the following audio processing effects: processing effects targeting audio acoustic features, and processing effects targeting audio spatiotemporal features.

3. The method according to claim 1, wherein the second area includes first material information, the first material information indicating the first material, and the second operation includes a first interactive operation on the first material information.

4. The method according to claim 3, further comprising: In response to a second interactive operation on the first material information, the first material is played.

5. The method according to claim 3, further comprising: A first prompt message is displayed in the second area, indicating that the second operation must meet a first condition, which is associated with at least one of the following: the number of audio-related materials and the total duration of the audio-related materials.

6. The method according to claim 1, further comprising: In response to an interactive operation on the first control in the second area, the recorded second audio is received.

7. The method of claim 6, further comprising: The second material information is displayed in the second area, and the second material information indicates the second audio. as well as In response to an interactive operation on the second material information, the edited second material information is displayed, which indicates the edited second audio.

8. The method according to claim 1, further comprising: The first editing track is displayed in the third area, and the first editing track is associated with the third audio. as well as In response to an interactive operation on the first edit track, a second component corresponding to the second material is displayed in the first area, the second material indicating the third audio.

9. The method according to claim 8, further comprising: In response to an interactive operation with the second control in the third area, a recorded fourth audio signal is received; A second editing track is displayed in the third area, and the second editing track is associated with the fourth audio. The second area is also used to display third material information, which indicates the fourth audio.

10. The method of claim 1, further comprising: Displays the audio identifier of the fifth audio source in the local device; as well as In response to an interactive operation on the audio identifier, a third component corresponding to the third material is displayed in the first area, the third material indicating the fifth audio.

11. The method of claim 1, wherein the first region further includes a fourth component corresponding to a fourth material, the fourth material indicating a sixth audio or second audio processing effect; and The first information indicates that the video is generated using the first material and the fourth material as reference materials.

12. The method according to claim 1, further comprising: In response to an interactive operation on the first component, a fourth area is displayed, the fourth area being associated with editing functions for the first material; as well as In response to an interactive operation targeting the fourth region, the edited first material is displayed.

13. The method of claim 12, wherein the fourth region is further associated with a replacement function for the first material.

14. The method of claim 12, wherein the first region further includes a fifth component corresponding to the fifth material, and the fourth region is further associated with an editing function for the fifth material.

15. The method according to claim 1, further comprising: In response to the fourth operation and the first area meets the second condition, a second prompt message is displayed, which instructs that the material associated with the first area be adjusted. The fourth operation instructs the addition of audio-related material to the first region, and the second condition is associated with the first model. The second condition indicates one or more of the following: the number of audio-related materials in the first region reaches a first threshold, or the total duration of audio-related materials in the first region reaches a second threshold.

16. The method of claim 15, further comprising: In response to the first model switching to the second model and the first region meeting the third condition, a third prompt message is displayed, which indicates that the material associated with the first region should be adjusted. The third condition is associated with the second model and indicates one or more of the following: the number of audio-related materials in the first region reaches a third threshold, or the total duration of audio-related materials in the first region reaches a fourth threshold.

17. The method according to claim 1, further comprising: In response to an interaction with a third control in the first area, at least one prompt is displayed, the at least one prompt being associated with a component in the first area, the at least one prompt including a first prompt associated with the first component, and the third operation including an interaction with the first prompt. The step of displaying first information in the first area in response to a third operation on the first area includes: displaying second information in the first area in response to an interaction operation on the first prompt item, wherein the first information includes the second information and the second information is associated with the first prompt item.

18. The method of claim 17, wherein the third operation further includes an input operation for the first region, and the display of first information in the first region in response to the third operation for the first region further includes: In response to the input operation, third information is displayed in the first area, the first information including the third information and the third information being associated with the input operation, the input operation indicating one or more of text input and voice input.

19. The method of claim 17, wherein the first region further comprises one or more of the following components: a sixth component corresponding to the sixth material, a seventh component corresponding to the seventh material, or an eighth component corresponding to the eighth material; wherein the sixth material indicates a video, the seventh material indicates an image, and the eighth material indicates a virtual object; The at least one prompt also includes one or more of the following prompts: a second prompt associated with the sixth component, a third prompt associated with the seventh component, or a fourth prompt associated with the eighth component.

20. An apparatus for generating video, comprising: A first display module is configured to display a second area in response to a first operation on a first area, the second area being used to provide audio or audio processing effects; as well as The second display module is configured to display a first component corresponding to a first material in the first area in response to a second operation on the second area, wherein the first material indicates a first audio or a first audio processing effect. as well as A third display module is configured to display first information in the first area in response to a third operation on the first area, the first information indicating that a video is generated using the first material as a reference material, and the content of the video is associated with the first information.