Method, device, equipment, storage medium and product for interface interaction

Abstract

A method, apparatus, device, storage medium and product for interface interaction are provided. The method includes presenting a first input component, the first input component being associated with a first media, the first media including a first image frame; receiving a first input through the first input component; and presenting a second media, the second media including the first image frame and a first audio, the first audio being generated based on the first input and the first media. In this way, the efficiency of human-computer interaction can be effectively improved.

Description

Technical Field

[0001] The examples in this article generally relate to the field of computer science, and in particular to methods, apparatuses, devices, computer-readable storage media, and products for user interface interaction. Background Technology

[0002] With the development of digital media technology, editing software is widely used in the creation and processing of video, audio, and other media content. Editing software typically provides functions such as importing materials, trimming clips, arranging timelines, adjusting visuals, processing audio, adding special effects, and exporting content. Users can combine and edit multiple media materials through interactive interfaces such as timelines, canvases, or nodes, and adjust the presentation of the media content according to creative needs, thereby generating corresponding media works. Summary of the Invention

[0003] In a first aspect, a method for user interface interaction is provided. The method includes: presenting a first input component associated with a first medium, the first medium including a first image frame; receiving a first input through the first input component; and presenting a second medium including the first image frame and first audio, the first audio being generated based on the first input and the first medium.

[0004] In a second aspect, an apparatus for interface interaction is provided. The apparatus includes: a first presentation module configured to present a first input component associated with a first medium, the first medium including a first image frame; a first receiving module configured to receive a first input through the first input component; and a second presentation module configured to present a second medium including the first image frame and first audio, the first audio being generated based on the first input and the first medium.

[0005] In a third aspect, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] Through the above scheme, the first input component is associated with a first medium including a first image frame, enabling the user to provide a first input to the first medium through the first input component and obtain a second medium including the first image frame and the first audio. This reduces the operations required to process the image content and audio separately and to recombine them, thereby improving the efficiency of human-computer interaction. Furthermore, since the first audio is generated based on the first input and the first medium, the audio generation process can be simultaneously constrained by the user's creative requirements and the first medium, thereby reducing the possibility of mismatch between the first audio and the image content and improving the matching degree between the image content and the first audio in the second medium.

[0009] It should be understood that the content described in this section is not intended to limit the key or important features of the examples in this article, nor is it intended to restrict the scope of the solution. Other features will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the various examples herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of the example environment is shown; Figures 2A to 2G Example interfaces for some scenarios are shown; Figure 3 The flowcharts show example processes of interface interactions in some scenarios; Figure 4 Schematic block diagrams of example devices for interface interaction in some scenarios are shown; and Figure 5 A block diagram of an electronic device capable of implementing multiple illustrative scenarios is shown. Detailed Implementation

[0011] The examples in this document will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples presented herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0012] It should be noted that the headings of any section / subsection provided herein are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0013] In the description of the examples in this document, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] The examples in this document may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and rules. In the examples, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0015] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0016] In one context, an intelligent system is an information processing system. An intelligent system uses at least one or more models to process one or more inputs. An intelligent system may include one or more models as components, or it may use models external to the system for processing. Those skilled in the art will recognize that there are various implementations of intelligent systems in the industry, including but not limited to agents, artificial intelligence (AI) assistants, AI workers, copilot systems, and chatbots. Inputs include data in one or more modalities. Modal data includes, but is not limited to, text, audio, images, video, application-specific data, or scientific computing data. Models include, but are not limited to, one or more combinations of the following: large-scale parametric models, deep learning models, neural network models, machine learning models, statistical models, rule-based models, or other models capable of processing inputs. The specific implementation of the model can employ various existing or improved models. Intelligent systems may involve one or more of the following operations: receiving single or multiple rounds of input; recognizing and / or understanding the input; generating answers, decisions, or processing results using one or more models; invoking one or more external tools, interfaces, knowledge bases, or other systems to obtain information or perform operations; processing tasks autonomously, through human-machine collaboration, or by delegating tasks to humans or other systems. Autonomous decision-making refers to execution without human intervention. Human-machine collaboration refers to partial human involvement or confirmation. The above explanation of "intelligent system" is illustrative, and all descriptions are examples, not all of which are necessary, nor do they constitute an exhaustive list.

[0017] As used in this article, "media" can refer to a collection of data that can carry or present information. Media can include video, audio, images, moving images, text, or combinations thereof, and can be a complete media file, a media stream, or a media segment.

[0018] As used herein, “image frame” can refer to image data in a medium that can be presented as a single image. “First image frame” can refer to one or more image frames in a first medium, and may include some or all of the image frames in the first medium.

[0019] As used in this article, "audio" can refer to data that can be played or used to generate playable sound. Audio can include voice-over for characters, narration, background music, ambient sounds, sound effects, or a combination of several of these audio elements.

[0020] As used in this article, "first content" can refer to content derived from a first medium and used to describe how audio was generated for that first medium. First content may include information such as dialogue, the time range of the dialogue, character timbre, playback duration, volume, background music, ambient sounds, or sound effects.

[0021] As used in this article, "secondary content" can refer to content used to generate primary media. Secondary content may include video creation requirements and may describe scene settings, character actions, dialogue, time points, or other information related to the generation of primary media.

[0022] As used herein, the term "first input" can refer to information received through a first input component and used to generate the first audio. The first input may include at least one of the following: text content entered or confirmed by the user, audio settings, time range, and video, audio, or text resources selected by the user.

[0023] As used in this article, "component" refers to a UI object that can present content, receive operations, or represent processing status. Components can take the form of nodes, panels, windows, cards, controls, display areas, or other UI forms.

[0024] As used in this document, the terms "first input component" and "second input component" are used to distinguish input components associated with different editing modes of the first media. The first input component can be used to receive first input related to audio generation, while the second input component can be used to receive creative requirements related to video generation or video editing.

[0025] As used in this article, "audio component" can refer to a UI component associated with audio processing. An audio component can be used to receive audio generation requests, associate media to be processed, or present information related to the audio generation process, and can take the form of an audio node.

[0026] As used in this document, "element" can refer to an interface object used to present information, indicate object relationships, or receive user actions. Elements can take the form of text, numbers, icons, labels, thumbnails, connectors, logos, or other visual forms.

[0027] As used in this article, "interface element" refers to an interface object that can receive trigger operations and enable an electronic device to perform corresponding functions or display corresponding content. Interface elements can serve as function entry points and can take the form of buttons, icons, text, controls, node ports, or interactive areas.

[0028] As used in this article, "descriptive information" can refer to information used to describe the media processing process or the media generation status. Descriptive information may include text, icons, progress indicators, progress values, animations, or a combination thereof.

[0029] As used in this article, "generation status" refers to the processing stage of a media generation task. Generation status can include states such as waiting to generate, generating in progress, generation completed, or generation failed.

[0030] As used in this article, a "trigger operation" can refer to an operation that enables an electronic device to perform a corresponding function or change the state of its interface. A trigger operation can consist of a single operation or multiple consecutive operations.

[0031] As used in this article, "associated location" can refer to an interface location determined by the association between two objects. The associated location of a component can be inside, above, below, adjacent to, or in other locations where the relationship between the two can be indicated by connecting elements.

[0032] As used in this article, "connection" can refer to establishing a data transfer relationship, media processing relationship, or visual relationship on an interface between two objects. Connections can be represented by connecting lines, connection ports, or associated identifiers, or they can be logical connections that are not visible in the interface.

[0033] As used in this document, "edit mode" can refer to a set of functional states used to determine media generation or editing objects, processing flows, or settable parameters. Edit modes can include video generation modes, audio generation modes, or other media editing modes. A first edit mode can be used to generate or edit video content of a first media, and a second edit mode can be used to generate or edit audio content of the first media.

[0034] As used in this article, "resources" can refer to data that can be selected and used for media generation or media processing. Resources can include video, audio, images, text, creative requirements, parameter settings, or other data that can be used as input or reference.

[0035] As used in this article, a "resource collection" can refer to one or more resources organized in a predetermined manner. Resources in a resource collection can come from the current creation area, local storage space, servers, or other data sources accessible by electronic devices.

[0036] As mentioned above, with the development of digital media technology, editing software is widely used in the creation and processing of video, audio, and other media content. Editing software typically provides functions such as importing materials, trimming clips, arranging timelines, adjusting visuals, processing audio, adding special effects, and exporting content. Users can combine and edit multiple media materials through interactive interfaces such as timelines, canvases, or nodes, and adjust the presentation of the media content according to creative needs, thereby generating corresponding media works.

[0037] A user interface interaction scheme is proposed. The scheme includes: presenting a first input component, which is associated with a first medium, the first medium including a first image frame; receiving a first input through the first input component; and presenting a second medium, the second medium including the first image frame and a first audio, the first audio being generated based on the first input and the first medium.

[0038] Through the above scheme, the first input component is associated with a first medium including a first image frame, enabling the user to provide a first input to the first medium through the first input component and obtain a second medium including the first image frame and the first audio. This reduces the operations required to process the image content and audio separately and to recombine them, thereby improving the efficiency of human-computer interaction. Furthermore, since the first audio is generated based on the first input and the first medium, the audio generation process can be simultaneously constrained by the user's creative requirements and the first medium, thereby reducing the possibility of mismatch between the first audio and the image content and improving the matching degree between the image content and the first audio in the second medium.

[0039] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0040] Example Environment Figure 1 A schematic diagram of example environment 100 is shown. (e.g.) Figure 1 As shown, example environment 100 may include electronic device 110.

[0041] In this example environment 100, electronic device 110 can run an application 120 that supports user interface interaction. Application 120 can be any suitable type of application for user interface interaction, including but not limited to: media editing applications or other suitable applications. User 140 can interact with application 120 via electronic device 110 and / or its attached devices.

[0042] exist Figure 1 In environment 100, if application 120 is active, electronic device 110 can use application 120 to present interface 150 for supporting interface interaction.

[0043] In some cases, electronic device 110 communicates with server 130 to provide services to application 120. Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, virtual reality (VR) / augmented reality (AR) devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some cases, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0044] Server 130 can be a single physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and basic cloud computing services. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for applications 120 that support user interface interaction in electronic devices 110.

[0045] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection can include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections. In some cases, server 130 and electronic device 110 can exchange signaling information through their communication connection.

[0046] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme.

[0047] The following description of the example will continue with reference to the accompanying drawings.

[0048] Example Interaction Figures 2A to 2G Example interfaces 200A to 200G are shown, illustrating interface interactions under various scenarios. Interfaces 200A to 200G can be derived from... Figure 1 The illustrated electronic device 110 is provided via application 120. Application 120 can be a media editing application for processing media content such as video, audio, images, or text, and can present nodes for carrying media and input components associated with those nodes in the creation area. The creation area can be an infinite canvas that can expand its display area and support panning and zooming, or it can be an area with fixed boundaries. Server 130 can provide media analysis, audio generation, and media compositing services to electronic device 110.

[0049] In some cases, the first medium includes a first image frame, and a first input component is associated with the first medium. The first medium may include video with audio, or video without audio or with audio effects to be adjusted. The first image frame may include all or part of the image frames in the first medium, and the content presented constitutes the corresponding screen content. The association of the first input component with the first medium may indicate that the content received by the first input component is used to process the first medium, or it may indicate that the position, connection relationship, or association identifier of the first input component in the creation area indicates the first medium. In some scenarios, the electronic device 110 may store node identifiers, node types, positions, and sizes for each node in the creation area, and determine the display position of the node in the interface based on the current scaling ratio and translation offset of the creation area. For elements connecting nodes, the electronic device 110 may determine the start and end points of the element based on the connection ports of the connected nodes; when a node is moved or the creation area is scaled or translated, the electronic device 110 may update the display positions of the corresponding node and connected elements.

[0050] In some cases, the primary media is presented in the primary area. For example, such as... Figure 2A As shown, the electronic device 110 can present an interface 200A and display video nodes in the creation area of ​​the interface 200A. The display area of ​​the video node can be used as a first area, and a first medium 202 can be presented in the node. The first medium 202 can be a video generated by the application 120, a video imported by the user, or a video output by other nodes in the creation area. Figure 2A The first media 202 is illustrated using a media named "Video 1" as an example. The first area can be a display area in the creation area used to present the media, which can be defined by the boundaries of the video node, or it can be other interface areas used to present the media.

[0051] In some cases, the first interface element is presented in response to a trigger action on the first medium. For example, such as... Figure 2AAs shown, in response to a user's click, touch, hover, or other triggering operation on the first media 202 or the video node carrying the first media 202, the electronic device 110 presents a toolbar 206 above the video node. The toolbar 206 may include multiple entry points for editing media, wherein the audio optimization control 208 may serve as a first interface element. Other controls in the toolbar 206 may be used to perform cutout, extension, or other media processing.

[0052] In some cases, the first input component is presented in response to a trigger action on the first UI element. As an example, the user can trigger... Figure 2A The audio optimization control 208 is in the middle; in response to this trigger operation, the electronic device 110 can be switched from the interface 200A to the audio optimization control 208; Figure 2B The interface 200B is shown, and a first input component 210 is presented at the associated location of the first media 202. The first input component 210 is used to receive authoring requirements for generating audio for the first media 202. The authoring requirements can be used to describe the goals and constraints of generating or editing the media. The authoring requirements can include at least one of natural language content, parameter settings, and reference resources, and can indicate information such as the content, type, time range, scene, character actions, dialogue text, timbre, duration, volume, background music, or sound effects of the media to be generated.

[0053] In some cases, the electronic device 110 presents a second input component associated with a first editing mode of the first media, and in response to a trigger operation on a first interface element, stops presenting the second input component and presents the first input component, which is associated with the second editing mode of the first media. As an example, such as... Figure 2A As shown, the second input component 204 can be associated with the video generation mode, which is the first editing mode, and is used to receive creative requirements for generating or editing video, and can present models, aspect ratios, or other settings related to video generation; such as Figure 2B As shown, after the audio optimization control 208 is triggered, the electronic device 110 can stop displaying the second input component 204 and display the first input component 210 in its original position. The first input component 210 can be associated with the video dubbing mode, which is the second editing mode, and displays the video dubbing mode and settings related to audio generation, thereby indicating that the current mode has been switched from video generation mode to audio generation mode.

[0054] In this way, users can switch from video generation to audio generation without having to navigate to other interfaces or relocate the primary media. This reduces interface switching and repetitive search operations, improves human-computer interaction efficiency, and also reduces the mixing of parameters between different editing modes, thus improving the accuracy of media processing.

[0055] In some cases, in response to a triggering action on a first interface element, first content obtained based on a first media is presented in a first input component. As an example, such as... Figure 2B As shown, the text area in the first input component 210 can pre-present first content. The first content may include natural language content describing how to generate audio for the first media 202, such as dialogue in the first media 202, the time range corresponding to the dialogue, character timbre, background music, ambient sounds, and sound effects, at least one of these. Users can directly use the first content, or they can add, delete, or modify the first content. The first content is obtained based on the first media and may refer to the electronic device 110 or intelligent system identifying, extracting, matching, deleting, supplementing, or reorganizing the content related to audio generation in the first media based on the screen content, original audio, media timeline, or other information associated with the first media, to form the first content describing how to generate audio for the first media.

[0056] In this way, users can complete the corresponding creative requirements by confirming or making partial modifications without having to input dialogue, time range and audio settings one by one from a blank state. This reduces input operations and repeated viewing of the first media content, improves human-computer interaction efficiency, reduces content mismatch between creative requirements and the first media, and improves the matching degree between the generated audio and the first media.

[0057] In some cases, the first media is generated based on the second content, which is generated by an intelligent system based on the screen content and the second content. The second content may include the video creation requirements used when generating the first media, and may describe the screen scene, character actions, dialogue content, and their time points. The electronic device 110 can acquire the screen frame sequence, media timeline, and second content of the first media, and provide them to the intelligent system. The intelligent system can divide the screen frame sequence into multiple screen segments according to the position of screen changes, identify the characters, character actions, character speaking periods, scene changes or events in each screen segment, and determine the start and end times of each screen segment in the media timeline. The intelligent system can perform semantic analysis on the second content, extract dialogue text, dialogue characters, dialogue time points, music descriptions, and sound effect descriptions, and match the extracted results with the corresponding screen segments according to the correspondence between time points and characters or events. The intelligent system can delete content in the second content that is only used to describe composition, color, or shot appearance and is unrelated to audio generation, and correct the corresponding time points according to the speaking time of characters in the scene clip, the time of scene changes, or the time of event occurrence, and then supplement audio generation descriptions such as timbre, duration, volume, music, or sound effects. The intelligent system can organize the matched and corrected information into natural language content including at least one of time range, sound type, sound content, timbre, duration, and volume to obtain the first content. After obtaining the first content, the electronic device 110 can pre-fill the first content into the first input component 210.

[0058] This approach eliminates the need for users to re-enter creative information related to the primary media, thereby reducing input operations, improving content generation efficiency, and enhancing the matching degree between the dialogue, time range, and audio description in the primary content and the primary media. It also reduces content omissions or time mismatches, thereby improving the quality of subsequent audio generation.

[0059] In some cases, electronic device 110 receives first input through a first input component. The first input may include first content confirmed or modified by the user in the first input component 210, or it may include creative requirements re-entered by the user. For example, the user may instruct in the first input to present dialogue within a first time frame, background music within a second time frame, and specify the dialogue text, character timbre, playback duration, and volume. The first input may also instruct to maintain the same character's timbre and set the playback duration or volume for different statements, or use natural language to describe the desired overall sound effect.

[0060] In some cases, the first input includes information associated with a timestamp of the first media. As an example, the first input might include "use timbre A at the third second," where "third second" indicates the corresponding time position in the first media, and "timbre A" indicates the timbre requirement corresponding to that time position. The intelligent system can determine the corresponding position in the first media based on the timestamp and generate the first audio content corresponding to that position based on the first media and the corresponding timbre requirement. In this way, users can specify the sound characteristics of the corresponding position through input containing a timestamp, reducing the need to locate audio segments and set parameters separately, and improving the degree to which the generated audio conforms to the time position requirements.

[0061] In some scenarios, during the generation of the second media, the electronic device 110 displays descriptive information in a first area, indicating the generation status of the second media. For example, after a user triggers the generation control in the first input component 210, the first area can display descriptive information such as "Generating" and the generation progress. In response to the completion of the second media generation, the electronic device 110 stops displaying this descriptive information and displays the second media in the first area. Thus, the user can know the processing status of the generation task in the original video node and directly view the generation result after the task is completed.

[0062] In some cases, the electronic device 110 presents a second medium, which includes a first image frame and a first audio, the first audio being generated by an intelligent system based on a first input and the first medium. As an example, such as... Figure 2C As shown, electronic device 110 can present interface 200C, with first media 202 presented in a first area. After the first audio is generated and combined with the screen content, electronic device 110 can stop presenting the first media 202 in the first area and present the second media 212 in the same first area. When both the first and second media are video, the first media can be referred to as the first video, and the second media as the second video. The second media 212 can retain the screen content of the first media 202; the second video can be obtained by replacing the audio of the first video with the first audio, or by adding the first audio to the first video. In some cases, the first image frame is all the image frames of the first media, and the second media includes all the image frames of the first media and the first audio. In this way, the audio can be updated while retaining the complete image frame sequence of the first media, allowing the user to view the audio processing result based on the original complete screen, reducing the operation of reconstructing the screen sequence.

[0063] In some scenarios, server 130 can provide the intelligent system with a first input and a first media 202. The intelligent system can obtain a first image frame from the first media 202 and generate a first audio corresponding to the progress of the video based on the character's actions, speaking time, scene changes, or event occurrence time in the first image frame. After obtaining the first audio, server 130 can maintain the video track or image frame sequence of the first media 202 and align the first audio with the video track according to the start and end times corresponding to the first audio. When replacing the original audio, the original audio track of the first media 202 can be deleted or disabled, a new target audio track containing the first audio can be created, and the target audio track and video track can be encapsulated as a second media 212. When adding the first audio, the original audio track can be retained, an additional audio track can be created in the media timeline, the first audio can be written into the corresponding time range of the additional audio track, and the original audio track and the additional audio track can be retained separately or mixed and encapsulated with the video track as a second media 212. After obtaining the second media 212, electronic device 110 can play the second media 212 in the original video node, so that the user can judge whether the first audio meets the creation requirements in combination with the video.

[0064] In some cases, a first element is presented in the first area, indicating that the first area is associated with multiple media, including a second media. For example, such as... Figure 2C As shown, the electronic device 110 can display a first element 214 in the upper right area of ​​the video node. The first element 214 may include an expansion identifier and quantity information, wherein the quantity information is used to indicate the number of media associated with the video node. Figure 2C The number "3" indicates that the node is associated with three media, and does not indicate the generation progress of the second media. After the second media 212 is presented, the electronic device 110 can also resume presenting the second input component 204 in order to continue receiving other video creation requests.

[0065] In this way, users can learn from the first element that the first area is associated with multiple media, including the second media, thus determining the number and association status of generated results without creating multiple media nodes separately, reducing interface space usage and improving the efficiency of information retrieval for users. Furthermore, this method can also reduce the decentralized maintenance of media association status, facilitating unified management of different media results by electronic devices.

[0066] In some cases, electronic device 110 presents multiple media in response to a first operation on a first element, the multiple media including a second media. As an example, a user can trigger... Figure 2C The first element 214; in response to this trigger operation, the electronic device 110 can present Figure 2DThe interface 200D shown presents multiple media associated with the same first region in the media list panel 216. The media list panel 216 can display a second media 212 and other generated candidate media. The second media 212 can be labeled "Video 2", the generated media 217 can be labeled "Video 3", and the generated media 218 can be labeled "Video 4".

[0067] In this way, users can manage different media more conveniently, thereby improving the efficiency of human-computer interaction. Furthermore, this method can also narrow down the scope of media searches and improve the efficiency of obtaining and presenting related media.

[0068] In some cases, the multiple media items presented in the media list panel 216 may correspond to one or more generated results for the same creative requirement. Users can select the media item presented as the current result in the first area. When the video node containing the first area is associated with a downstream node, the downstream node can use the currently selected media item as input for subsequent processing. Users can also trigger any media item to view the creative requirements used to generate it, and regenerate the media item after modifying the creative requirements; the regenerated result can continue to be presented as media associated with the first area.

[0069] This approach reduces the need for interface switching and repetitive node creation and connection operations in the creation area, improving human-computer interaction efficiency. Furthermore, it avoids inconsistencies arising from separately managing video and audio files, enhancing the matching degree between the initial audio and video progress and semantics, thereby improving video generation quality.

[0070] In some scenarios, the electronic device 110 presents an audio component, which is associated with the first input component. As an example, a user can trigger an audio node creation entry or drag and drop an audio node into the creation area. In response to this action, such as... Figure 2E As shown, the electronic device 110 can present an interface 200E, and present an audio component 220 and a first input component 222 located below the audio component 220 in the interface 200E.

[0071] In some cases, the audio component 220 can take the appearance of an audio node and display waveform areas, playback controls, and media duration. The first input component 222 can display prompts such as "Please add the video you want to dub," and can display the video dubbing mode and settings related to audio generation. Figure 2E The audio component 220 shown is not yet associated with the first media, and its waveform region and zero duration are used to indicate the state to be generated.

[0072] In some cases, a second interface element is presented in a first input component, a resource set is presented in response to a triggering operation on the second interface element, and a fourth operation is received for selecting at least one resource from the resource set, wherein the first input includes the selected at least one resource. As an example, such as... Figure 2E As shown, the add control 224 in the first input component 222 can serve as a second interface element. After the user triggers the add control 224, the electronic device 110 can present a resource collection, which may include video, audio, and text resources in the creation area, as well as video, audio, and text files that can be imported from the local device. The user can select at least one resource from the resource collection, and the selected resource can be provided to the intelligent system as part of the first input.

[0073] This approach reduces the need to search, copy, or re-import reference resources across different interfaces, improving human-computer interaction efficiency. Furthermore, it narrows the scope of resources the intelligent system needs to process, reduces the transmission and processing of irrelevant resources, and enhances the match between the generated results and the user-selected reference resources.

[0074] In some cases, the electronic device 110 receives a second operation for receiving the first media and presenting the first media in the first input component. As an example, a user can select a video node in the creation area, connect the video node to the audio component 220, or import a local video through the first input component 222. Figure 2F The diagram illustrates the interface state after the user selects video 226 as the first media in interface 200F. Video 226 can be displayed as "Video 5" and presented as a video thumbnail 225 in the first input component 222. The video thumbnail 225 can display image frames from video 226, allowing the user to view the received video 226. In this way, the user can confirm the media to be processed through the video 226 presented in the first input component, thereby improving the efficiency of the user's information acquisition.

[0075] In some cases, the second operation is used to associate the first media with the audio component, and the electronic device 110 also presents a second element connecting the first media and the audio component. As an example, such as... Figure 2FAs shown, a connecting line 228 can be presented between the presentation node of video 226 and audio component 220. The connecting line 228 can serve as a second element, indicating that video 226 is provided to audio component 220 through its connection endpoints and extension direction. In some scenarios, the user can drag the connecting line from the connection port of the video node where video 226 is located in the creation area; during dragging, the electronic device 110 can dynamically adjust the extension direction of the connecting line according to the dragging position. When the user drags the connecting line to the connection port of the corresponding audio node of audio component 220 and releases it, the electronic device 110 can establish the association between video 226 and audio component 220, and keep the connecting line 228 presented between the video node and the audio node. In this way, the user can intuitively determine that the first media has been associated with the audio component through the second element and understand the processing relationship of the media data, thereby improving the efficiency of the user in obtaining information.

[0076] In some cases, the first input may also include text content entered by the user through the first input component 222 and / or other reference resources selected through a resource set. The user may specify dialogue, narration, timbre, volume, music, sound effects, and corresponding time ranges in the text content. The selected audio resources may be used to provide timbre references; the selected text resources may be used to provide character voice descriptions, lines, or other audio creation requirements that can be reused in different generation tasks. For videos selected or uploaded by the user, the electronic device 110 may keep the first input component 222 in an input-ready state so that the user can indicate the creative intent for re-dubbing.

[0077] In some cases, in response to the completion of second media generation, electronic device 110 stops presenting the audio component and presents the second media and a third element, the third element connecting the first and second media. As an example, after a user provides a first input via the first input component 222 and triggers generation, the intelligent system can generate first audio based on the first input and the video 226 serving as the first media. After generation, as... Figure 2G As shown, the electronic device 110 can switch from interface 200F to interface 200G, stop displaying the audio component 220 in the form of an audio node, and display video 230 as a second medium. Video 230 can be displayed as "Video 6" and includes the first image frame in video 226 and the generated first audio.

[0078] In some cases, the electronic device 110 presents a connecting line 232 between video 226 and video 230. The connecting line 232 can serve as a third element to indicate that video 230 is a dubbing result derived from video 226. Video 226 can remain in the creation area, and the user can trigger playback and comparison of video 226 and video 230 separately. Thus, although the generation process uses the first audio as an intermediate processing result, the creation area still uses video 230, formed by the combination of the visual content and the first audio, as the playable generated result.

[0079] In some scenarios, the intelligent system can determine multiple time ranges based on the timeline of the video 226, and generate dialogue, background music, ambient sounds, or sound effects for the corresponding time ranges based on the first input. The electronic device 110 or server 130 can arrange or mix the various audio contents into a first audio based on timestamps, and align the start time, end time, and duration of the first audio with the progress of the video 226. When the duration of the first audio is inconsistent with the duration of the video 226, the audio segment can be adjusted based on the first input, or silent content can be added within the corresponding time range, so that the combined video 230 can play the video and the first audio according to a unified timeline.

[0080] In some scenarios, after a user triggers video 230, electronic device 110 can present the first input used to generate video 230, allowing the user to modify the text content, replace reference resources, or re-trigger generation. During regeneration, video 226 can still be provided to the intelligent system as the first medium. After obtaining new first audio, electronic device 110 can again combine it with the visual content of video 226 to present a new second medium, thereby supporting the user to iteratively adjust the dubbing based on the synchronization effect of the visuals and sound.

[0081] In this way, users can directly play and judge the correspondence between the voice-over and the visuals, reducing the need to manually export audio, create a timeline, re-import audio, and connect result nodes, thus improving the efficiency of human-computer interaction. Furthermore, it can also improve the matching degree between the first audio and the visual progress, thereby improving the quality of video generation.

[0082] In some cases, the first audio can be voice-over, narration, background music, ambient sounds, or sound effects, or it can include audio composed of two or more of these audio contents. The first input can describe the time range, content, timbre, playback duration, volume, or interrelationship of different audio contents in natural language. The intelligent system can combine the video content to determine the applicable video segment for the corresponding description and generate the corresponding sound for that video segment. This can be used to optimize the audio of the original video or to generate a re-dubbing result that differs from the original audio content.

[0083] Through the above scheme, the first input component is associated with a first medium including a first image frame, enabling the user to provide a first input to the first medium through the first input component and obtain a second medium including the first image frame and the first audio. This reduces the operations required to process the image content and audio separately and to recombine them, thereby improving the efficiency of human-computer interaction. Furthermore, since the first audio is generated based on the first input and the first medium, the audio generation process can be simultaneously constrained by the user's creative requirements and the first medium, thereby reducing the possibility of mismatch between the first audio and the image content and improving the matching degree between the image content and the first audio in the second medium.

[0084] Example process Figure 3 A flowchart illustrating an example process 300 for interface interaction under certain conditions is shown. Process 300 can be implemented at electronic device 110. See below for reference. Figure 1 To describe process 300.

[0085] like Figure 3 As shown, in block 310, electronic device 110 presents a first input component, which is associated with a first medium, the first medium including a first image frame.

[0086] In frame 320, electronic device 110 receives a first input via a first input component.

[0087] In frame 330, electronic device 110 presents a second medium, which includes a first image frame and a first audio, the first audio being generated based on the first input and the first medium.

[0088] In some cases, presenting the first input component includes: presenting a first interface element in response to a triggering operation on a first media; and presenting the first input component in response to a triggering operation on a first interface element.

[0089] In some cases, process 300 may also include presenting a second input component, which is associated with a first editing mode of the first media.

[0090] In some cases, in response to a triggering operation on a first interface element, presenting a first input component includes: in response to a triggering operation on a first interface element, stopping the presentation of a second input component and presenting the first input component, the first input component being associated with a second editing mode of a first media.

[0091] In some cases, process 300 further includes: in response to a triggering operation, presenting first content in the first input component, the first content being obtained based on a first medium.

[0092] In some cases, the first media is generated based on the second content, and the first content is generated by an intelligent system based on the image content and the second content.

[0093] In some cases, the presentation of the first media in the first area and the presentation of the second media include: stopping the presentation of the first media in the first area and presenting the second media in the first area.

[0094] In some cases, process 300 further includes: presenting a first element in a first region; and, in response to a first operation on the first element, presenting multiple media, including a second media.

[0095] In some cases, the first image frame is all the image frames of the first medium.

[0096] In some cases, process 300 further includes: in a first area, presenting descriptive information indicating the generation status of the second media; wherein presenting the second media includes: in response to the completion of the generation of the second media, stopping the presentation of the descriptive information and presenting the second media in the first area.

[0097] In some cases, process 300 further includes: receiving a second operation for receiving the first media; and presenting the first media in the first input component.

[0098] In some cases, process 300 may also include: presenting an audio component, which is associated with the first input component.

[0099] In some cases, the second operation is used to associate the first media with the audio component, and process 300 further includes: presenting a second element that connects the first media and the audio component.

[0100] In some cases, presenting the second media includes: stopping the presentation of the audio component and presenting the second media and a third element, the third element connecting the first and second media.

[0101] In some cases, receiving the first input includes: presenting a second interface element in a first input component; presenting a resource set in response to a triggering operation on the second interface element; and receiving a fourth operation for selecting at least one resource from the resource set, wherein the first input includes at least one resource. In some cases, the first input includes information associated with a timestamp of a first media.

[0102] Example devices and equipment A corresponding apparatus for implementing the above methods or processes is also provided. Figure 4A schematic structural block diagram of an example device 400 for interface interaction is shown, according to some scenarios. Device 400 can be implemented as or included in electronic device 110. The various modules / components in device 400 can be implemented by hardware, software, firmware, or any combination thereof.

[0103] like Figure 4 As shown, the device 400 includes: a first presentation module 410 configured to present a first input component associated with a first medium, the first medium including a first image frame; a first receiving module 420 configured to receive a first input through the first input component; and a second presentation module 430 configured to present a second medium including the first image frame and a first audio, the first audio being generated based on the first input and the first medium.

[0104] In some cases, the first presentation module 410 is also configured to: present a first interface element in response to a trigger operation on the first media; and present a first input component in response to a trigger operation on the first interface element.

[0105] In some cases, the first presentation module 410 is also configured to: present a second input component associated with a first editing mode of the first media; and, in response to a trigger operation on a first interface element, stop presenting the second input component and present the first input component associated with a second editing mode of the first media.

[0106] In some cases, device 400 also includes a third presentation module configured to present first content on a first input component in response to a trigger operation on a first interface element, the first content being obtained based on a first media.

[0107] In some cases, the first media is generated based on the second content, and the first content is generated by an intelligent system based on the image content and the second content.

[0108] In some cases, the first media is presented in the first area, and the second presentation module 430 is also configured to: stop presenting the first media in the first area and present the second media in the first area.

[0109] In some cases, device 400 also includes a fourth presentation module configured to present a first element in a first region; and to present multiple media, including a second media, in response to a first operation on the first element.

[0110] In some cases, the first image frame is all the image frames of the first medium.

[0111] In some cases, the second presentation module 430 is also configured to: present descriptive information in a first area, the descriptive information indicating the generation status of the second media; and, in response to the completion of the generation of the second media, stop presenting the descriptive information and present the second media in the first area.

[0112] In some cases, device 400 further includes a sixth presentation module configured to receive a second operation for receiving the first media; and to present the first media in a first input component.

[0113] In some cases, the sixth presentation module is also configured to present an audio component, which is associated with the first input component.

[0114] In some cases, the second operation is used to associate the first media with the audio component, and the device 400 also includes a seventh presentation module configured to present a second element that connects the first media and the audio component.

[0115] In some cases, the second presentation module 430 is also configured to: stop presenting the audio component and present a second media and a third element, the third element connecting the first media and the second media.

[0116] In some cases, the first receiving module 420 is further configured to: present a second interface element in a first input component; present a resource set in response to a trigger operation on the second interface element; and receive a fourth operation for selecting at least one resource in the resource set, wherein the first input includes at least one resource.

[0117] In some cases, the first input includes information associated with the timestamp of the first media.

[0118] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some cases, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0119] Figure 5 A block diagram of an electronic device 500 in which one or more examples can be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein. Figure 5 The electronic device 500 shown can be used to implement the electronic device 110 discussed above.

[0120] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processing units or processors 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0121] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0122] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various examples.

[0123] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0124] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0125] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0126] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0128] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0129] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0130] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for interface interaction, comprising: A first input component is presented, the first input component being associated with a first medium, the first medium including a first image frame; The first input is received through the first input component; as well as A second medium is presented, the second medium including the first image frame and a first audio, the first audio being generated based on the first input and the first medium.

2. The method of claim 1, wherein the presentation of the first input component comprises: In response to a trigger operation on the first media, the first interface element is presented; as well as In response to a trigger operation on the first interface element, the first input component is presented.

3. The method according to claim 2, further comprising: A second input component is presented, which is associated with a first editing mode of the first media; Furthermore, the presentation of the first input component in response to a trigger operation on the first interface element includes: In response to the triggering operation, the presentation of the second input component is stopped, and the first input component, which is associated with the second editing mode of the first media, is presented.

4. The method according to claim 2, further comprising: In response to a trigger operation on the first interface element, first content is presented on the first input component, the first content being obtained based on the first media.

5. The method of claim 1, wherein the first media is presented in the first region, and the presentation of the second media comprises: Stop displaying the first media in the first area, and display the second media in the first area.

6. The method of claim 5, further comprising: In the first region, the first element is presented; as well as In response to a first operation on the first element, multiple media are presented, including the second media.

7. The method of claim 5, further comprising: In the first area, descriptive information is presented, which indicates the generation status of the second media; as well as The second medium for presentation includes: In response to the completion of generating the second media, the presentation of the description information is stopped, and the second media is presented in the first area.

8. The method according to claim 1, further comprising: Receive a second operation, the second operation being used to receive the first medium; as well as The first media is presented in the first input component.

9. The method according to claim 8, further comprising: An audio component is presented, which is associated with the first input component.

10. The method of claim 9, wherein the second operation is used to associate the first media with the audio component, the method further comprising: A second element is presented, which connects the first media and the audio component.

11. The method of claim 9, wherein presenting the second medium comprises: Stop presenting the audio component and present the second media and the third element, the third element connecting the first media and the second media.

12. The method of claim 1, wherein receiving the first input comprises: The second interface element is displayed in the first input component; In response to a trigger operation on the second interface element, a resource collection is presented; as well as A fourth operation is received, the fourth operation being used to select at least one resource from the resource set, wherein the first input includes the at least one resource.

13. The method of claim 1, wherein the first image frame is all image frames of the first medium.

14. The method of claim 1, wherein the first input includes information associated with a timestamp of the first media.

15. A device for interface interaction, comprising: A first presentation module is configured to present a first input component, the first input component being associated with a first medium, the first medium including a first image frame; The first receiving module is configured to receive a first input through the first input component; as well as The second presentation module is configured to present a second medium, the second medium including the first image frame and a first audio, the first audio being generated based on the first input and the first medium.

16. An electronic device comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 14 when executed by the at least one processor.

17. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 14.

18. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 14.

Images