Method for generating sound effect video and electronic equipment

Abstract

The invention provides a sound effect video generation method and electronic equipment, the method is applied to first electronic equipment, and the method comprises the following steps: obtaining prompt information for generating a sound effect video; determining a first instruction according to the prompt information, wherein the first instruction comprises instructions used for indicating prompt content of a video picture, audio content and a sound effect type; sending the first instruction to the second electronic equipment; receiving a first video and a first audio sent by a second electronic device; and generating a video with a sound effect according to the first video and the first audio, the sound effect being related to the sound effect type. According to the technical scheme, the user can conveniently create various types of sound effect videos by using the electronic equipment, so that the interestingness and diversity of chat or creation of the user are improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and more specifically, to a method and electronic device for generating audio and video effects. Background Art

[0002] Currently, users can use electronic devices to easily create personalized works such as emoticons and short videos. For example, electronic devices can generate video images or audio through prompt text, but the video images or audio are too simple and lack the experience of audio and video linkage. If electronic devices can generate video images with certain sound effects, it will increase the diversity of user creations.

[0003] Therefore, how to generate videos with sound effects becomes a technical problem that needs to be solved. Summary of the invention

[0004] The present application provides a method for generating a video with sound effects and an electronic device. The electronic device can generate a video with sound effects according to prompt information input by a user, so that the user can use the electronic device to conveniently create various types of videos with sound effects.

[0005] In a first aspect, a method for generating an audio effect video is provided, the method being applied to a first electronic device, the method comprising:

[0006] Obtain instructions for generating a video with sound effects; determine a first instruction based on the prompt information, wherein the first instruction includes indication information for indicating the prompt content, audio content, and sound effect type of the video screen; send the first instruction to a second electronic device; receive a first video and a first audio sent by the second electronic device, wherein the content of the first video is related to the prompt content of the video screen, and the content of the first audio is related to the audio content; generate a video with sound effects based on the first video and the first audio, wherein the sound effect is related to the sound effect type.

[0007] Exemplarily, the number of the first videos may be one or more, and the number of the first audios may be one or more.

[0008] The sound effect type may be an emotional sound effect, a stereo sound effect, an environmental sound effect, etc. The embodiment of the present application does not limit the specific form of the sound effect type.

[0009] Based on the embodiment of the present application, the first electronic device can obtain prompt information for generating a sound effect video, and determine a first instruction including prompt content of the video screen, audio content and sound effect type based on the prompt information, and send the first instruction to the second electronic device; receive the first video and first audio sent by the second electronic device, and generate a video with sound effects based on the first video and the first audio.

[0010] In this way, users can use electronic devices to conveniently create various types of audio and video effects, thereby enhancing the fun and diversity of user chats or creations.

[0011] In combination with the first aspect, in certain implementations of the first aspect, when the sound effect is an emotional sound effect, the first audio is an audio having the emotional sound effect, wherein generating a video with the sound effect based on the first video and the first audio includes: generating a video with the emotional sound effect based on the first video and the audio having the emotional sound effect.

[0012] For example, the emotional sound effect may be a joyful sound effect, a sad sound effect, etc.

[0013] In an embodiment of the present application, the electronic device can generate a video with emotional sound effects, so that the video can express the user's tone, emotion, atmosphere and other information, thereby enhancing the richness of the user's expression.

[0014] In combination with the first aspect, in certain implementations of the first aspect, when the sound effect is a stereo sound effect, the first audio is an audio without sound effects, and the method further includes: receiving a sound source trajectory sent by the second electronic device, the sound source trajectory being a movement trajectory of the sound source in the first video; wherein, generating a video with sound effects based on the first video and the first audio includes: generating an audio with stereo sound effects based on the audio without sound effects and the sound source trajectory; generating a video with stereo sound effects based on the first video and the audio with stereo sound effects.

[0015] Based on the embodiments of the present application, when the sound effect is a stereo sound effect, the first audio received by the first electronic device may be audio without sound effects, and the first electronic device may also receive the sound source track of the sound source sent by the second electronic device, and generate audio with stereo sound effects based on the sound source track and the audio without sound effects, and then fuse the audio with stereo sound effects and the first video to generate a video with stereo sound effects.

[0016] In this way, the electronic device can generate a video with stereo sound effects, and the stereo sound effects are closely linked to the video images, so that the user can identify the direction of the sound source and experience the sense of space, giving the user an immersive feeling.

[0017] In combination with the first aspect, in certain implementations of the first aspect, before receiving the sound source trajectory sent by the second electronic device, the method also includes: receiving a request message sent by the second electronic device, the request message being used to request determination of the sound source of the sound in the first video; and sending indication information to the second electronic device, the indication information being used to indicate the sound source of the sound in the first video.

[0018] In the embodiment of the present application, when the second electronic device cannot determine the sound source, it can send a request message to the first electronic device so that the first electronic device determines the sound source according to the user's operation, and then the first electronic device can send indication information to the second electronic device to indicate the sound source in the first video. This technical solution is conducive to the second electronic device to determine the sound source and further determine the sound source trajectory.

[0019] In other examples, the second electronic device may also determine the sound source through a target detection algorithm, and further determine the trajectory of the sound source.

[0020] In combination with the first aspect, in some implementations of the first aspect, obtaining prompt information for generating sound effect video includes: in an instant messaging scenario, obtaining prompt information for generating sound effect video in response to a user's operation.

[0021] In other examples, this technical solution can also be applied to scenarios such as short video creation and advertising creation.

[0022] In combination with the first aspect, in some implementations of the first aspect, the method further includes: displaying or sending the video with sound effects.

[0023] In an embodiment of the present application, in an instant messaging scenario, the electronic device can display the video with sound effects, or send the video with sound effects to another electronic device, thereby improving the diversity of materials for user communication.

[0024] In combination with the first aspect, in some implementations of the first aspect, the prompt information includes one or more of pictures, videos, texts, dynamic pictures, and audios.

[0025] In other examples, the prompt information may also include other content, which is not limited in the embodiments of the present application.

[0026] In a second aspect, a method for generating a sound effect video is provided, which is applied to a second electronic device, and the method includes: receiving a first instruction sent by a first electronic device, the first instruction including instructions for indicating the prompt content, audio content and sound effect type of a video screen; generating a corresponding first video and first audio according to the first instruction, wherein the content of the first video is related to the prompt content of the video screen, and the content of the first audio is related to the audio content; and sending the first video and the first audio to the first electronic device.

[0027] For example, the second electronic device may be a server, a computer, etc.

[0028] Based on the embodiment of the present application, the second electronic device can receive the first instruction sent by the first electronic device, generate the corresponding first video and first audio according to the first instruction, and send the first video and first audio to the first electronic device, thereby saving the computing power of the first electronic device.

[0029] In other examples, the electronic device may also first generate a first video, and generate a first audio based on the generated first video, thereby improving the consistency between the video image and the audio.

[0030] In combination with the second aspect, in certain implementations of the second aspect, when the sound effect is a stereo sound effect, the first audio is an audio without sound effects, and the method further includes: determining a sound source in the first video; determining a sound source track of the sound source in the first video; and sending the sound source track to the first electronic device.

[0031] Based on the embodiment of the present application, when the sound effect is a stereo sound effect, the second electronic device can first generate the first video and the audio without the sound effect, then determine the sound source in the first video and the sound source track of the sound source in the first video, and send the sound source track to the first electronic device. This is conducive to the first electronic device synthesizing the audio with a stereo sound effect.

[0032] In combination with the second aspect, in certain implementations of the second aspect, when the sound effect is a stereo sound effect, generating a corresponding first video and first audio according to the first instruction includes: generating the first video and audio without sound effects according to the first instruction; determining a sound source in the first video; determining a sound source track of the sound source in the first video; and generating a first audio with stereo sound effects according to the sound source track and the audio without sound effects.

[0033] Based on the embodiment of the present application, after determining the sound source and the sound source track in the first video, the second electronic device can directly fuse the sound source track and the audio without sound effects to generate the first audio with stereo sound effects, thereby saving the computing power and power consumption of the first electronic device.

[0034] In combination with the second aspect, in certain implementations of the second aspect, determining the sound source of the sound in the first video includes: sending a request message to the first electronic device, the request message being used to request determination of the sound source in the first video; receiving indication information sent by the first electronic device, the indication information being used to indicate the sound source in the first video; and determining the sound source in the first video based on the indication information.

[0035] Based on the embodiments of the present application, when the second electronic device cannot directly determine the sound source in the first video, it can also send a request message to the first electronic device to determine the sound source, and receive indication information sent by the first electronic device to indicate the sound source, and determine the sound source in the first video according to the indication information.

[0036] In this technical solution, the user can specify the sound source in the first video, thereby improving the diversity of the sound effect video.

[0037] In a third aspect, a method for generating a video with sound effects is provided, the method being applied to a first electronic device, the method comprising: obtaining prompt information for generating a video with sound effects; determining a third instruction based on the prompt information, the third instruction comprising instructions for indicating the prompt content, audio content and sound effect type of a video screen; determining a second video and a second audio with sound effects based on the third instruction, wherein the content of the second video is related to the prompt content of the video screen, the content of the second audio is related to the audio content, and the sound effect of the second audio is related to the sound effect type; and generating a video with sound effects based on the second video and the second audio.

[0038] Based on the embodiment of the present application, when the first electronic device has sufficient computing power, the first electronic device can generate a related second video and a second audio with sound effects according to the prompt information input by the user, and merge the second video and the second audio to generate a video with sound effects.

[0039] In other examples, the first electronic device may also directly generate a video with sound effects according to the prompt information input by the user, which is not limited in the embodiments of the present application.

[0040] In a fourth aspect, an electronic device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more programs, and when the one or more programs are executed by one or more processors, the method for generating sound effect video as described in the first aspect, the second aspect, the third aspect and any possible implementation thereof is executed.

[0041] In a fifth aspect, a device for generating audio and video effects is provided, comprising a module for implementing the method for generating audio and video effects as described in the first aspect, the second aspect, the third aspect and any possible implementation thereof.

[0042] In the sixth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal so that the method for generating sound effect video as described in the first aspect, the second aspect, the third aspect and any possible implementation thereof is executed.

[0043] In the seventh aspect, a readable storage medium is provided, wherein instructions are stored in the readable storage medium. When the instructions are executed on an electronic device, the method for generating sound effect video as described in the first aspect, the second aspect, the third aspect and any possible implementation thereof is executed.

[0044] In an eighth aspect, a program product is provided, the program product comprising a program code, and when the program code runs on an electronic device, the method for generating a sound effect video as described in the first aspect, the second aspect, the third aspect and any possible implementation thereof is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0046] Figure 2 It is a schematic diagram of the software structure of the electronic device provided in the embodiment of the present application.

[0047] Figure 3 It is a schematic diagram of a group of GUIs provided in an embodiment of the present application.

[0048] Figure 4 It is a schematic diagram of another set of GUIs provided in an embodiment of the present application.

[0049] Figure 5 It is a schematic diagram of another set of GUIs provided in an embodiment of the present application.

[0050] Figure 6 It is a schematic diagram of another set of GUIs provided in an embodiment of the present application.

[0051] Figure 7 It is a schematic diagram of another set of GUIs provided in an embodiment of the present application.

[0052] Figure 8 This is a system framework diagram to which the method for generating audio and video effects of the present application can be applied.

[0053] Fig. 9 It is a schematic interactive diagram of a method for generating audio and video provided in an embodiment of the present application.

[0054] Fig.10 It is a schematic interactive diagram of a method for generating audio and video provided in an embodiment of the present application.

[0055] Fig.11 It is a schematic flowchart of a method for generating audio and video provided in an embodiment of the present application.

[0056] Fig.12It is a schematic interactive diagram of a method for generating audio and video provided in an embodiment of the present application.

[0057] Fig.13 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0059] The method for generating audio and video in the embodiment of the present application can be applied to electronic devices such as smart phones, foldable phones, tablet computers, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), netbooks, vehicle-mounted devices, and Internet of Things (IOT) devices.

[0060] Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0061] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0063] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0064] The processor 110 may also be provided with a memory for storing instructions and data.

[0065] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus interface, etc.

[0066] The charging management module 140 is used to receive charging input from a charger.

[0067] The power management module 141 is used to connect the battery 142 , the charging management module 140 and the processor 110 .

[0068] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0069] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 100 .

[0070] The modulation and demodulation processor may include a modulator and a demodulator, wherein the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal, and the demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal.

[0071] The wireless communication module 160 can provide wireless communication solutions for application in the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth low energy (BLE), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0072] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0073] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0074] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

[0075] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0076] The ISP is used to process data fed back by the camera 193. The camera 193 is used to capture still images or videos.

[0077] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0078] The video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs.

[0079] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 .

[0080] The internal memory 121 may be used to store computer executable program codes, wherein the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121 .

[0081] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0082] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals.

[0083] The speaker 170A, also called a "horn", is used to convert audio electrical signals into sound signals.

[0084] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals.

[0085] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0086] The earphone jack 170D is used to connect a wired earphone.

[0087] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194 .

[0088] The gyro sensor 180B may be used to determine the motion posture of the electronic device 100 .

[0089] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0090] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes).

[0091] The distance sensor 180F is used to measure the distance.

[0092] The fingerprint sensor 180H is used to collect fingerprints.

[0093] The touch sensor 180K is also called a “touch panel.” The touch sensor 180K may be disposed on the display screen 194 , and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen.”

[0094] Bone conduction sensor 180M can obtain vibration signals.

[0095] Buttons 190 include a power button, a volume button, and the like.

[0096] Motor 191 can generate vibration prompts.

[0097] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0098] The SIM card interface 195 is used to connect a SIM card.

[0099] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes an operating system of a layered architecture as an example to exemplify the software structure of the electronic device 100.

[0100] Figure 2 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the system library, and the kernel layer. The application layer can include a series of application packages.

[0101] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, wallet, etc.

[0102] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0103] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0104] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0105] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0106] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0107] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0108] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0109] The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and disappear automatically after a short stay without user interaction.

[0110] The core library consists of two parts: one part is the function that the Java language needs to call, and the other part is the core library.

[0111] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0112] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0113] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0114] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0115] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0116] A 2D graphics engine is a drawing engine for 2D drawings.

[0117] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0118] Before introducing the technical solution of the present application, some technical terms that may be involved in the present application are briefly introduced as follows.

[0119] Spatial audio algorithm: refers to an algorithm that simulates and reproduces the position, direction, and distance of sound in three-dimensional space.

[0120] Audio effect video: a video with a certain type of audio effect, or it can be understood that the audio in the audio effect video has a certain type of audio effect. In the embodiment of the present application, the audio effect type may include emotional sound effects, spatial sound effects, etc. For example, the emotional sound effect may include joyful sound effects, sad sound effects, neutral sound effects, etc., and the spatial sound effect may also be called stereo sound effects or three-dimensional sound effects.

[0121] Currently, users can use electronic devices to easily create personalized works such as emoticons and short videos. For example, electronic devices can generate video images or audio through prompt text, but the video images or audio are too simple and lack the experience of audio and video linkage. If electronic devices can generate video images with certain sound effects, it will increase the diversity of user creations.

[0122] In view of this, the present application provides a method and electronic device for generating a video with sound effects. In the technical solution, the electronic device can generate a video with sound effects according to prompt information input by the user, so that the user can use the electronic device to conveniently create various types of videos with sound effects.

[0123] The following will combine Figure 3-7 The graphical user interface (GUI) in the embodiment introduces the technical solution of the present application.

[0124] It is understandable that the embodiment of the present application uses the method of generating audio and video effects in an instant messaging scenario as an example for explanation, but this should not limit the application scenario of the present application. In other embodiments, the method can also be applied to audio and video creation scenarios such as flower petal editing, and the embodiment of the present application does not limit the specific application scenario.

[0125] Figure 3 is a schematic diagram of a set of GUIs provided in an embodiment of the present application. Figure 3 (a) to (f) in the figure show the process of a user using an electronic device to generate an audio and video effect in an instant messaging (IM) scenario.

[0126] For example, see Figure 3 In (a), the GUI may be a display interface 210 of an IM of an electronic device. The display interface 210 may include a communication message display area and an input method display area. In addition to the input method input keys, the input method display area may also include an entry mark 211 for generating audio and video effects.

[0127] The embodiment of the present application does not limit the specific position of the entrance mark 211. For example, the entrance mark 211 can be located above the input button area, or the entrance mark 211 can also be suspended above the input button.

[0128] When the electronic device detects that the user clicks the entry mark 211, the following display may be displayed: Figure 3 GUI shown in (b) in .

[0129] See also Figure 3 In (b), the input method display area may have a number of materials, such as picture 211, picture 222, video 223, dynamic picture 224, etc.

[0130] In other examples, after clicking the entry mark 211, the user may also input text prompt information in the input box so that the electronic device displays materials related to the prompt information.

[0131] It should be understood that the user can also switch the material displayed by the electronic device by sliding left and right in the display area of ​​the material.

[0132] When the electronic device detects that the user long presses the picture 221, the following display may be displayed: Figure 3 In some other examples, the picture 221 may also include a creation button, and when the electronic device detects that the user clicks the creation button, it may also display the following Figure 3 GUI shown in (c) in FIG.

[0133] See also Figure 3 In (c), a tab sheet 225 may pop up on the picture 221, and the tab sheet 225 may include audio and video creation options.

[0134] In other examples, the tab sheet may also include multiple options, for example, the tab sheet 225 may include an audio effect video creation option and an add audio effect option. Alternatively, the tab sheet may only include an add audio effect option.

[0135] When the electronic device detects that the user clicks the tab sheet 225 and inputs the prompt information 1, the following display may be displayed: Figure 3 GUI shown in (d).

[0136] See also Figure 3 In (d), the electronic device can display the prompt information 1 in the input box.

[0137] When the electronic device detects that the user clicks the confirmation button, it can display the following Figure 3 GUI shown in (e) in .

[0138] See also Figure 3 In (e), the electronic device can display video 226, video 227, video 228, etc. generated for the picture 221 according to the prompt information 1. It should be understood that the video 226, video 227, and video 228 can be videos with sound effects.

[0139] The prompt information 1 may include audio content and sound effect type, so that the electronic device can generate an audio file with sound effects according to the prompt information 1. The electronic device can also generate a video clip related to the picture 221 according to the picture identification number (ID) and / or picture content of the picture 221 selected by the user. The videos 226, 227 and 228 can be used as candidate sound effect videos for the user to select.

[0140] In this case, the prompt information obtained by the electronic device includes the above-mentioned picture 221 and prompt information 1.

[0141] The sound effect type may include emotional sound effects, spatial sound effects, etc. For example, the emotional sound effects may include joyful sound effects, neutral sound effects, sad sound effects, etc., and the spatial sound effects may include stereo sound effects, which may also be called three-dimensional (3D) sound effects.

[0142] For example, the prompt information 1 may be "Come out and eat hot pot together, stereo sound", then the audio content is "Come out and eat hot pot together", and the sound effect type is "stereo sound effect".

[0143] In other examples, the electronic device may also display only one candidate sound effect video, and the embodiment of the present application does not limit the number of the candidate sound effect videos.

[0144] In other examples, after the user inputs audio content in the input box, the electronic device can also automatically pop up supported sound effect type options for the user to select, thereby facilitating the user to create sound effect videos.

[0145] In other examples, when the user long presses the video 226, the content of the video 226 can also be previewed. For example, when the electronic device detects that the user long presses the video 226, a preview display interface of the video 226 can also be displayed.

[0146] When the electronic device detects that the user clicks on the video 226, the following may be displayed: Figure 3 GUI shown in (f) in .

[0147] It should be understood that when the electronic device detects the user clicking on the video 226, it can be determined that the user has selected the video 226 as the final generated sound effect video, and the electronic device can merge the audio with the sound effect and the generated video 226 to obtain a video with the sound effect. At this time, the electronic device can send the video 226 to another user as an instant message.

[0148] Alternatively, when the electronic device displays the video 226, the video 226 has been integrated with the audio with sound effects.

[0149] See also Figure 3 In (f), the communication message display area of ​​the electronic device includes the video 226 sent by the current user.

[0150] When the user who sent the instant message clicks on the video 226, the electronic device can play the video with sound effects.

[0151] In this way, users can create various types of audio and video effects and send them to each other in the instant messaging scenario with only a few operations. This technical solution provides users with rich communication materials, thereby enriching the content of instant messaging and enhancing the fun of instant messaging.

[0152] Figure 4 is a schematic diagram of another set of GUIs provided in an embodiment of the present application. Figure 4 (a) to (f) show the process of a user using an electronic device to generate audio and video in an IM scenario.

[0153] It should be understood that Figure 4 For (a) to (d) in Figure 3 For the sake of brevity, the relevant descriptions of (a) to (d) in are not repeated here.

[0154] When the electronic device detects that the user clicks the OK button, it can display the following Figure 4 GUI shown in (e) in .

[0155] See also Figure 4 In (e), the electronic device can display video 226, video 227, video 228 and corresponding audio 229 with sound effects generated for picture 221 according to prompt information 1. It should be understood that video 226, video 227 and video 228 can be videos with audio or without audio.

[0156] When the electronic device detects that the user clicks on the video 227, the following Figure 4 GUI shown in (f) in .

[0157] In other examples, when the electronic device detects that the user long presses the video 227, a preview display interface of the video 227 can also be displayed.

[0158] See also Figure 4 In (f), the communication message display area of ​​the electronic device includes the video 227 sent by the current user and the corresponding sound effect audio 2291.

[0159] In some examples, the video 227 also incorporates audio 2291 . When a user clicks on the video 227 , the electronic device can play the video 227 and can hear the corresponding sound effects.

[0160] For example, if the sound effect is a stereo sound effect, when the electronic device plays the video 227, the user can identify the direction of the sound source and experience the sense of space, giving the user an immersive feeling.

[0161] In other examples, the video 227 may not include the audio 2291. In this case, the user can click on the audio 2291 to make the electronic device play the audio 2291. In this way, the user receiving the instant message does not need to let his electronic device play the video 227, and can also obtain the corresponding audio content.

[0162] Based on the embodiments of the present application, users can create various types of audio and video effects in the instant messaging scenario with only a few operations. This technical solution provides users with rich communication materials, thereby enriching the content of instant messaging and enhancing the fun of instant messaging.

[0163] In other cases, the electronic device can also add audio with sound effects to the picture or video according to the user's operation. Figure 5 Introduce the technical solution.

[0164] Figure 5 It is a schematic diagram of another set of GUIs provided in an embodiment of the present application.

[0165] It should be understood that Figure 5 For (a) and (b) in Figure 3 For the sake of brevity, the descriptions of (a) and (b) are not repeated here.

[0166] When the electronic device detects that the user has long pressed the image 221, the following display may be displayed: Figure 5 In some other examples, the picture 221 may also include a creation button, and when the electronic device detects that the user clicks the creation button, it may also display the following Figure 5 GUI shown in (c) in FIG.

[0167] See also Figure 5 In (c), a tab sheet 231 may be displayed on the picture 221, and the tab sheet 231 may include an option to add a sound effect.

[0168] In other examples, the tab sheet may also include multiple options. For example, the tab sheet 231 may include an audio and video creation option and an add audio effect option.

[0169] When the electronic device detects that the user clicks the tab sheet 231 and inputs the prompt information 2, the following may be displayed: Figure 5 GUI shown in (d).

[0170] See also Figure 5 In (d), the prompt information 2 can be displayed in the input box of the electronic device.

[0171] Exemplarily, the prompt information 2 may be of the same type as the content included in the prompt information 1. For example, the prompt information 2 may include audio content and sound effect type. For details, please refer to the relevant description in the above text.

[0172] When the electronic device detects that the user clicks the confirmation button, it can display the following Figure 5 GUI shown in (e) in .

[0173] See also Figure 5 In (e), the electronic device can display the picture 221 and the pictures 233 and 234 generated according to the prompt information 2 and the corresponding audio 235 with sound effects. It should be understood that the pictures 233 and 234 can be pictures related to the picture content of the picture 221. For example, the pictures 233 and 234 belong to the same series as the picture 221.

[0174] When the electronic device detects that the user clicks on the picture 221, the following Figure 5 GUI shown in (f) in .

[0175] See also Figure 5 In (f), the communication message display area of ​​the electronic device includes a picture 221 and corresponding audio 2351 sent by the current user.

[0176] Thus, the user who receives the instant message can click on the audio 2351 to listen to the audio file with corresponding sound effects.

[0177] It is understandable that electronic devices can also use the same method as described above to add audio with sound effects to videos, emoticons, dynamic pictures, etc., and the embodiments of the present application are not limited to this.

[0178] Based on the embodiments of the present application, users can add various types of sound effects to multimedia files such as pictures, videos, and dynamic pictures, thereby enriching the content of instant messaging and enhancing the fun of communication.

[0179] In some cases, when an electronic device generates a video with stereo sound, in order for the video that is integrated with audio of the stereo sound to present the stereo sound, the electronic device needs to identify the sound source from the video screen and track the position of the sound source during video playback to obtain the sound source trajectory information.

[0180] It should be understood that when the sound source is a common sound source such as a person, vehicle, animal, airplane, etc., the electronic device can call the sound source detection algorithm to obtain the sound source in the video screen. However, when the above common sound sources do not exist in the video screen, the electronic device may not be able to obtain the sound source. At this time, the user needs to manually select the sound source in the first frame of the video screen. Subsequently, the electronic device can track the sound source according to the user's selection to obtain the sound source trajectory information. Figure 6 Introduce the technical solution.

[0181] Figure 6 It is a schematic diagram of another set of GUIs provided in an embodiment of the present application.

[0182] It should be understood that Figure 6 For (a) to (d) in Figure 3 For the sake of brevity, the relevant descriptions of (a) to (d) in are not repeated here.

[0183] When the electronic device detects that the user clicks the OK button, it can display the following Figure 6 GUI shown in (e) in .

[0184] See also Figure 6In (e), the electronic device can display video 226, video 227, video 228, etc. generated for picture 221 according to prompt information 3. It should be understood that video 226, video 227, and video 228 may include audio, but at this time the audio does not have sound effects.

[0185] When the electronic device detects that the user clicks on the video 227, the following Figure 6 GUI shown in (f) in .

[0186] See also Figure 6 In (f), the electronic device may display a prompt box 236, which may include text content "Please click on the target sound source".

[0187] It is understandable that the embodiment of the present application does not limit the specific content included in the prompt box 236. In other examples, the text content included in the prompt box 236 can also be "No sound source is identified, please click the target sound source" and the like.

[0188] The user can select a sound source in the video 227 to instruct the electronic device to identify the selected target as the sound source and identify its trajectory. After the electronic device detects the user's operation of selecting a target in the video screen of the video 227, the electronic device can display the following information: Figure 6 GUI shown in (g).

[0189] See also Figure 6 In (g), a display frame 2271 appears in the video 227 according to the user's selection operation.

[0190] When the electronic device detects that the user's selection operation is completed, the following display may be displayed: Figure 6 Alternatively, after the user's selection operation is completed, the electronic device detects that the user clicks the confirmation button, and then displays the following Figure 6 GUI shown in (h).

[0191] See also Figure 6 In (h), the communication message display area of ​​the electronic device includes a video 226 sent by the current user. The video 226 is a video with stereo sound effects.

[0192] Exemplarily, the electronic device may generate audio with stereoscopic sound effects through a spatial audio algorithm based on the audio and the acquired sound source trajectory information, and then fuse the audio with stereoscopic sound effects with the video 226 to generate a video with stereoscopic sound effects.

[0193] When the electronic device detects that the user clicks on the video 226 , the video with stereo sound effect can be played.

[0194] In this way, when the electronic device does not detect the sound source in the generated video screen, it can identify the sound source according to the user's prompt, and further track the sound source to obtain the sound source track information, and fuse the previously generated audio with the sound source track to generate audio with stereo sound effects. After that, the audio with stereo sound effects is fused with the video to generate a video with stereo sound effects. This technical solution can generate a video with stereo sound effects, so that the user can identify the direction of the sound source and experience the sense of space, giving the user an immersive feeling.

[0195] In some cases, the prompt information input by the user may be only text content, and the text content may include the prompt content of the video screen, the audio content and the sound effect type. The electronic device may generate a video with sound effects according to the prompt information. Figure 7 Introduce the technical solution.

[0196] Figure 7 It is a schematic diagram of another set of GUIs provided in an embodiment of the present application.

[0197] It should be understood that Figure 7 For (a) to (b) in Figure 3 The relevant descriptions from (a) to (b) in .

[0198] When the electronic device detects the user inputting the prompt information 4, it can display Figure 7 GUI shown in (c) in FIG.

[0199] Exemplarily, the prompt information 4 may be “an eagle is staring at the rabbit opposite, come out and eat hot pot together, stereo sound effect”, then the prompt content of the video screen in the prompt information 4 is “an eagle is staring at the rabbit opposite, come out and eat hot pot together”, the audio content is “come out and eat hot pot together”, and the audio type is “stereo sound effect”.

[0200] See also Figure 7 In (c), the above prompt information 4 is displayed in the input box of the electronic device.

[0201] When the electronic device detects that the user clicks the OK button, it can display the following Figure 7 GUI shown in (d).

[0202] The electronic device can parse the prompt information 4 to obtain the prompt content, audio content and sound effect type of the corresponding video screen.

[0203] In some implementations, the electronic device may send the prompt content and audio content of the video screen to the cloud server, and obtain the video, audio, and sound source trajectory information sent by the cloud server.

[0204] It should be understood that Figure 7For (d) and (e) in Figure 3 Relevant descriptions of (e) and (f) in .

[0205] Based on the embodiments of the present application, the electronic device can generate corresponding audio and video effects according to the prompt information input by the user, thereby enhancing the fun of communication and the diversity of creating audio and video effects.

[0206] Figure 8 This is a system framework diagram to which the method for generating audio and video effects of the present application can be applied. Figure 8 The system may include an electronic device 300a and an artificial intelligence (AI) platform 300b.

[0207] Exemplarily, the electronic device 300a may include at least an instruction acquisition module 310, an instruction parsing module 320, a sound effect rendering module 330, and a synthesis module 340. The AI ​​platform 300b may include at least an audio generation module 350, a video generation module 360, and a calculation module 370.

[0208] The instruction acquisition module 310 may be used to acquire prompt information input by the user, and the prompt information may include text, pictures, videos, etc. The prompt information may include prompt content for generating video images, audio content, and sound effect types, etc. For example, the prompt information may be prompt information 1, prompt information 2, prompt information 4, etc., as mentioned above.

[0209] The instruction parsing module 320 can parse the acquired prompt information by the user to obtain the prompt content, audio content and sound effect type of the corresponding video screen.

[0210] Exemplarily, the prompt information can be edited based on the format of regular expressions, and the instruction parsing module 320 can parse the above prompt information based on regular expressions. For example, the prompt information is [an eagle staring at the rabbit opposite, "Let's have a hot pot together tonight" | stereo sound], then the ordinary text "an eagle staring at the rabbit opposite" in the prompt information is the video screen prompt content, the quoted text content "Let's have a hot pot together tonight" is the audio content, and the sound effect type is after the "|" separator. Alternatively, the special field "-keyword" in the prompt information is used as a separator, such as the prompt information is [-image: an eagle staring at the rabbit opposite -voice: "Let's have a hot pot together tonight" -effect: stereo sound]. Alternatively, the user can also directly select pictures, videos, animated images, etc. as video screen prompt content, and the audio content and audio type are entered in text form.

[0211] The electronic device 300a can send the prompt content, audio content and sound effect type of the video screen obtained by parsing the prompt information by the instruction parsing module 320 to the AI ​​platform 300b. The video generation module 360 ​​in the AI ​​platform 300b can generate a corresponding video according to the prompt content of the video screen, and the audio generation module 350 can generate a corresponding audio with sound effects according to the audio content and sound effect type.

[0212] Exemplarily, the AI ​​platform 300b may be a cloud server. The audio generation module 350 may synthesize the corresponding audio using speech synthesis technology for the audio content, and then adjust the speed, pitch, timbre, etc. of the audio based on signal processing technology to obtain audio with sound effects. Alternatively, the audio generation module 350 may also generate the corresponding audio with sound effects using emotional speech synthesis technology, text-to-speech (TTS) technology, or a text prompt to generate an audio model.

[0213] Exemplarily, the video generation module 360 ​​can use a semantic vector retrieval model according to the prompt content of the video screen, and obtain a preset number of video clips that are most relevant to the prompt content through a video material library, such as the preset number can be 3 or 1.

[0214] The semantic vector retrieval model first uses a deep learning model to convert unstructured data (such as text, images, videos, audio, etc.) into vector representations, and then uses the similarity or distance between vectors for retrieval.

[0215] In an embodiment of the present application, the semantic vector retrieval model first converts the prompt content of the video screen into a content representation vector; then, the content representation vector is compared with the representation vector of the video frame in the video material library to calculate the similarity between them; and a number of video frames are selected in order of similarity from high to low, and the corresponding video clips are located according to the videos and timestamps to which they belong, and the video clips are the final generated video.

[0216] It is understandable that the semantic vector retrieval model can use Euclidean distance, cosine similarity, Hamming distance, etc. to calculate the above similarity.

[0217] The video material library can be TV series clips, movies, etc.

[0218] In other examples, when the sound effect type is a stereo sound effect, it is also necessary to determine the sound source track of the sound source in the video determined by the video generation module. For example, the calculation module 370 can use a sound source target detection algorithm to determine the sound source in the video screen, and then use a target tracking algorithm to track the position of the sound source in each frame of the video screen, so as to obtain the sound source track of the sound source. Alternatively, the calculation module 370 can also use an active speaker judgment model to locate the speaker of the common dialogue and other shots in the video clip to determine the corresponding sound source.

[0219] For example, the target tracking algorithm may include but is not limited to optical flow method, Kalman filtering, correlation filtering and other algorithms.

[0220] In other examples, when the target detection algorithm cannot determine the sound source in the video screen, the user can select or click on the sound source in the first frame of the video screen, and the subsequent calculation module 370 can use the target tracking algorithm to track the position of the sound source in each frame of the video screen, thereby obtaining the sound source trajectory of the sound source.

[0221] In some examples, when the sound effect type is not a stereo sound effect, the AI ​​platform 300b can send the generated video and the audio with the sound effect to the electronic device 300a, and the synthesis module 340 in the electronic device 300a can fuse the video and the audio to obtain a video with the sound effect. Alternatively, the AI ​​platform 300 can also fuse the generated video and the audio with the sound effect into a video with the sound effect and send it to the electronic device 300a.

[0222] In some examples, when the sound effect type is stereo sound effect, the AI ​​platform 300b can send the generated video, audio and acquired sound source trajectory information to the electronic device 300a, and the sound effect rendering module 330 in the electronic device 300a can fuse the audio and sound source trajectory information to generate audio with stereo sound effect.

[0223] The sound rendering module 330 may calculate the audio with stereo sound effects using a spatial audio algorithm. For example, the spatial audio algorithm may be a distance-based amplitude panning (DBAP) algorithm, which no longer relies on the generalization area formed between the speakers, but calculates the gain coefficient of the speaker based on the distance between the sound source and the speaker, so that the listener perceives that when the sound source moves in space, its direction and distance change with the distance.

[0224] Alternatively, the AI ​​platform 300b may first fuse the audio and the acquired sound source trajectory information to obtain audio with stereo sound effects, and send the video and the audio with stereo sound effects to the electronic device 300a. The synthesis module 340 in the electronic device 300a fuses the video with the audio with stereo sound effects to obtain a video with stereo sound effects.

[0225] Alternatively, the AI ​​platform 300b may first fuse the audio and the acquired sound source trajectory information to obtain the audio with stereo sound effects, and fuse the video and the audio with stereo sound effects to obtain the video with stereo sound effects. Afterwards, the AI ​​platform 300b may send the video with stereo sound effects to the electronic device 300a, and the electronic device 300a may play the video with stereo sound effects.

[0226] It should be understood that the various functional modules included in the above electronic device 300a and the AI ​​platform 300b are only illustrative. In other examples, the electronic device 300a and the AI ​​platform 300b may also include more functional modules or fewer functional modules, or their various functional modules may be combined into one functional module, which is not limited in the embodiments of the present application.

[0227] In some other examples, the AI ​​platform 300b may also be located in the electronic device 300a as a part of the electronic device 300a, for example, the AI ​​platform 300b may be located in the central processing unit (CPU), or the graphic processing unit (GPU), or the neural network processing unit (NPU) of the electronic device 300a, which is not limited in the embodiments of the present application. Alternatively, the AI ​​platform 300b may also be located in another electronic device.

[0228] It can be understood that in the embodiments of the present application, audio can also be referred to as an audio file, and video can also be referred to as a video file, and the two have the same meaning.

[0229] The following will combine Figures 9 to 12 The present invention introduces the technical solution of the method for generating audio and video in the embodiment of the present application.

[0230] Fig. 9 is a schematic interactive diagram of a method for generating audio and video effects provided in an embodiment of the present application. Fig. 9 As shown, the method 400 may be implemented in an electronic device, and the method 400 may include steps 410 to 460.

[0231] Exemplarily, the first electronic device may be a mobile phone, a tablet, a laptop or other electronic device, and the second electronic device may be a server, a cloud server or other device.

[0232] 410. The first electronic device obtains prompt information.

[0233] Exemplarily, the prompt information may be one or more of text, picture, video, and dynamic picture, for example, the prompt information 1, picture 221, and prompt information 4 mentioned above.

[0234] The prompt information may include prompt content, audio content and sound effect type of the video screen. In other examples, the prompt information may also include other content.

[0235] This sound effect type may include emotional sound effects and spatial sound effects, and for details, please refer to the relevant description in the previous article.

[0236] 420. The first electronic device determines a first instruction according to the prompt information.

[0237] Exemplarily, the first electronic device may parse the prompt information to determine the prompt content, audio content, and sound effect type of the video screen in the prompt information.

[0238] In some other examples, step 410 and step 420 may also be in one step.

[0239] 430: The first electronic device sends a first instruction to the second electronic device. Correspondingly, the second electronic device receives the first instruction.

[0240] For example, the second electronic device may be a cloud server, and the first electronic device may send the first instruction to the second electronic device via a network.

[0241] 440. The second electronic device determines the corresponding video and the audio with sound effects according to the first instruction.

[0242] Exemplarily, the second electronic device may have an audio generation module and a video generation module. The audio generation module may be used to generate audio of corresponding sound effects, and the video generation module may be used to generate corresponding videos. For details, please refer to the relevant description in the above text.

[0243] 450, the second electronic device sends the video and the audio with sound effects to the first electronic device. Correspondingly, the first electronic device receives the video and the audio with sound effects.

[0244] Exemplarily, the number of the videos may be one or more. When the number of videos is more than one, each video corresponds to an audio with sound effects. Alternatively, multiple videos may also correspond to one audio with sound effects.

[0245] 460. The first electronic device fuses the video and the audio with the sound effect to generate a video with the sound effect.

[0246] When there are multiple videos, the first electronic device can merge the target video and the target audio with sound effects according to the user's selection to obtain the video with sound effects.

[0247] It should be understood that the manner in which the first electronic device fuses the video and the audio with sound effects can be found in the related description of the electronic device 300a in the foregoing text, and for the sake of brevity, it will not be repeated here.

[0248] It should be understood that after step 460, the user can view the generated video with sound effects on the first electronic device, or send the generated video with sound effects to another user in the form of an instant message, etc. Alternatively, the video with sound effects can also be sent to another user in the form of an emoticon package, etc.

[0249] It should also be understood that the specific execution order of the above steps 410-460 is not limited in the present embodiment. In other examples, some steps in steps 410-460 can be combined or not executed or replaced by other steps, which is not limited in the present embodiment.

[0250] Based on the embodiment of the present application, the first electronic device can obtain prompt information, and parse the prompt information to obtain the first instruction therein; then, the first electronic device can send the first instruction to the second electronic device; the second electronic device can determine the corresponding video and audio with sound effects according to the first instruction, and send the video and audio with sound effects to the first electronic device, and then the first electronic device can fuse the video and audio with sound effects to generate a video with sound effects. In this technical solution, the user only needs to input the corresponding prompt information, and the first electronic device can generate a video with the corresponding sound effect, so that the user can easily create various types of sound effect videos and enhance the fun of chatting or creation.

[0251] In other examples, the second electronic device may also fuse the video and the audio with sound effects to generate a video with sound effects. Afterwards, the second electronic device may send the video with sound effects to the first electronic device, and this technical solution may reduce the power consumption of the first electronic device.

[0252] Fig.10 is a schematic interactive diagram of another method for generating audio and video effects provided by an embodiment of the present application. Fig.10 As shown, the method 500 may be implemented in an electronic device, and the method 500 may include steps 510 to 580.

[0253] 510. The first electronic device obtains prompt information.

[0254] It should be understood that step 510 may refer to the relevant description of step 410 .

[0255] 520. The first electronic device determines a second instruction according to the prompt information, and the sound effect type included in the second instruction is stereo sound effect.

[0256] For example, after parsing the prompt information, the first electronic device determines that the sound effect type is stereo sound effect.

[0257] 530: The first electronic device sends a second instruction to the second electronic device. Correspondingly, the second electronic device receives the second instruction.

[0258] 540. The second electronic device determines the corresponding video and the audio without stereo sound effect according to the second instruction.

[0259] Exemplarily, the video generation module in the second electronic device can be used to generate the corresponding video. Since the generation of stereo sound effects requires obtaining the sound source track of the video, in this case, the audio generation module can first generate audio without stereo sound effects.

[0260] 550. The second electronic device determines a sound source trajectory according to the video.

[0261] Exemplarily, the computing module in the second electronic device can determine the sound source trajectory based on the video generated by the video generation module. For example, for video A, the computing module can first determine the sound source B in video A, and track the position of the sound source B in each frame of video A, so as to determine the sound source trajectory of the sound source B.

[0262] 560, the second electronic device sends the video, the audio without stereo sound effect, and the sound source track to the first electronic device. Correspondingly, the first electronic device receives the video, the audio without stereo sound effect, and the sound source track.

[0263] Exemplarily, the number of the video may be one or more. When the number of the video is more than one, each video corresponds to one audio. Alternatively, multiple videos may correspond to one audio.

[0264] 570. The first electronic device fuses the audio without stereo sound effect with the sound source track to generate audio with stereo sound effect.

[0265] For example, the audio rendering module in the first electronic device can fuse the audio without stereo sound effect with the sound source track to generate audio with stereo sound effect.

[0266] In other examples, the second electronic device may also fuse the audio without stereo sound effects with the sound source track, and after generating the audio with stereo sound effects, send the audio with stereo sound effects to the first electronic device.

[0267] 580. The first electronic device fuses the audio with stereo sound effect and the video to generate a video with stereo sound effect.

[0268] For example, the synthesis module of the first electronic device may fuse the audio with stereo sound effect and the video to generate a video with stereo sound effect.

[0269] In other examples, the second electronic device may also merge the audio and video with stereo sound effects to generate a video with stereo sound effects, and then send the video with stereo sound effects to the first electronic device.

[0270] It should be understood that after step 580, the user can view the generated video with stereo sound effects on the first electronic device, or send the generated video with stereo sound effects to another user in the form of an instant message, etc. Alternatively, the video with stereo sound effects can also be sent to another user in the form of an emoticon package, etc.

[0271] It should also be understood that the specific execution order of the above steps 510-580 is not limited in the embodiment of the present application. In other examples, some steps in steps 510-580 can also be combined or not executed or replaced by other steps, which is not limited in the embodiment of the present application. For example, steps 510 and 520 can be combined and executed, and steps 570 and 580 can be combined and executed.

[0272] Based on the embodiments of the present application, the user only needs to input corresponding prompt information, and the first electronic device can generate a corresponding stereo sound video, so that the user can identify the direction of the sound source and experience the sense of space when watching the video, giving the user an immersive feeling.

[0273] In some cases, when the performance of an electronic device is relatively strong, corresponding functional modules can be deployed inside the electronic device to generate corresponding video and audio without the need for a server, cloud server or other electronic devices.

[0274] Fig.11 FIG. 1 is a schematic flow chart of a method for generating audio and video effects provided in an embodiment of the present application. Fig.11 As shown, the method 600 may be applied to a first electronic device, and the method 600 may include steps 610 to 640.

[0275] 610. The first electronic device obtains prompt information.

[0276] 620. The first electronic device determines a third instruction according to the prompt information.

[0277] For steps 610 to 620 , reference may be made to the relevant descriptions of steps 410 to 420 .

[0278] 630. The first electronic device determines the corresponding video and the video with sound effects according to the third instruction.

[0279] Exemplarily, the first electronic device may include an audio generation module and a video generation module, which may be used to generate a corresponding video and a video with sound effects, respectively.

[0280] 640. The first electronic device fuses the video and the audio with the sound effect to generate a video with the sound effect.

[0281] Exemplarily, the first electronic device may include a synthesis module, which may be used to fuse the video and the audio with sound effects to generate the video with sound effects.

[0282] In other examples, when the sound effect type is stereo sound effect, the first electronic device may first generate two-dimensional audio, and then after obtaining the sound source trajectory of the sound source in the video, fuse the two-dimensional audio with the sound source trajectory to obtain audio with stereo sound effect; then, the first electronic device fuses the audio with stereo sound effect with the video to obtain a video with stereo sound effect.

[0283] Based on the embodiments of the present application, the first electronic device can generate a video with sound effects according to the prompt information input by the user, so that the user can easily create various types of sound effect videos and enhance the fun of chatting or creation.

[0284] It is understandable that the technical solutions of the above-mentioned embodiments may be combined with each other or partially combined to form a new technical solution, and such a solution should not be considered to exceed the protection scope of this application.

[0285] Fig.12 is a schematic interactive diagram of a method for generating audio and video effects provided in an embodiment of the present application. Fig.12 As shown, the method 700 may be applied in an electronic device, and the method 700 may include steps 710 to 760.

[0286] The first electronic device may be a smart phone, a tablet or other electronic device, and the second electronic device may be a server, a computer or other electronic device with certain computing power.

[0287] 710. The first electronic device obtains prompt information for generating an audio effect video.

[0288] For example, the first electronic device can obtain prompt information for generating audio and video effects by detecting information input by the user in the instant messaging scenario, such as Figure 3 shown.

[0289] In another example, the first electronic device can also obtain prompt information for generating a sound effect video in a video editing and creation application such as Petal Edit.

[0290] For example, the prompt information may include information indicating the prompt content of the video screen, information indicating the audio content, and information indicating the type of sound effect. For details, please refer to the relevant description in the above text.

[0291] 720. The first electronic device determines a first instruction according to the prompt information, where the first instruction includes an instruction for indicating the prompt content, audio content, and sound effect type of the video screen.

[0292] The first electronic device may parse the prompt information to obtain instructions for indicating prompt content of the video screen, audio content, and sound effect type.

[0293] The sound effect type may be an emotional sound effect, a stereo sound effect, an environmental sound effect, etc. The embodiment of the present application does not limit the specific form of the sound effect type.

[0294] 730. The first electronic device sends a first instruction to the second electronic device.

[0295] Exemplarily, the first electronic device may send the first instruction to the second electronic device via a network. Alternatively, the first electronic device may send the first instruction to the second electronic device via Bluetooth, Wi-Fi, etc., which is not limited in the present embodiment.

[0296] 740. The second electronic device determines the corresponding first video and first audio according to the first instruction.

[0297] The second electronic device determines the corresponding first video and first audio according to the first instruction, which can be understood as the second electronic device retrieving the corresponding first video and first audio from the material database, or the second electronic device generates the corresponding first video and first audio using a neural network model.

[0298] Exemplarily, the second electronic device may generate the corresponding first video through a video generation module, and generate the corresponding first audio through an audio generation module.

[0299] In other examples, the electronic device may also first generate a first video, and generate a first audio based on the generated first video, thereby improving the consistency between the video image and the audio.

[0300] The number of the first video may be one or more, for example, it may be multiple related video clips. The number of the first audio may be one or more. Alternatively, the number of the first video may be multiple, and the number of the first audio may be one. This is not limited in the embodiments of the present application.

[0301] At 750, the second electronic device sends the first video and the first audio to the first electronic device.

[0302] 760. The first electronic device generates a video with sound effects according to the first video and the first audio.

[0303] After receiving the first video and the first audio, the first electronic device may fuse the first video and the first audio to generate a video with sound effects. The manner in which the first electronic device fuses the first video and the first audio can refer to the relevant description above.

[0304] Based on the embodiment of the present application, the first electronic device can obtain prompt information for generating a sound effect video, and determine a first instruction including prompt content of the video screen, audio content and sound effect type based on the prompt information, and send the first instruction to the second electronic device; receive the first video and first audio sent by the second electronic device, and generate a video with sound effects based on the first video and the first audio.

[0305] In this way, users can use electronic devices to conveniently create various types of audio and video effects, thereby enhancing the fun and diversity of user chats or creations.

[0306] In some embodiments, when the sound effect is an emotional sound effect, the first audio is an audio with an emotional sound effect, wherein generating a video with the sound effect according to the first video and the first audio includes:

[0307] The first electronic device generates a video with the emotional sound effect according to the first video and the audio with the emotional sound effect.

[0308] For example, the emotional sound effect may be a joyful sound effect, a sad sound effect, etc.

[0309] In an embodiment of the present application, the electronic device can generate a video with emotional sound effects, so that the video can express the user's tone, emotion, atmosphere and other information, thereby enhancing the richness of the user's expression.

[0310] In some embodiments, when the sound effect is a stereo sound effect, the first audio is an audio without a sound effect, and the method 700 further includes:

[0311] The first electronic device receives a sound source trajectory sent by the second electronic device, where the sound source trajectory is a movement trajectory of the sound source in the first video; wherein generating a video with sound effects according to the first video and the first audio includes:

[0312] The first electronic device generates audio with stereo sound effects based on the audio without sound effects and the sound source trajectory;

[0313] The first electronic device generates a video with a stereo sound effect according to the first video and the audio with a stereo sound effect.

[0314] Exemplarily, the first electronic device generates audio with stereo sound effects based on audio without sound effects and sound source tracks, and generates a video with stereo sound effects based on the first video and the audio with stereo sound effects, as described in the foregoing.

[0315] Based on the embodiment of the present application, when the sound effect is a stereo sound effect, the first audio received by the first electronic device may be audio without sound effects, and the first electronic device may also receive the sound source track of the sound source sent by the second electronic device, and generate audio with stereo sound effects according to the sound source track and the audio without sound effects, and then fuse the audio with stereo sound effects and the first video to generate a video with stereo sound effects. The sound source track is closely related to the content of the first video.

[0316] In this way, the electronic device can generate a video with stereo sound effects, and the stereo sound effects are closely linked to the video images, so that the user can identify the direction of the sound source and experience the sense of space, giving the user an immersive feeling.

[0317] In some embodiments, before receiving the sound source trajectory sent by the second electronic device, the method 700 further includes:

[0318] The first electronic device receives a request message sent by the second electronic device, where the request message is used to request to determine a sound source in the first video;

[0319] The first electronic device sends indication information to the second electronic device, where the indication information is used to indicate a sound source in the first video.

[0320] For example, see Figure 6 In (f) and (g), after receiving the request message, the first electronic device can display a display frame 236 in the display interface to prompt the user to determine the sound source in the first video.

[0321] After the first electronic device detects that the user selects the sound source 2271 in the video 227, it can send indication information to the second electronic device.

[0322] In the embodiment of the present application, when the second electronic device cannot determine the sound source, it can send a request message to the first electronic device so that the first electronic device determines the sound source according to the user's operation, and then the first electronic device can send indication information to the second electronic device to indicate the sound source in the first video. This technical solution is conducive to the second electronic device to determine the sound source and further determine the sound source trajectory.

[0323] In other examples, the second electronic device may also determine the sound source through a target detection algorithm, and further determine the trajectory of the sound source.

[0324] In some embodiments, the method 700 further includes:

[0325] The first electronic device displays or sends a video with sound effects.

[0326] For example, see Figure 3 In (e) and (f), the electronic device can send the video 226 with sound effects to a user on another electronic device.

[0327] In an embodiment of the present application, in an instant messaging scenario, the electronic device can display the video with sound effects, or send the video with sound effects to another electronic device, thereby improving the diversity of materials for user communication.

[0328] In some embodiments, the prompt information includes one or more of a picture, a video, a text, a dynamic picture, and an audio. In other examples, the prompt information may also include other content, which is not limited in the embodiments of the present application.

[0329] In some embodiments, when the sound effect is a stereo sound effect, the first audio is an audio without a sound effect, and the method 700 further includes:

[0330] The second electronic device determines the sound source of the sound in the first video;

[0331] The second electronic device determines a sound source trajectory of the sound source in the first video;

[0332] The second electronic device sends the sound source trajectory to the first electronic device.

[0333] Based on the embodiment of the present application, when the sound effect is a stereo sound effect, the second electronic device can first generate the first video and the audio without the sound effect, then determine the sound source in the first video and the sound source track of the sound source in the first video, and send the sound source track to the first electronic device. This is conducive to the first electronic device synthesizing the audio with a stereo sound effect.

[0334] In some embodiments, when the sound effect is a stereo sound effect, generating the corresponding first video and first audio according to the first instruction includes:

[0335] The second electronic device generates the first video and the audio without sound effects according to the first instruction;

[0336] The second electronic device determines the sound source of the sound in the first video;

[0337] The second electronic device determines a sound source trajectory of the sound source in the first video;

[0338] The second electronic device generates the first audio with stereo sound effect according to the sound source trajectory and the audio without sound effect.

[0339] Based on the embodiment of the present application, after determining the sound source and the sound source track in the first video, the second electronic device can directly fuse the sound source track and the audio without sound effects to generate the first audio with stereo sound effects, thereby saving the computing power and power consumption of the first electronic device.

[0340] In some embodiments, determining a sound source in the first video includes:

[0341] The second electronic device sends a request message to the first electronic device, where the request message is used to request to determine the sound source in the first video;

[0342] The second electronic device receives the indication information sent by the first electronic device, where the indication information is used to indicate the sound source in the first video;

[0343] The second electronic device determines the sound source in the first video according to the instruction information.

[0344] Based on the embodiments of the present application, when the second electronic device cannot directly determine the sound source in the first video, it can also send a request message to the first electronic device to determine the sound source, and receive indication information sent by the first electronic device to indicate the sound source, and determine the sound source in the first video according to the indication information.

[0345] In this technical solution, the user can specify the sound source in the first video, thereby improving the diversity of the sound effect video.

[0346] The embodiment of the present application also provides a method for generating a video with sound effects. The method is applied to a first electronic device, and the method may include: obtaining prompt information for generating a video with sound effects; determining a third instruction according to the prompt information, wherein the third instruction includes instructions for indicating the prompt content, audio content, and sound effect type of the video screen; determining a second video and a second audio with sound effects according to the third instruction, wherein the content of the second video is related to the prompt content of the video screen, the content of the second audio is related to the audio content, and the sound effect of the second audio is related to the sound effect type; generating a video with sound effects according to the second video and the second audio.

[0347] The number of the second video and the second audio can be one or more.

[0348] Based on the embodiment of the present application, when the first electronic device has sufficient computing power, the first electronic device can generate a related second video and a second audio with sound effects according to the prompt information input by the user, and merge the second video and the second audio to generate a video with sound effects.

[0349] In other examples, the first electronic device may also directly generate a video with sound effects according to the prompt information input by the user, which is not limited in the embodiments of the present application.

[0350] Fig.13 is a schematic block diagram of an electronic device provided in an embodiment of the present application. Fig.13 As shown, the electronic device 800 may include one or more processors 810; one or more memories 820; the one or more memories 820 store one or more instructions, and when the instructions are executed by one or more processors 810, the method for generating sound effect video as described in any possible implementation method in the foregoing text is executed.

[0351] Exemplarily, the device 800 may be the electronic device 100, the electronic device 200, the first electronic device, the second electronic device, etc. mentioned above.

[0352] An embodiment of the present application also provides a chip, including a processor and a communication interface, the communication interface is used to receive signals and transmit the signals to the processor, the processor processes the signals so that the method for generating audio and video as described in any possible implementation method described above is executed.

[0353] The chip can be a chip system or an independent chip, etc.

[0354] This embodiment also provides a readable storage medium, which stores instructions. When the instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method for generating audio and video effects in the above-mentioned embodiment.

[0355] This embodiment also provides a program product. When the program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the method for generating audio and video effects in the above-mentioned embodiment.

[0356] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store instructions, and when the device is running, the processor can execute the instructions stored in the memory so that the device executes the method of generating audio and video in the above-mentioned method embodiments.

[0357] Among them, the equipment, readable storage medium, program product or device provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0358] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0359] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0360] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0361] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0362] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0363] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0364] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating a sound effect video, characterized in that: The method is applied to a first electronic device, and the method includes: Get the prompt information for generating audio and video effects; Determine a first instruction according to the prompt information, wherein the first instruction includes an instruction for indicating prompt content, audio content, and sound effect type of a video screen; Sending the first instruction to a second electronic device; receiving a first video and a first audio sent by the second electronic device, wherein the content of the first video is related to the prompt content of the video screen, and the content of the first audio is related to the audio content; A video with sound effects is generated according to the first video and the first audio, wherein the sound effects are related to the sound effect type.

2. The method according to claim 1, characterized in that When the sound effect is an emotional sound effect, the first audio is an audio having the emotional sound effect, wherein generating a video having the sound effect according to the first video and the first audio includes: A video with the emotional sound effect is generated according to the first video and the audio with the emotional sound effect.

3. The method according to claim 1, characterized in that When the sound effect is a stereo sound effect, the first audio is an audio without a sound effect, and the method further includes: receiving a sound source trajectory sent by the second electronic device, where the sound source trajectory is a movement trajectory of the sound source in the first video; The step of generating a video with sound effects according to the first video and the first audio includes: Generating audio with stereo sound effects according to the audio without sound effects and the sound source trajectory; A video with stereo sound effect is generated according to the first video and the audio with stereo sound effect.

4. The method according to claim 3, characterized in that Before receiving the sound source track sent by the second electronic device, the method further includes: receiving a request message sent by the second electronic device, wherein the request message is used to request to determine a sound source in the first video; Send indication information to the second electronic device, where the indication information is used to indicate a sound source in the first video.

5. The method according to any one of claims 1 to 4, characterized in that The step of obtaining prompt information for generating a sound effect video includes: In an instant messaging scenario, prompt information for generating audio and video effects is obtained in response to a user's operation.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Display or send the video with the sound effect.

7. The method according to any one of claims 1 to 6, characterized in that The prompt information includes one or more of pictures, videos, texts, audios, and dynamic pictures.

8. A method for generating a sound effect video, characterized in that: The method is applied to a second electronic device, and the method includes: Receiving a first instruction sent by a first electronic device, wherein the first instruction includes an instruction for indicating prompt content, audio content, and sound effect type of a video screen; Generate a corresponding first video and first audio according to the first instruction, wherein the content of the first video is related to the prompt content of the video screen, and the content of the first audio is related to the audio content; The first video and the first audio are transmitted to the first electronic device.

9. The method according to claim 8, characterized in that When the sound effect is a stereo sound effect, the first audio is an audio without a sound effect, and the method further includes: Determining a sound source in the first video; Determining a sound source trajectory of the sound source in the first video; The sound source trajectory is sent to the first electronic device.

10. The method according to claim 8, characterized in that When the sound effect is a stereo sound effect, generating a corresponding first video and first audio according to the first instruction includes: generating the first video and the audio without sound effects according to the first instruction; Determining a sound source in the first video; Determining a sound source trajectory of the sound source in the first video; A first audio with a stereo sound effect is generated according to the sound source track and the audio without the sound effect.

11. The method according to claim 9 or 10, characterized in that: The determining the sound source of the first video includes: Sending a request message to the first electronic device, wherein the request message is used to request to determine a sound source in the first video; Receiving indication information sent by the first electronic device, where the indication information is used to indicate a sound source in the first video; A sound source in the first video is determined according to the indication information.

12. A method for generating a sound effect video, characterized in that: The method is applied to a first electronic device, and the method includes: Get the prompt information for generating audio and video effects; Determine a third instruction according to the prompt information, wherein the third instruction includes an instruction for indicating the prompt content, audio content and sound effect type of the video screen; Determine a second video and a second audio with sound effects according to the third instruction, wherein the content of the second video is related to the prompt content of the video screen, the content of the second audio is related to the audio content, and the sound effect of the second audio is related to the sound effect type; A video with sound effects is generated based on the second video and the second audio.

13. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by one or more processors, the method for generating sound effect video as described in any one of claims 1-12 is executed.

14. A chip, characterized in that: The chip includes a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal so that the method for generating audio and video as described in any one of claims 1-12 is executed.

15. A readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on a device, the method for generating a sound effect video as described in any one of claims 1 to 12 is executed.

Images