Method and system for replacing cover image of dynamic photo and electronic equipment

Abstract

The present application relates to the field of electronic devices, and more particularly, to a method and system for replacing a cover image of a dynamic photo, and an electronic device, the method comprising: loading a first network parameter to a neural network according to a first cover image of a first dynamic photo and a first video frame corresponding to the first cover image, the first dynamic photo comprises N video frames, the N video frames comprise the first video frame, and N is an integer greater than or equal to 2; in response to a selection operation of the user, determining a second video frame in the N video frames; inputting the second video frame into the neural network to generate a second cover image; and replacing the first cover image with a second cover image. The neural network is trained according to the original cover image and the corresponding video frame, so that the neural network is more adaptive to the video frame in the first dynamic photo, the image quality of the second cover image is similar to or the same as that of the original cover image, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of electronic devices, and more particularly, to a method, a system, and an electronic device for replacing a cover image of a live photo. Background Art

[0002] A live photo is a special form of photo, which consists of a high-definition cover image and a video of about 3 seconds. The cover image of the live photo is a high-definition and high-resolution image obtained through image signal processing (ISP) of the photo stream image signal after the electronic device responds to the user's operation of taking a photo. The video of the live photo is a low-resolution video obtained by caching the scenes 1.5 seconds before and after the electronic device responds to the user's operation of taking a photo and passing it through the video stream ISP. Compared with ordinary photos, the live photo has the scene information of a few seconds before and after taking the photo. The user can reselect the cover image in the video of the live photo. However, the resolution of the video of the live photo is reduced, and there are differences in effects in terms of noise details, dynamic range, color brightness, etc. compared with the original cover image, which will cause a large difference in the image quality between the reselected cover image and the original cover image, reducing the user experience. Therefore, how to ensure that the replaced cover image has the same or similar image quality as the original cover image has become an urgent technical problem to be solved. Summary of the Invention

[0003] This application provides a method, a system, and an electronic device for replacing a cover image of a live photo, which can generate a cover image with similar or the same image quality as the original cover image to replace the original cover image, helping to improve the user experience.

[0004] In a first aspect, a method for replacing a cover image of a live photo is provided. The method includes: loading first network parameters into a neural network according to a first cover image of a first live photo and a first video frame corresponding to the first cover image, where the first live photo includes N video frames, the N video frames include the first video frame, N≥2 and is an integer; determining a second video frame among the N video frames in response to a user's selection operation; inputting the second video frame into the neural network to generate a second cover image; and using the second cover image to replace the first cover image.

[0005] In the embodiments of the present application, a neural network can be trained based on the original cover image, i.e., the first cover image, and the video frame corresponding to the first cover image, and the second video frame can be input into the trained neural network according to the user's selection to generate the second cover image. Since the neural network is trained based on the original cover image and the corresponding video frame, the neural network will be more adapted to the video frames in the first live photo. Therefore, the image quality of the second cover image will be similar to or the same as that of the original cover image, which helps to improve the user experience.

[0006] In combination with the first aspect, in some implementation manners of the first aspect, before loading the first network parameter to the neural network according to the first cover image of the first live photo and the first video frame corresponding to the first cover image, the method further includes: generating a first live photo in response to the user's shooting operation.

[0007] In combination with the first aspect, in some implementation manners of the first aspect, the method further includes: saving the metadata corresponding to the first live photo.

[0008] In combination with the first aspect, in some implementation manners of the first aspect, loading the first network parameter to the neural network according to the first cover image of the first live photo and the first video frame corresponding to the first cover image includes: loading the first network parameter to the neural network according to the first cover image, the first video frame, and the metadata.

[0009] In combination with the first aspect, in some implementation manners of the first aspect, the metadata includes one or more of the following: electronic image stabilization (EIS) information, original image data, and ISO sensitivity.

[0010] In combination with the first aspect, in some implementation manners of the first aspect, the method is applied to an electronic device.

[0011] In combination with the first aspect, in some implementation manners of the first aspect, loading the first network parameter to the neural network according to the first cover image of the first live photo and the first video frame corresponding to the first cover image includes: the electronic device sending the first cover image and the first video frame to the server so that the server determines the first network parameter according to the first cover image and the first video frame; the electronic device receiving the first network parameter; and the electronic device loading the first network parameter to the neural network.

[0012] In addition, in the embodiments of the present application, the training of the model is placed on the server, which can make full use of the powerful computing power on the server side, not only improving the training speed but also taking into account the performance of the neural network.

[0013] In a second aspect, a system is provided, which includes an electronic device and a server. The electronic device is configured to send a first cover image of the first dynamic photo and a first video frame corresponding to the first cover image to the server. The first dynamic photo includes N video frames, and the N video frames include a first video frame and a second video frame, where N≥2 and N is an integer. The server is configured to: determine a first network parameter according to the first cover image and the first video frame; send the first network parameter to the electronic device; The electronic device is further configured to: determine a second video frame in response to a user's selection operation; load the first network parameter into a neural network; input the second video frame into the neural network to generate a second cover image; and replace the first cover image with the second cover image.

[0014] In the embodiments of the present application, a neural network can be trained according to the original cover image, that is, the first cover image, and the video frame corresponding to the first cover image, and the second video frame can be input into the trained neural network according to the user's selection to generate a second cover image. Since the neural network is trained according to the original cover image and the corresponding video frame, the neural network will be more adapted to the video frames in the first dynamic photo. Therefore, the image quality of the second cover image will be similar to or the same as that of the original cover image, which helps to improve the user experience.

[0015] In addition, in the embodiments of the present application, the training of the model is placed on the server, which can make full use of the powerful computing power on the server side, not only improving the training speed but also taking into account the performance of the neural network.

[0016] In combination with the second aspect, in some implementation manners of the second aspect, the electronic device is further configured to generate the first dynamic photo in response to a user's shooting operation.

[0017] In combination with the second aspect, in some implementation manners of the second aspect, the electronic device is further configured to save the metadata corresponding to the first dynamic photo.

[0018] In combination with the second aspect, in some implementation manners of the second aspect, the electronic device is specifically configured to send the first cover image, the first video frame, and the metadata to the server; the server is specifically configured to determine the first network parameter according to the first cover image, the first video frame, and the metadata.

[0019] In combination with the second aspect, in some implementation manners of the second aspect, the metadata includes one or more of the following: digital image stabilization (EIS) information, raw image data, and ISO sensitivity.

[0020] In a third aspect, an electronic device is provided, which includes one or more processors; one or more memories; the one or more memories store one or more computer programs, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the above aspect or any possible implementation manner of the above aspect to be executed.

[0021] In a fourth aspect, a computer-readable storage medium is provided, which includes a computer program or instructions that, when running on a computer, cause the method of the first aspect and any possible implementation method of the first aspect to be executed.

[0022] In a fifth aspect, a computer program product is provided, which includes a computer program or instructions that, when running on a computer, cause the method of the first aspect and any possible implementation method of the first aspect to be executed.

[0023] In a sixth aspect, a computer program is provided, which, when running on a computer, causes the method in the first aspect and any possible implementation manner thereof to be executed.

[0024] In a seventh aspect, an electronic device according to an embodiment of the present application includes a module / unit that executes the method of the above aspect or any possible design method of the above aspect; these modules / units can be implemented by hardware or by hardware executing corresponding software.

[0025] Among them, for the beneficial effects of the third aspect to the seventh aspect, please refer to the beneficial effects of the first aspect, and will not be repeated here. Description of the Drawings

[0026] Figure 1 is a schematic structural diagram of the electronic device provided by the embodiment of the present application.

[0027] Figure 2 is a software structure block diagram of the electronic device provided by the embodiment of the present application.

[0028] Figure 3 is a system architecture provided by the embodiment of the present application.

[0029] Figure 4 is a structural diagram of a convolutional neural network provided by the embodiment of the present application.

[0030] Figure 5 is a schematic flowchart of the method for replacing the cover image of a dynamic photo provided by the embodiment of the present application.

[0031] Figure 6 is a set of GUIs provided by the embodiment of the present application.

[0032] Figure 7 It is a schematic flowchart of a method for replacing the cover image of a live photo provided by an embodiment of the present application.

[0033] Figure 8 It is a schematic flowchart of a method for replacing the cover image of a live photo provided by an embodiment of the present application.

[0034] Figure 9 It is a schematic diagram of the composition of an electronic device provided by an embodiment of the present application. Specific implementation manners

[0035] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings.

[0036] The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and claims of the present application, the singular forms "a", "an", "the", "above", "said", "this" are also intended to include, for example, the expression "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" means one, two or more than two. The term "and / or" is used to describe the association relationship of associated objects and means that three relationships can exist; for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship.

[0037] Referring to "one embodiment" or "some embodiments" described in this specification means that specific features, structures or characteristics described in combination with the embodiment are included in one or more embodiments of the present application. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" and the like appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0038] The following describes an electronic device, a user interface for such an electronic device, and examples of using such an electronic device. In some embodiments, the electronic device may be a portable electronic device that also includes other functions such as a personal digital assistant and / or music player functions, such as a mobile phone, a tablet computer, a wearable electronic device with wireless communication functions (such as a smart watch), etc. Exemplary embodiments of the portable electronic device include, but are not limited to, those equipped with or other operating systems. The above portable electronic device may also be other portable electronic devices, such as a laptop computer, etc. It should also be understood that in some other embodiments, the above electronic device may not be a portable electronic device, but a desktop computer.

[0039] Exemplarily, Figure 1 FIG. shows a schematic structural diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0040] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0041] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0042] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0043] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0044] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0045] The wireless communication function of the electronic device 100 can be implemented by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modulation and demodulation processor, baseband processor, etc.

[0046] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0047] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0048] The modulation and demodulation processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to speaker 170A, receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor can be an independent device. In some other embodiments, the modulation and demodulation processor can be independent of the processor 110 and be disposed in the same device as the mobile communication module 150 or other functional modules.

[0049] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0050] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-CDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0051] Electronic device 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0052] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0053] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0054] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera sensor. The optical signal is converted into an electrical signal, and the camera sensor transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0055] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the sensor. The sensor can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The sensor converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0056] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0057] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0058] The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission pattern between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0059] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to achieve the data storage function. For example, files such as music and videos are saved in the external memory card.

[0060] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.). The data storage area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0061] The electronic device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and the application processor, etc. Such as music playback, recording, etc.

[0062] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0063] The speaker 170A, also known as a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or hands-free calls through the speaker 170A.

[0064] The receiver 170B, also known as an "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by placing the receiver 170B close to the human ear.

[0065] The microphone 170C, also known as a "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak close to the microphone 170C with the mouth to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0066] The pressure sensor 180A is used to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations with the same touch position but different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity greater than or equal to a first pressure threshold acts on the alarm application icon, the instruction to create a new alarm is executed.

[0067] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access to application locks, fingerprint photography, fingerprint answering of incoming calls, etc. For example, when the mobile phone detects a user's touch operation on the lock screen interface, the mobile phone can collect the user's fingerprint information through the fingerprint sensor 180H and match the collected fingerprint information with the pre-set fingerprint information in the mobile phone. If the match is successful, the mobile phone can enter the non-lock screen interface from the lock screen interface.

[0068] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from the display screen 194.

[0069] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers is through software interfaces. The following will be combined with Figure 2 to describe the modules involved in the embodiments of the present application. In some embodiments, the system can be divided into four layers, from top to bottom are the application layer, the application framework and service layer, the hardware abstraction layer (HAL), and the kernel layer. The application layer can include a series of application program packages.

[0070] Such as Figure 2 shown, the application layer can include a camera, settings, third-party applications, etc. Among them, the third-party applications can include a gallery, a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, music, a video, a short message, etc.

[0071] The application framework and service layer provide application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework and service layer can include some predefined functions.

[0072] Such as Figure 2 shown, the application framework and service layer can include a camera service, a media library, etc.

[0073] The camera service is used to notify the camera module to start shooting in order to save the cover image and video frames generated by the ISP module.

[0074] The media library supports the playback and recording of various common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0075] The application framework and service layer may also include a status monitoring service module, etc., such as a physical state recognition module for analyzing and recognizing user gestures; a sensor service module for monitoring the sensor data uploaded by various sensors in the hardware layer to determine the physical state of the electronic device 100.

[0076] The hardware abstraction layer is a software layer that shields the upper-layer applications from the underlying hardware. The upper-layer applications do not need to care about how the underlying hardware works and only need to provide a unified interface to the upper layer.

[0077] As Figure 2 shown, the hardware abstraction layer includes a camera module, an ISP module, a reselected cover image algorithm module, and a file system.

[0078] Among them, the camera module is used to obtain raw image data.

[0079] The ISP module is used to generate a cover image and video frames based on the raw image data.

[0080] The reselected cover image algorithm module is used to generate a new cover image.

[0081] The file system is used to manage various data in the system.

[0082] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0083] The hardware layer may include various sensors, such as Figure 1 the various sensors introduced in , the acceleration sensor, the gyroscope sensor, the touch sensor, etc. involved in the embodiments of the present application.

[0084] It should be noted that Figure 2Only taking one way of dividing the system framework as an example should not be construed as a specific limitation on the embodiments of the present application. In the embodiments of the present application, when the operating systems installed on the electronic device are different, different frameworks can be adopted for different operating systems. It can be understood that when different frameworks are adopted, the division methods of each layer of the framework, the specific naming, and the specific layer where each of the above modules is located can be different. For example, the hardware abstraction layer and the kernel layer can be collectively referred to as the kernel layer.

[0085] A live photo is a special form of photo, which consists of a high-definition cover image and a video of about 3 seconds. The cover image of the live photo is a high-definition and high-resolution image obtained through image signal processing (ISP) of the captured stream image after the electronic device responds to the user's operation of taking a photo. The video of the live photo is a low-resolution video obtained by caching the scenes 1.5 seconds before and after the electronic device responds to the user's operation of taking a photo and passing it through the video stream ISP. In other words, the resolution of the cover image of the live photo is higher than that of the video.

[0086] Compared with ordinary photos, live photos have scene information of 3 seconds before and after taking the photo, and the cover image can be reselected in the video of the live photo. However, as described above, the resolution of the video of the live photo is reduced, and there are differences in effects in terms of noise details, dynamic range, color brightness, etc. compared with the original cover image. This will cause a large difference in the image quality between the reselected cover image and the original cover image, reducing the user experience.

[0087] Based on this, the embodiments of the present application provide a method for replacing the cover image of a live photo. After the user reselects the cover image of the live photo, the cover image can be enhanced, so that the reselected cover image has the same or similar image quality as the original cover image, which helps to improve the user experience.

[0088] Before introducing the embodiments of the present application, first introduce several concepts that may be involved in the embodiments of the present application:

[0089] Image enhancement: Image enhancement is an important branch in the field of image processing. Through image enhancement technology, the image quality can be improved without re-collecting data to meet more practical application requirements. For example, image enhancement technology can purposefully emphasize the overall or local characteristics of an image (or video), make the original unclear image clear or emphasize certain interesting features, expand the differences between the features of different objects in the image, suppress uninteresting features, so as to improve the image quality, enrich the information volume, strengthen the image interpretation and recognition effects, and meet the needs of certain special analyses.

[0090] Neural Network: A neural network can be composed of neural units. A neural unit can refer to an arithmetic unit with x s and intercept 1 as inputs. The output of this arithmetic unit can be:

[0091]

[0092] where s = 1, 2, …… n, n is a natural number greater than 1, W s is the weight of x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting multiple such single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be a region composed of several neural units.

[0093] Deep Neural Network: A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Dividing the DNN according to the positions of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layers are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i + 1-th layer.

[0094] Although the DNN looks very complex, in terms of the work of each layer, it is actually not complex. Simply put, it is the following linear relationship expression: where, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also known as the coefficient), and α() is the activation function. Each layer simply performs such a simple operation on the input vector to obtain the output vector Due to the large number of layers in the DNN, the number of coefficients W and offset vectors is also relatively large. The definitions of these parameters in the DNN are as follows: Taking the coefficient W as an example: Suppose in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer where the coefficient W is located, and the subscript corresponds to the third-layer index 2 of the output and the second-layer index 4 of the input.

[0095] In summary, the coefficient from the k-th neuron in the (L - 1)-th layer to the j-th neuron in the L-th layer is defined as

[0096] It should be noted that there is no W parameter in the input layer. In a deep neural network, more hidden layers enable the network to better depict complex situations in the real world. In theory, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is a process of learning the weight matrix, and its ultimate goal is to obtain the weight matrices of all layers of the trained deep neural network (the weight matrix formed by vectors W of many layers).

[0097] Convolutional Neural Network: A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A convolutional neural network contains a feature extractor composed of convolutional layers and subsampling layers, and this feature extractor can be regarded as a filter. A convolutional layer refers to the neuron layer in a convolutional neural network that performs convolutional processing on the input signal. In the convolutional layer of a convolutional neural network, a neuron can be connected to only some adjacent-layer neurons. In a convolutional layer, there are usually several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weight here is the convolutional kernel. Sharing weights can be understood as a way of extracting image information that is independent of position. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the convolutional neural network, the convolutional kernel can learn reasonable weights. Additionally, the direct benefit of sharing weights is to reduce the connections between layers of the convolutional neural network while also reducing the risk of overfitting.

[0098] Loss Function: During the training of a deep neural network, since we hope the output of the deep neural network is as close as possible to the value we truly want to predict, we can compare the predicted value of the current network with the true target value, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is too high, we adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the true target value or a value very close to the true target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function. They are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0099] Backpropagation Algorithm: A neural network can use the backpropagation (BP) algorithm to correct the magnitudes of the parameters in the neural network model during training, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forward propagating the input signal until the output will generate an error loss, and updating the parameters in the neural network model by backpropagating the error loss information, so as to make the error loss converge. The backpropagation algorithm is a reverse propagation movement dominated by the error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0100] As Figure 3 shown, an embodiment of the present application provides a system architecture 300. In Figure 3 it, the data acquisition device 360 is used to acquire training data. For the method of generating dynamic photos in the embodiment of the present application, the training data may include the original cover image of the dynamic photo and the video frames corresponding to the original cover image.

[0101] After acquiring the training data, the data acquisition device 360 stores these training data in the database 330, and the training device 320 trains to obtain the target model / rule 301 based on the training data maintained in the database 330.

[0102] The following describes how the training device 320 obtains the target model / rule 301 based on the training data. The training device 320 processes the input video frames, compares the output image with the cover image, until the difference between the image output by the training device 320 and the cover image is less than a certain threshold, thereby completing the training of the target model / rule 301.

[0103] The above-mentioned target model / rule 301 can be used to implement the method for generating dynamic photos according to the embodiments of the present application. The target model / rule 301 in the embodiments of the present application can specifically be a convolutional neural network.

[0104] The target model / rule 301 trained according to the training device 320 can be applied to different systems or devices, such as being applied to Figure 3 the execution device 310 shown in the figure. The execution device 310 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle-mounted terminal, etc., or can also be a server or the cloud, etc. In Figure 3 it, the execution device 310 is configured with an input / output (I / O) interface 312 for data interaction with external devices. A user can input data to the I / O interface 312 through the client device 340.

[0105] The preprocessing modules 313 and 314 are used to preprocess the input data (such as the image to be processed) received by the I / O interface 312. In the embodiments of the present application, there may also be no preprocessing modules 313 and 314 (or there may only be one of the preprocessing modules), and the computing module 311 can directly process the input data.

[0106] When the execution device 310 preprocesses the input data, or when the computing module 311 of the execution device 310 performs calculations and other related processing, the execution device 310 can call the data, code, etc. in the data storage system 350 for corresponding processing, or can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 350. The computing module 311 uses the target model / rule 301 to process the input data.

[0107] Finally, the I / O interface 312 returns the processing result to the client device 340, so as to provide it to the user. The client device 340 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an AR / VR, a vehicle-mounted terminal, etc.

[0108] It should be noted that the training device 320 can generate corresponding target models / rules 301 based on different training data for different targets or different tasks. The corresponding target models / rules 301 can be used to achieve the above-mentioned targets or complete the above-mentioned tasks, so as to provide the required results for the user.

[0109] It should be noted that Figure 3This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationships among the devices, components, modules, etc. shown in the figure do not constitute any limitation.

[0110] For example, other devices in the system architecture 200 except the client device 340 can be integrated in the same device, or all devices in the system architecture 200 can be integrated in the same device.

[0111] Such as Figure 3 As shown, the target model / rule 301 is trained according to the training device 320. The target model / rule 301 can be a neural network in the present application in the embodiments of the present application. Specifically, the neural network constructed in the embodiments of the present application can be a CNN, a deep convolutional neural network (DCNN), a recurrent neural network (RNNs), and so on.

[0112] Since CNN is a very common neural network, the structure of CNN will be introduced in detail below in combination with Figure 4 As described in the above basic concept introduction, CNN is a deep neural network with a convolutional structure and is a deep learning architecture. The deep learning architecture refers to performing multiple levels of learning at different abstraction levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to the input image.

[0113] Such as Figure 4 As shown, the CNN 400 can include an input layer 410, a convolutional / pooling layer 420 (where the pooling layer is optional), and a neural network layer 430. The relevant content of these layers will be introduced in detail below.

[0114] Convolutional / pooling layer 420:

[0115] Convolutional layer:

[0116] Below, the internal working principle of one convolutional layer will be introduced by taking the convolutional layer 421 as an example.

[0117] The convolutional layer 421 may include a number of convolutional operators, also known as kernels. Essentially, a convolutional operator can be a weight matrix, which is usually predefined. During the process of performing a convolution operation on an image, the weight matrix typically processes the input image pixel by pixel (or two pixels at a time... depending on the value of the stride) along the horizontal direction, thus completing the work of extracting specific features from the image. The size of this weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as that of the input image. During the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolving with a single weight matrix will produce a convolved output with a single depth dimension. However, in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (rows × columns), i.e., multiple matrices of the same type, are applied. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image, where the dimension can be understood as being determined by the aforementioned "multiple". Different weight matrices can be used to extract different features from the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and yet another weight matrix is used to blur the unwanted noise in the image, etc. These multiple weight matrices have the same size (rows × columns), and the size of the feature maps extracted by these multiple weight matrices of the same size is also the same. Then, the multiple feature maps of the same size that are extracted are combined to form the output of the convolution operation.

[0118] The weight values in these weight matrices need to be obtained through a large amount of training in practical applications. Each weight matrix formed by the weight values obtained through training can be used to extract feature information from the input image, so that the convolutional neural network 300 can make correct predictions.

[0119] When the convolutional neural network 400 has multiple convolutional layers, the initial convolutional layer (such as 421) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 400 increases, the features extracted by the subsequent convolutional layers (such as 426) become more and more complex, such as high-level semantic features. The higher the semantic features, the more suitable they are for the problem to be solved.

[0120] Pooling layer / Pooling layer 420:

[0121] Since it is often necessary to reduce the number of training parameters, a pooling layer is often introduced periodically after the convolutional layer. For example, Figure 4As shown, it can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. During image processing, the purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a max pooling operator for sampling the input image to obtain a smaller-sized image. The average pooling operator can calculate the average value of the pixel values in the image within a specific range as the result of average pooling. The max pooling operator can take the pixel with the maximum value within the specific range as the result of max pooling. Additionally, just as the size of the weight matrix in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the size of the image. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel point in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

[0122] Neural network layer 430:

[0123] After being processed by the convolutional layer / pooling layer 420, the convolutional neural network 400 is still not sufficient to output the required output information. As mentioned before, the convolutional layer / pooling layer 420 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other relevant information), the convolutional neural network 400 needs to use the neural network layer 430 to generate one or a set of outputs corresponding to the number of required classes. Therefore, the neural network layer 430 can include multiple hidden layers (such as Figure 4 431, 432 to 43n as shown) and an output layer 440. The parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type can include image text recognition, image classification, image super-resolution reconstruction, etc.

[0124] After the multiple hidden layers in the neural network layer 430, that is, the last layer of the entire convolutional neural network 400 is the output layer 440. The output layer 440 has a loss function similar to categorical cross-entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 400 (such as Figure 4 the propagation from 410 to 440 is forward propagation) is completed, the backpropagation (such as Figure 4 the propagation from 440 to 410 is backpropagation) will start to update the weight values and biases of the previously mentioned layers to reduce the loss of the convolutional neural network 400, that is, the error between the result output by the convolutional neural network 400 through the output layer and the ideal result.

[0125] It should be noted that as Figure 4The convolutional neural network 400 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0126] Figure 5 The schematic flowchart of the method for replacing the cover image of a live photo provided by an embodiment of the present application is shown. As Figure 5 shown, the method includes:

[0127] S501, the electronic device generates a first live photo in response to a first operation of the user.

[0128] For example, as Figure 6 shown in (a) of, the electronic device displays an interface 601, which is a shooting interface. The interface 601 includes a shooting control 602. When the electronic device detects an operation of the user clicking the shooting control 602, in response to this operation, shooting can be performed to generate a first live photo. The first live photo includes a first cover image and N video frames. The first cover image can also be referred to as the first cover frame. N≥2 and is an integer. The first cover image is the original cover image.

[0129] The first cover image may be an image captured when the electronic device detects an operation of the user clicking the shooting control 602. The N video frames are video frames cached by the electronic device 1.5 seconds before and after the operation of the user clicking the shooting control 60. The N video frames include a first video frame corresponding to the first cover image, that is, the first cover image will also be processed into one of the N video frames. In other words, one of the N video frames has the same picture content as the first cover image.

[0130] The first cover image can be understood as the image displayed by the first live photo in the gallery interface.

[0131] For example, as Figure 6 shown in (b) of, the electronic device displays an interface 603, which is a gallery interface. The interface 603 includes a first live photo 604. The image presented by the first live photo 604 in the interface 603 is the first cover image.

[0132] It can be understood that the first cover image is generated by the camera ISP, and the N video frames are generated by the video ISP. The image quality of the first cover image is higher than that of the N video frames.

[0133] It should be noted that in the embodiments of the present application, the parameters for characterizing image quality are not limited, and the parameters for characterizing image quality include but are not limited to: resolution, dynamic range, contrast, peak signal-to-noise ratio, etc. Taking the resolution as an example, the image quality of the first cover image being higher than that of the N video frames can be understood as the resolution of the first cover image being higher than that of the N video frames.

[0134] It should also be noted that the above is only illustrated by taking 1.5 seconds as an example, but the embodiments of the present application do not make specific limitations on this.

[0135] In some embodiments, when the electronic device generates the first dynamic photo, it can also save metadata, and the metadata includes but is not limited to: digital image stabilization (EIS) information, raw image data, ISO (sensitivity), etc.

[0136] It can be understood that each video frame can correspond to its own EIS information, raw image data, sensitivity, etc., and the metadata of the first cover image and the first video frame can be the same.

[0137] Alternatively, in some other embodiments, the first dynamic photo can also be a dynamic photo downloaded by the user, or a dynamic photo shared by other users.

[0138] S502, the electronic device determines a second video frame in response to the user's second operation.

[0139] The electronic device can determine a second video frame in response to the user's second operation. The second video frame is one of the N video frames, and the second video frame is the video frame that will become the cover image.

[0140] In some embodiments, the user's second operation can be a user manual selection operation.

[0141] For example, as shown in (b) of Figure 6 , when the electronic device detects the operation that the user clicks on the first dynamic photo 604 in the interface 603, in response to this operation, it can display a GUI as shown in (c) of Figure 6 .

[0142] As shown in (c) of Figure 6 , when the electronic device responds to the operation that the user clicks on the first dynamic photo 604, it can zoom in on the first dynamic photo 604 and display one or more controls, such as an edit control 605, a share control, etc. When the electronic device detects the operation that the user clicks on the edit control 605, in response to this operation, it can display a GUI as shown in (d) of Figure 6 .

[0143] As shown inFigure 6 As shown in (d) in , in response to a user's operation of clicking on the editing control 605, the electronic device can display the N video frames included in the first dynamic photo 604, and the N video frames include a first video frame 606 and a second video frame 607.

[0144] In some embodiments, the electronic device can identify the first video frame 606 to indicate that the first video frame 606 corresponds to the cover image.

[0145] When the electronic device detects a user's operation of clicking on the second video frame 607, in response to this operation, it can display a GUI as shown in (e) in . Figure 6 in .

[0146] As Figure 6 shown in (e) in , in response to a user's operation of clicking on the second video frame 607, the electronic device can identify the second video frame 607. When the electronic device detects a user's operation of clicking on the control 608, it can determine the second video frame.

[0147] S503, the electronic device sends the first cover image and the first video frame to the server.

[0148] Correspondingly, the server receives the first cover image and the first video frame sent by the electronic device.

[0149] In some embodiments, the electronic device can also send metadata to the server. For example, the electronic device can send the original image data corresponding to the first cover image and the first video frame to the server.

[0150] Correspondingly, the server can also receive the metadata sent by the electronic device.

[0151] It should be noted that the execution order of S502 and S503 is not limited in the embodiments of the present application. For example, the electronic device can first execute S502 and then execute S503. For another example, the electronic device can execute S503 after executing S501, and then when the electronic device detects a second operation of the user, the electronic device executes S502, that is, the electronic device first executes S503 and then executes S502.

[0152] S504, the server trains a neural network based on the first cover image and the first video frame.

[0153] In some embodiments, after receiving the first cover image and the first video frame, the server can input the first video frame into the neural network, and the neural network processes the first video frame, compares the output training image with the first cover image, until the difference between the output training image and the first cover image is less than a certain threshold, thereby completing the training of the neural network.

[0154] In some embodiments, after receiving the first cover image and the first video frame, the server may input the first video frame and the first cover image into a neural network. The neural network generates a training image and compares the output training image with the first cover image until the difference between the output training image and the first cover image is less than a certain threshold, thereby completing the training of the neural network.

[0155] In some embodiments, after receiving the first cover image and the first video frame, the server may input the first video frame, the first cover image, and metadata into a neural network. The neural network generates a training image and compares the output training image with the first cover image until the difference between the output training image and the first cover image is less than a certain threshold, thereby completing the training of the neural network.

[0156] In some embodiments, after receiving the first cover image and the first video frame, the server may input the first video frame and metadata into a neural network. The neural network generates a training image and compares the output training image with the first cover image until the difference between the output training image and the first cover image is less than a certain threshold, thereby completing the training of the neural network.

[0157] It should be noted that in the embodiments of the present application, there are no specific limitations on the method for training the neural network and the type of the neural network. Existing or future proposed neural networks for image enhancement can be used for training. Training the neural network with the above training data to implement the method for generating dynamic photos provided in the embodiments of the present application should not be considered beyond the scope of the present application.

[0158] In some embodiments, when training the neural network, the server may train the neural network from scratch to determine the first network parameters of the neural network.

[0159] In some embodiments, all or part of the network parameters are pre-set in the neural network. When training the neural network, the server may update all or part of the network parameters in the neural network to determine the first network parameters of the neural network. For these embodiments, the pre-set all or part of the network parameters may be factory pre-set or can be updated according to the system version, and the neural network of the electronic device may include the same network parameters, or the all or part of the network parameters may be synchronously sent by the electronic device when sending the first cover image and the first video frame to the server.

[0160] S505, the server sends the first network parameters to the electronic device.

[0161] Correspondingly, the electronic device receives the first network parameters sent by the server.

[0162] After the server trains a neural network based on the first cover image and the first video frame, it can send the first network parameters of the neural network to the electronic device. It can be understood that the first network parameters are determined according to the first cover image and the first video frame, or the first network parameters are determined according to the first cover image, the first video frame, and the metadata. The first network parameters are the network parameters corresponding to the first live photo. In the embodiments of the present application, a set of network parameters can be determined for each live photo according to the above method. In other words, in the embodiments of the present application, a unique set of network parameters can be trained for each photo.

[0163] In some embodiments, the network parameters of each live photo can be all or partially different.

[0164] S506, the electronic device loads the first network parameters into the neural network.

[0165] After receiving the first network parameters, the electronic device can load the first network parameters into the neural network, and the neural network can be used to generate a cover image.

[0166] S507, the electronic device inputs the second video frame into the neural network to generate a second cover image.

[0167] In some embodiments, the electronic device can input the second video frame into the neural network loaded with the first network parameters to generate a second cover image.

[0168] In some embodiments, the electronic device can input the second video frame and the metadata corresponding to the second video frame into the neural network loaded with the first network parameters to generate a second cover image.

[0169] S508, the electronic device uses the second cover image to replace the first cover image.

[0170] After the electronic device generates the second cover frame, it can use the second cover image to replace the first cover image.

[0171] In some embodiments, the electronic device can delete the first cover image.

[0172] In some embodiments, the electronic device can hide the first cover image.

[0173] It can be understood that since the first network parameters are determined according to the first cover image and the first video frame, or the first network parameters are determined according to the first cover image, the first video frame, and the metadata, the neural network will be more adapted to the video frame in the first live photo. Therefore, when the electronic device inputs the second video frame into the neural network, the image quality of the generated second cover image will be similar to or the same as that of the first cover image.

[0174] In the embodiments of the present application, an electronic device may upload the original first cover image of a dynamic photo and the first video frame corresponding to the first cover image to a server. The server trains based on the first cover image and the first video frame to determine first network parameters, and sends the first network parameters to the electronic device. Thus, the electronic device may load the first network parameters into a neural network, and input a second video frame into the neural network according to a user's selection to generate a second cover image. Since the first network parameters are determined based on the first cover image and the first video frame, the neural network will be more adapted to the video frames in the first dynamic photo. Therefore, the image quality of the second cover image will be similar to or the same as that of the first cover image, which helps to improve the user experience.

[0175] In addition, in the embodiments of the present application, the training of the model is placed on the server, which can make full use of the powerful computing power on the server side, not only improving the training speed, but also taking into account the performance of the neural network.

[0176] It should be noted that since it takes a certain amount of time to execute S503 - S506, the following two implementation manners may be included for this situation:

[0177] One possible implementation manner is to wait until S503 - S506 are executed, and then input the second video frame into the neural network to generate a second cover image.

[0178] One possible implementation manner is that the neural network included in the electronic device has pre - set network parameters. During the process of waiting to execute S503 - S506, the electronic device may first input the second video frame into the neural network to generate a third cover image, and use the third cover image to replace the first cover image. In this possible implementation manner, after S503 - S506 are executed, the electronic device may also input the second video frame into the neural network loaded with the first network parameters to generate a second cover image, and then use the second cover image to replace the third cover image.

[0179] Alternatively, in some other embodiments, the server may not send the first network parameters to the electronic device, and the electronic device may not generate a second cover image either. Instead, after the electronic device determines the second video frame, it sends the second video frame to the server. After the server inputs the second video frame into the neural network to generate a second cover image, it sends the second cover image to the electronic device.

[0180] In the above - mentioned embodiments, the server may train a neural network according to the cover image reported by the electronic device and the video frame corresponding to the cover image. However, the embodiments of the present application are not limited thereto. In some other embodiments of the present application, the electronic device may also train the neural network locally. The following will be combined with Figure 7 for introduction.

[0181] Figure 7 The schematic flowchart showing the method for replacing the cover image of a live photo provided by an embodiment of the present application is as follows Figure 7 As shown, the method includes:

[0182] S701, the electronic device generates a first live photo in response to a first operation by the user.

[0183] It should be understood that for a detailed description of step S701, reference can be made to the description of step S501. For the sake of brevity, it will not be elaborated here.

[0184] In some embodiments, when generating the first live photo, the electronic device may also save metadata, which includes but is not limited to: EIS information, original image data, sensitivity, etc.

[0185] Alternatively, in some other embodiments, the first live photo may also be a live photo downloaded by the user from a third-party application or the Internet, or a live photo shared by other users.

[0186] S702, the electronic device trains a neural network based on the first cover image and the first video frame.

[0187] In some embodiments, after receiving the first cover image and the first video frame, the electronic device may input the first video frame into the neural network. The neural network processes the first video frame, compares the output training image with the first cover image until the difference between the output training image and the first cover image is less than a certain threshold, thereby completing the training of the neural network.

[0188] In some embodiments, after receiving the first cover image and the first video frame, the electronic device may input the first video frame and the first cover image into the neural network. The neural network generates a training image, compares the output training image with the first cover image until the difference between the output training image and the first cover image is less than a certain threshold, thereby completing the training of the neural network.

[0189] In some embodiments, after receiving the first cover image and the first video frame, the electronic device may input the first video frame, the first cover image, and the metadata into the neural network. The neural network generates a training image, compares the output training image with the first cover image until the difference between the output training image and the first cover image is less than a certain threshold, thereby completing the training of the neural network.

[0190] In some embodiments, after receiving the first cover image and the first video frame, the electronic device may input the first video frame and metadata into a neural network. The neural network generates a training image, and compares the output training image with the first cover image until the difference between the output training image and the first cover image is less than a certain threshold, thereby completing the training of the neural network.

[0191] It should be noted that in the embodiments of the present application, there are no specific limitations on the method for training the neural network and the type of the neural network. Existing or future proposed neural networks for image enhancement can be used for training. Training the neural network with the above training data to implement the method for generating dynamic photos provided by the embodiments of the present application should not be considered to exceed the scope of the present application.

[0192] In some embodiments, when training the neural network, the electronic device may train the neural network from scratch to determine the first network parameters of the neural network.

[0193] In some embodiments, all or part of the network parameters are pre-set in the neural network. When training the neural network, the electronic device may update all or part of the network parameters in the neural network to determine the first network parameters of the neural network.

[0194] S703, in response to a second operation of the user, the electronic device determines a second video frame.

[0195] It should be understood that for a detailed description of step S703, reference may be made to the description of step S502. For the sake of brevity, it will not be repeated here.

[0196] In some embodiments, the second operation of the user may be a user manual selection operation.

[0197] S704, the electronic device inputs the second video frame into the neural network to generate a second cover image.

[0198] After the electronic device completes the training of the neural network, it may input the second video frame into the neural network to generate a second cover image.

[0199] S705, the electronic device replaces the first cover image with the second cover image.

[0200] After the electronic device generates the second cover frame, it may replace the first cover image with the second cover image.

[0201] In some embodiments, the electronic device may delete the first cover image.

[0202] In some embodiments, the electronic device may hide the first cover image.

[0203] It can be understood that since the first network parameter is determined based on the first cover image and the first video frame, or the first network parameter is determined based on the first cover image, the first video frame, and the metadata, the neural network will be more adapted to the video frame in the first live photo. Therefore, when the electronic device inputs the second video frame into the neural network, the image quality of the generated second cover image will be similar to or the same as that of the first cover image.

[0204] In the embodiment of the present application, the electronic device can train a neural network according to the first cover image and the first video frame, and input the second video frame into the trained neural network according to the user's selection to generate a second cover image. Since the neural network is trained according to the first cover image and the first video frame, the neural network will be more adapted to the video frame in the first live photo. Therefore, the image quality of the second cover image will be similar to or the same as that of the first cover image, which helps to improve the user experience.

[0205] Similar to the above description, the electronic device can input the second video frame into the neural network to generate a second cover image after generating the first network parameter, or the neural network included in the electronic device has pre-set network parameters. During the process of generating the first network parameter, the electronic device can first input the second video frame into the neural network to generate a third cover image, and use the third cover image to replace the first cover image. After the electronic device generates the first network parameter, the electronic device can also input the second video frame into the neural network loaded with the first network parameter to generate a second cover image, and then use the second cover image to replace the third cover image.

[0206] Figure 8 FIG. shows a schematic flowchart of a method for replacing the cover image of a live photo provided by an embodiment of the present application. As Figure 8 shown, the method includes:

[0207] S801, loading a first network parameter into a neural network according to a first cover image of a first live photo and a first video frame corresponding to the first cover image, where the first live photo includes N video frames, the N video frames include the first video frame, N≥2 and is an integer.

[0208] In some embodiments, the electronic device can send the first cover image and the first video frame corresponding to the first cover image to the server, and the server trains according to the first cover image and the first video frame to determine the first network parameter of the neural network. The server sends the first network parameter to the electronic device, and then the electronic device loads the first network parameter into the neural network. The first network parameter is associated with the first live photo, and the first cover image can be the original cover image.

[0209] In some embodiments, the electronic device may be trained according to the first cover image and the first video frame. After the training is completed, the network parameters of the neural network are the first network parameters.

[0210] In some embodiments, the first live photo is generated by the electronic device in response to a user's shooting action.

[0211] In some embodiments, the first live photo is a live photo downloaded by the user.

[0212] In some embodiments, the first live photo is a live photo shared by other users.

[0213] In some embodiments, when generating the first live photo, the electronic device may also save the metadata of the first live photo. Furthermore, the electronic device or the server may determine the first network parameters according to the first cover image, the first video frame, and the corresponding metadata.

[0214] S802. In response to the user's selection operation, determine the second video frame among the N video frames.

[0215] It should be understood that for the description of step S803, reference may be made to the above description. For the sake of brevity, it will not be repeated here.

[0216] S803. Input the second video frame into the neural network to generate a second cover image.

[0217] In some embodiments, the electronic device may input the second video frame into the neural network to generate a second cover image.

[0218] In some embodiments, the electronic device may send the second video frame to the server. The server inputs the second video frame into the neural network to generate a second cover image, and then the server sends the second cover image to the electronic device.

[0219] In some embodiments, the electronic device may input the second video frame and the metadata corresponding to the second video frame into the neural network to generate a second cover image.

[0220] In some embodiments, the electronic device may send the second video frame and the metadata corresponding to the second video frame to the server. The server inputs the second video frame and the metadata corresponding to the second video frame into the neural network to generate a second cover image, and then the server sends the second cover image to the electronic device.

[0221] S804. Replace the first cover image with the second cover image.

[0222] In some embodiments, after the electronic device generates the second cover image or receives the second cover image sent by the server, it may replace the first cover image with the second cover image.

[0223] In some embodiments, the electronic device may delete the first cover image.

[0224] In some embodiments, the electronic device may hide the first cover image.

[0225] In some embodiments, the electronic device may send the first live photo to the server. The server replaces the first cover image with the second cover image and then sends the first live photo with the replaced cover image back to the electronic device. In other words, in these embodiments, the electronic device may send the first live photo and the second video frame (or indication information for indicating the second video frame) to the server. The server trains a neural network based on the first video frame and the first cover image (or the first video frame, the first cover image, and the corresponding metadata), then inputs the second video frame (or the second video frame and the corresponding metadata) into the neural network to generate the second cover image, and then replaces the first cover image with the second cover image.

[0226] In the embodiments of the present application, a neural network may be trained based on the original cover image and the corresponding video frame, and the second video frame is input into the trained neural network according to the user's selection to generate the second cover image. Since the neural network is trained based on the original cover image and the corresponding video frame, the neural network will be more adapted to the video frame in the first live photo. Therefore, the image quality of the second cover image will be similar to or the same as that of the original cover image, which helps to improve the user experience.

[0227] The above mainly introduces a method for replacing the cover image of a live photo provided by the embodiments of the present application from the perspectives of the electronic device and the server. It can be understood that in order to implement the above functions, the electronic device and the server include the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described function for each specific application, but this implementation should not be considered to exceed the scope of the present application.

[0228] In the embodiments of the present application, the processors in the electronic device and the server can be divided into functional modules (or units) according to the above method examples. For example, each functional module (or unit) can be corresponding to each function, or two or more functions can be integrated into one processing module (or unit). The above integrated module (or unit) can be implemented in the form of hardware or in the form of a software functional module (or unit). It should be noted that the division of modules (or units) in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0229] In the case of dividing each functional module (or unit) corresponding to each function, Figure 9 FIG. shows a schematic diagram of the composition of an electronic device 900 provided by an embodiment of the present application. As Figure 9 shown, the electronic device 900 includes: a shooting module 910 and a processing module 920.

[0230] Among them, in some embodiments:

[0231] The shooting module 910 is configured to generate a first dynamic photo in response to a first operation of the user.

[0232] The processing module 920 is configured to determine a first network parameter according to a first cover image of the first dynamic photo and a first video frame corresponding to the first cover image.

[0233] The processing module 920 is further configured to:

[0234] Load the first network into the neural network;

[0235] Determine a second video frame in response to a second operation of the user;

[0236] Input the second video frame into the neural network to generate a second cover image;

[0237] Replace the first cover image with the second cover image.

[0238] In some embodiments, the shooting module 910 is further configured to save the metadata of the first dynamic photo.

[0239] In some embodiments, the processing module 920 is specifically configured to determine the first network parameter according to the first cover image, the first video frame, and the metadata.

[0240] In some embodiments, the electronic device 900 further includes a transceiver module 930.

[0241] Among them, the shooting module 910 is configured to generate a first dynamic photo in response to a first operation of the user.

[0242] A transceiver module 930, configured to send a first cover image of a first dynamic photo and a first video frame corresponding to the first cover image to a server.

[0243] The transceiver module 930 is further configured to receive first network parameters sent by the server, where the first network parameters are determined by the server according to the first cover image and the first video frame.

[0244] A processing module 920, configured to:

[0245] Load the first network parameters into a neural network;

[0246] In response to a second operation of a user, determine a second video frame;

[0247] Input the second video frame into the neural network to generate a second cover image;

[0248] Use the second cover image to replace the first cover image.

[0249] In some embodiments, a shooting module 910 is further configured to save metadata of the first dynamic photo.

[0250] In some embodiments, the transceiver module 930 is further configured to send the metadata of the first dynamic photo to the server.

[0251] In some embodiments, the first network parameters are determined according to the first cover image, the first video frame, and the metadata.

[0252] Similar to the above description, after generating the first network parameters, the processing module 920 may input the second video frame into the neural network to generate a second cover image, or the neural network included in the processing module 920 has pre-set network parameters. During the process of generating the first network parameters, the processing module 920 may first input the second video frame into the neural network to generate a third cover image, and use the third cover image to replace the first cover image. After the processing module 920 generates the first network parameters, the processing module 920 may further input the second video frame into the neural network loaded with the first network parameters to generate a second cover image, and then use the second cover image to replace the third cover image.

[0253] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to execute the technical solutions in the above embodiments. The implementation principle and technical effects are similar to those of the related embodiments of the above method, and will not be elaborated here.

[0254] An embodiment of the present application provides a readable storage medium. The readable storage medium contains instructions that, when running on an electronic device, cause the electronic device to execute the technical solutions of the above embodiments. The implementation principles and technical effects are similar and will not be elaborated here.

[0255] An embodiment of the present application provides a chip. The chip is used to execute instructions that, when the chip runs, execute the technical solutions in the above embodiments. The implementation principles and technical effects are similar and will not be elaborated here.

[0256] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0257] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0258] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in an electrical, mechanical, or other form.

[0259] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0261] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0262] As described above, the above is only the specific implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the embodiments of the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A method for replacing a cover image of a dynamic photo, characterized in that The method includes: Loading a first network parameter into a neural network according to a first cover image of a first dynamic photo and a first video frame corresponding to the first cover image, where the first dynamic photo includes N video frames, the N video frames include the first video frame, and N≥2 and is an integer; Determining a second video frame among the N video frames in response to a user's selection operation; Inputting the second video frame into the neural network to generate a second cover image; Replacing the first cover image with the second cover image.

2. The method according to claim 1, wherein, Before loading the first network parameter into the neural network according to the first cover image of the first dynamic photo and the first video frame corresponding to the first cover image, the method further includes: Generating the first dynamic photo in response to a user's shooting operation.

3. The method according to claim 2, wherein The method further includes: Saving metadata corresponding to the first dynamic photo.

4. The method according to claim 3, characterized in that Loading the first network parameter into the neural network according to the first cover image of the first dynamic photo and the first video frame corresponding to the first cover image includes: Loading the first network parameter into the neural network according to the first cover image, the first video frame, and the metadata.

5. The method according to claim 3 or 4, characterized in that, The metadata includes one or more of the following: Digital image stabilization (EIS) information, raw image data, and ISO sensitivity.

6. The method according to any one of claims 1 to 5, characterized in that The method is applied to an electronic device.

7. The method according to claim 6, wherein Loading the first network parameter into the neural network according to the first cover image of the first dynamic photo and the first video frame corresponding to the first cover image includes: The electronic device sending the first cover image and the first video frame to a server so that the server determines the first network parameter according to the first cover image and the first video frame; The electronic device receiving the first network parameter; The electronic device loading the first network parameter into the neural network.

8. A system, characterized in that, The system includes an electronic device and a server, where The electronic device is configured to send a first cover image of a first dynamic photo and a first video frame corresponding to the first cover image to the server, the first dynamic photo includes N video frames, the N video frames include the first video frame and a second video frame, and N≥2 and is an integer; The server is configured to: Determine a first network parameter according to the first cover image and the first video frame; Send the first network parameter to the electronic device; The electronic device is further configured to: Determine the second video frame in response to a user's selection operation; Load the first network parameter into the neural network; Input the second video frame into the neural network to generate a second cover image; Replace the first cover image with the second cover image.

9. The system according to claim 8, characterized in that, The electronic device is further configured to generate the first dynamic photo in response to a user's shooting operation.

10. The system according to claim 9, wherein, The electronic device is further configured to save metadata corresponding to the first dynamic photo.

11. The system according to claim 10, wherein The electronic device is specifically configured to send the first cover image, the first video frame, and the metadata to the server; The server is specifically configured to determine the first network parameter according to the first cover image, the first video frame, and the metadata.

12. The system according to claim 10 or 11, characterized in that, The metadata includes one or more of the following: Electronic Image Stabilization (EIS) information, raw image data, ISO sensitivity.

13. An electronic device, characterized in that, Comprising one or more processors; one or more memories; the one or more memories store one or more computer programs, the one or more computer programs include instructions which, when executed by the one or more processors, cause the method according to any one of claims 1 to 7 to be executed.

14. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface is configured to receive a signal and transmit the signal to the processor, and the processor processes the signal so that the method according to any one of claims 1 to 7 is executed.

15. A computer-readable storage medium, characterized in that, Computer instructions are stored in the computer-readable storage medium, and when the computer instructions are run on a computer, the method according to any one of claims 1 to 7 is caused to be executed.

16. A computer program product, characterized in that, When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images