Image processing method and related device

Abstract

The embodiment of the present application provides an image processing method and related devices, which relate to the field of terminal technology. The method includes: at a first moment, the electronic device responds to a first operation that triggers a first application to shoot an image, and obtains a first photographed image using a first encoder and a first decoder. At a second moment after the first moment, the electronic device responds to the first operation again, and obtains a second photographed image using the first encoder and the second decoder, wherein between the first moment and the second moment, the electronic device does not release the first encoder, so that the first encoder does not need to be repeatedly loaded during the operation of the first application, which is conducive to simplifying the workflow of the electronic device, improving the image processing efficiency of the electronic device, reducing the operating burden of the electronic device, and improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and related devices. Background Art

[0002] An electronic device may support a shooting function, and a user may use various methods to record what he sees based on the shooting function of a camera application. For example, a new recording method may be realized through specific functions in the camera application, such as night scene photography, telephoto photography, high dynamic range (HDR), and time-lapse photography.

[0003] However, in some scenarios, when users are using the shooting function, the electronic device may sometimes freeze, affecting the user experience. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and related devices, which are helpful in simplifying the workflow of electronic devices, improving the image processing efficiency of electronic devices, reducing the operating burden of electronic devices, and improving user experience.

[0005] In a first aspect, an embodiment of the present application proposes an image processing method, which is applied to an electronic device, and the method includes: at a first moment, the mode of a first application is a first mode, and in response to a first operation on the first application, a first photographed image is obtained by using a first encoder and a first decoder, the first application is an application with a shooting function, the first operation is used to trigger the first application to shoot an image, the first encoder is used to extract features of multiple frames of images collected by the first application, and the first decoder is used to obtain the first photographed image based on image features output by the first encoder in the first mode, and the mode of the first application is related to the shooting mode and shooting parameters; at a second moment, the mode of the first application is a second mode, and in response to the first operation, a second photographed image is obtained by using the first encoder and the second decoder, and the second decoder is used to obtain the second photographed image based on image features output by the first encoder in the second mode, the second mode is different from the first mode, the second moment is after the first moment, and between the first moment and the second moment, the electronic device does not release the first encoder.

[0006] Optionally, the first application may be a camera application, or any other application having a shooting function or capable of calling a camera. In the case where the first application is a camera application, the first operation may be, for example, Figure 1The operation of the photo button 103 in the camera application interface shown in a, or any other operation that can trigger the first application to capture an image, the first moment may be moment 2 in method 400, the second moment may be moment 3 in method 400, the first decoder may be decoder A in method 400, and the second decoder may be decoder B in method 400, and the present application does not limit this.

[0007] It should be understood that the first encoder corresponds to the first mode, the second encoder corresponds to the second mode, and the mode of the first application is related to the shooting mode and the shooting parameters. For example, the first mode can be a night scene shooting mode and a first shooting parameter, and the second mode can be a night scene shooting mode and a second shooting parameter. This application does not limit this.

[0008] In an embodiment of the present application, after the electronic device obtains the first photographed image using the first encoder and the first decoder at a first moment, at a second moment after the first moment, in response to a first operation that triggers the first application to take an image, the electronic device obtains the second photographed image using the first encoder and the second decoder, wherein between the first moment and the second moment, the electronic device does not release the first encoder, so that the first encoder does not need to be repeatedly loaded during the running of the first application, which is conducive to simplifying the workflow of the electronic device, improving the image processing efficiency of the electronic device, reducing the operating burden of the electronic device, and improving the user experience.

[0009] In some implementations, between the first moment and the second moment, the electronic device does not release the first encoder but releases the first decoder, which is beneficial for releasing the memory of the electronic device and facilitating the subsequent loading of other decoders.

[0010] Optionally, the first encoder may be a common encoder for all decoders included in the electronic device, or an encoder applicable to a part of the decoders included in the electronic device. For another part of the decoders included in the electronic device, other decoders obtained based on targeted parameter training may also be set. This application does not make any specific limitations on this.

[0011] In combination with the first aspect, in certain implementations of the first aspect, a first photographed image is obtained using a first encoder and a first decoder, including: obtaining the first photographed image using the first encoder, a second encoder, and the first decoder, the second encoder being used to process image features output by the first encoder in the first mode to adapt to the first decoder.

[0012] Among them, the second encoder can have similar capabilities to the encoder A described below, which will not be repeated here.

[0013] The method provided in the embodiment of the present application is advantageous in that, when the encoder and decoder are decoupled, each decoder has an encoder adapted thereto, which is advantageous in maintaining the working performance of the encoder and / or decoder in the electronic device on the basis of simplifying the workflow of the electronic device, improving the image processing efficiency of the electronic device, and reducing the operating burden.

[0014] In combination with the first aspect, in some implementations of the first aspect, the first decoder is loaded at the first moment in response to a first operation on the first application.

[0015] Optionally, the second encoder may also be loaded at a first moment in response to a first operation on a first application, but this application is not limited to this.

[0016] In the method provided in the embodiment of the present application, the first decoder can be loaded at a first moment in response to a first operation for a first application, and the timing of loading the first encoder and the first decoder can be different. This approach is conducive to reducing the operating pressure of the electronic device in the first application during the image capturing and image processing stages, and is conducive to improving image processing efficiency.

[0017] In combination with the first aspect, in some implementations of the first aspect, before the first moment, the method further includes:

[0018] Load the first encoder.

[0019] In the method provided in the embodiment of the present application, the first encoder can be loaded at a first moment, and the first decoder can be loaded at a first moment in response to a first operation for a first application. The timing of loading the first encoder and the first decoder can be different. This method is conducive to reducing the operating pressure of the electronic device in the first application during the image capturing and image processing stages, and is conducive to improving image processing efficiency.

[0020] In combination with the first aspect, in certain implementations of the first aspect, loading the first encoder includes: loading the first encoder in response to a second operation, the second operation being used to start the first application; or, loading the first encoder in response to a third operation, the third operation being used to set a mode of the first application; or, loading the first encoder when the first application has been started for a preset duration.

[0021] In combination with the first aspect, in some implementations of the first aspect, the method further includes: at a third moment, in response to a fourth operation, releasing the first encoder, the fourth operation being used to close the first application.

[0022] In the embodiment of the present application, the first encoder is released only when the first application is closed. In this way, the first encoder only needs to be loaded once when the first application is run, which is conducive to simplifying the workflow of the electronic device, saving computing power of the electronic device and improving efficiency.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the first encoder includes one or more of S encoders trained respectively for S types of frames in the multi-frame images, where S is a non-negative integer.

[0024] The encoder in the related art may have the ability to extract features for S types of frames. Such an encoder model has a large number of channels, and its parameter quantity is usually large, which occupies a large amount of memory of the electronic device.

[0025] In the embodiment of the present application, S encoders are trained respectively for the S types of frames in the multi-frame images, and the scenarios that a single encoder needs to face become simpler. On the one hand, it is beneficial to reduce the difficulty of training the encoder and reducing the difficulty of development. On the other hand, it is beneficial to reduce the overall parameter amount of the encoder, which is beneficial to reducing the occupation of the encoder's memory in the electronic device and reducing the operating burden of the electronic device.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the types of frames in the multi-frame images include one or more of long frames, short frames, or ordinary frames.

[0027] In a second aspect, an embodiment of the present application provides an image processing device, which may be an electronic device, or a chip or chip system in an electronic device. The image processing device may include a display unit and a processing unit. When the image processing device is an electronic device, the display unit may be a display screen. The display unit is used to perform the display step so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing device is an electronic device, the processing unit may be a processor. The image processing device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (eg, a register, a cache, etc.), or a storage unit within the electronic device that is located outside the chip (eg, a read-only memory, a random access memory, etc.).

[0028] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory, wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0029] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program or instructions. When the computer program or instructions are run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0030] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0031] In a sixth aspect, the present application provides a chip or a chip system, which includes one or more processors and a communication interface, wherein the communication interface and the one or more processors are interconnected by lines, and the one or more processors are used to run computer programs or instructions to execute the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0032] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0033] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of an application scenario applicable to the embodiment of the present application;

[0035] Figure 2 A schematic diagram of a camera application workflow provided in an embodiment of the present application;

[0036] Figure 3A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0037] Figure 4 A schematic flow chart of an image processing method provided in an embodiment of the present application;

[0038] Figure 5 A schematic flow chart of another image processing method provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0042] 1. Encoder: The encoder is the part of the neural network model that converts input data (usually images in image processing) into a low-dimensional or abstract representation. It gradually extracts and summarizes the features of the input image through a series of network layers (such as convolutional layers, pooling layers, etc.), mapping it to a smaller dimensional representation space. This representation is usually called the encoding or hidden state, which captures the key information in the input image while removing redundancy and noise.

[0043] In an embodiment of the present application, the encoder can perform feature extraction on the image captured by the image sensor during the image fusion processing. In the process of extracting features, noise and redundant information in the image can usually be removed while retaining abstract information that is useful for subsequent tasks.

[0044] 2. Decoder: The decoder is another part of the neural network model that is responsible for recovering the original or target domain image data from the encoder's output (i.e., the encoded or hidden state). The decoder also gradually generates higher-level image representations through a series of network layers (such as transposed convolutional layers, upsampling layers, etc.), and finally outputs an image that is similar to the input image or has specific attributes.

[0045] In the embodiment of the present application, the camera algorithm model including the encoder-decoder realizes functions including but not limited to image compression, image denoising, image super-resolution, image style transfer, image generation and other tasks. By adjusting the network structure and parameters of the encoder and decoder, different image processing tasks can be optimized to achieve better performance.

[0046] 3. RAW image: It can be understood as the unprocessed image captured by the image sensor. RAW image can also be understood as the original image.

[0047] 4. YUV image: It can be understood as an image processed by color encoding method, where Y represents luminance, that is, grayscale value, U represents chrominance, which is used to describe image color, and V represents chroma, which is used to describe image saturation.

[0048] 5. Preview images and take photos

[0049] The preview image may be data collected in real time by a camera of the electronic device and used to be displayed in a preview screen. For example, when the electronic device receives an operation of a user to start a camera application, the electronic device may collect the preview image captured by the camera and display it in real time in a preview screen of the camera application.

[0050] The photographed image may be data obtained based on a photographing button in the electronic device. For example, when the electronic device receives a trigger operation of the user on the photographing button, the electronic device may obtain a photographed image obtained based on a camera at the time of photographing.

[0051] 6. HDR mode: High dynamic range (HDR) mode is also called dual-frame HDR mode. HDR mode can synthesize multiple photos with different exposures to make the brightness range of the photos wider and show more details and colors. HDR mode photography can obtain multiple frames with different exposure parameters, for example, you can obtain long-frame and short-frame images in one shot.

[0052] In the embodiments of the present application, "long frame", "short frame" and "normal frame" are all relative concepts. Normal frame can be understood as the image frame output by the image sensor in the preview stage, which is used as a reference frame to provide a reference brightness; long frame and short frame can be frames with different exposure times from normal frame, which are generated by the image sensor in response to the shooting parameters issued according to the button pressing event. Long frame can be an image frame with an exposure time longer than that of normal frame, which can provide information about dark areas, so that the details of dark areas can also be presented in the photo; short frame can be an image frame with an exposure time shorter than that of normal frame, which can provide information about highlight areas, and minimize overexposed areas in photos.

[0053] In the embodiment of the present application, the camera algorithm model can output high-quality images by fusing long frames, short frames and ordinary frames.

[0054] 7. Other terms

[0055] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0056] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0058] 8. Electronic devices

[0059] The electronic device of the embodiment of the present application may include a handheld device with a camera function, a vehicle-mounted device, etc. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, electronic devices in 5G networks or future evolved public land mobile communication networks (public landmobile networks). The electronic devices in the network (PLMN) and the like are not limited to this in the embodiments of the present application.

[0060] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0061] In addition, in the embodiments of the present application, the electronic device may also be an electronic device in an Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0062] The electronic device in the embodiments of the present application may also be referred to as: electronic device, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0063] Figure 1 An application scenario diagram of an embodiment of the present application is shown exemplarily. Figure 1 FIG. 1 shows a camera interface 101 of the camera application, and the user can take a photo by clicking a photo button 103 in the interface 101. Optionally, the user can also select a shooting mode of the camera application, for example, by clicking the “more” option in the interface 101 to enter the following example: Figure 1 The interface 102 shown in b in FIG. 102 is shown, and “HDR” is selected in the interface 102 to take a photo in HDR mode. It should be understood that not all controls on the photo taking interface are shown in the embodiment of the present application, and the photo taking interface may be designed with more or fewer controls, which is not limited in the present application. The user holds the electronic device and points it at the target object (e.g. Figure 1 104), click the camera button 103, the mobile phone can be triggered to take a photo, and a photo of the target object can be obtained. Optionally, after clicking the camera button, the user can view the taken photos by clicking the control 105.

[0064] In some scenarios, the HDR mode can generate images with a higher dynamic range, which can better present image details in bright and dark light. After the user presses the photo button, the electronic device can select multiple frames of images for image fusion processing in response to the photo button pressing event, so as to obtain a better quality image for display.

[0065] In some implementations, the image fusion processing of the electronic device can be achieved through a camera algorithm model. The camera algorithm model may include an encoder and a decoder. The encoder can be used to extract image features of the input multi-frame images, and the decoder can be used to enhance and optimize the image according to specific task requirements to reconstruct a high-quality image.

[0066] In a possible implementation, the electronic device includes multiple camera algorithm models, and different camera algorithm models may correspond to different modes of camera applications. The workflow 200 of the camera application in the electronic device may be as follows: Figure 2 As shown. For example, Figure 2 As shown, after the camera application is started, at time 1, the camera application is in mode 1, and the camera application responds to the camera button (which can be understood as Figure 1 The camera application selects the image frame to be processed by a pressing event of the button 103 shown in the figure, and then loads the encoder and decoder 1 of the camera algorithm model corresponding to mode 1, uses the encoder to extract the image feature 1 of the image frame to be processed, uses the decoder 1 to perform inference on the image feature 1 extracted by the encoder, and then outputs the photographed image 1 obtained by using the image frame to be processed, and releases the encoder and decoder 1. At time 2 after time 1, the camera application can respond to the pressing event of the photo button and execute a similar process again. That is, each time the user presses the photo button, the camera application needs to load the encoder and decoder of the target algorithm model corresponding to the photo mode once to process the multiple frames of images selected by the camera application in this shooting, and after each image after fusion processing is output, the encoder and decoder loaded for the image must be released, and then loaded again after the next photo button is pressed.

[0067] It should be understood that in order to improve the camera algorithm model's ability to process image fusion, the camera algorithm model usually has more parameters. The camera application loads the encoder and decoder each time it performs image processing. On the one hand, this makes the camera application workflow more cumbersome, and on the other hand, it also makes the electronic device's operation burden heavier. In some scenarios, users may feel that the electronic device is stuck when using the camera application, affecting the user's experience of playing with the device.

[0068] In one possible implementation, an electronic device may include multiple camera algorithm models, any of which may include an encoder and a decoder. The functions of the encoders in the multiple camera algorithm models may be similar and may all be used to extract features of images to be processed, while the decoders may have differences due to different output purposes (for example, the decoders may have the ability to adapt to different modes of camera applications based on mode parameter training of different camera applications).

[0069] In view of this, an embodiment of the present application provides an image processing method and a related device. After the electronic device takes a picture once, the encoder is not released. When taking the picture next time, the electronic device does not need to load the encoder again, and only loads the corresponding decoder for the camera application mode. In this way, the encoder does not need to be loaded repeatedly during the operation of the camera application, which is conducive to simplifying the workflow of the electronic device, improving the image processing efficiency of the electronic device, reducing the operating burden of the electronic device, and improving the user experience.

[0070] It is worth noting that the modes of the camera application described in the embodiments of the present application may include shooting modes of the camera application, such as HDR mode, portrait mode, night mode, etc., and may also include shooting parameters set by the user or automatically set by the camera application, such as pixels, sensitivity, aperture, exposure compensation, etc., which are not specifically limited in this application.

[0071] Furthermore, in the embodiments of the present application, "shooting" and "photographing" can be understood to express the same or similar meanings. For example, both can express taking photos, recording videos, etc., and the present application does not limit this.

[0072] Before introducing the image processing method provided in the embodiment of the present application, the software architecture of the electronic device is first described.

[0073] Figure 3 The software architecture diagram of the electronic device is shown as an example. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The layered architecture can adopt an Android system, a Linux operating system, a Unix operating system, an iOS operating system, a Windows operating system, or other operating systems, which is not limited in the embodiments of the present application. The following takes the Android system with a layered architecture as an example to illustrate the software architecture of the electronic device provided in the embodiments of the present application.

[0074] In some implementations, the layered architecture can divide the software system of the electronic device into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom: application layer (applications), application framework layer (application framework), hardware abstraction layer (hardware abstraction layer, HAL), kernel layer (kernel) and hardware layer.

[0075] like Figure 3As shown, the application layer may include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the application framework layer. For example, it may include a camera application, or other system applications, or third-party applications. This application does not limit this.

[0076] The application framework layer provides an API and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework layer may include a camera access interface, an activity management service, and a view system. Among them, the camera access interface can be used to provide an application programming interface and a programming framework for the camera application; the activity management service is the core service in the Android system responsible for managing the application life cycle and the Activity stack; the view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications, and the display interface of the electronic device can be composed of one or more views. For example, a display interface including a preview image of a camera application may include a control view of the camera application and a view for displaying a preview image.

[0077] Optionally, the camera access interface may be an Android Interface Definition Language (AIDL) interface, which is not limited in this application.

[0078] The HAL layer may include a camera hardware abstraction layer, a camera algorithm model, a frame queue module, and a display synthesis module (surfaceflinger). In the embodiment of the present application, the camera hardware abstraction layer may provide virtual hardware for the camera device; the camera algorithm model may include the running code and data for implementing the image processing method provided in the embodiment of the present application, such as an encoder and a decoder; the frame queue module is used to store images captured by the image sensor of the electronic device; the display synthesis module is used to synthesize the layers from the camera application and other applications (such as the system desktop) into one frame for display on the display screen.

[0079] In some examples, the camera algorithm model is also used to send digital signals to the digital signal processor driver (not shown in the figure) in the driver layer, so that the digital signal processor driver calls the digital signal processor (not shown in the figure) in the hardware layer for digital signal processing, and the digital signal processor can return the processed digital signal to the camera algorithm model through the digital signal processor driver; the camera algorithm library is also used to send digital signals to the graphics signal processor driver (not shown in the figure) in the driver layer, so that the graphics signal processor driver calls the graphics processor (not shown in the figure) in the hardware layer for digital signal processing, and the graphics processor can return the processed graphics data to the camera algorithm model through the graphics processor driver.

[0080] The kernel layer is a layer between hardware and software. In the present application, the kernel layer may include a camera device driver and a display driver. The camera device driver is used to drive the sensor of the camera to collect images; the display driver is used to deliver images to the display screen.

[0081] Optionally, the kernel layer may further include a digital signal processor driver and an image processor driver, wherein the digital signal processor driver is used to drive the digital signal processor to process images; and the image processor driver is used to drive the graphics processor to process images.

[0082] The hardware layer may include: sensors, such as image acquisition sensors, which can also be understood as cameras, display screens, and optionally, image signal processors (ISP), digital signal processors, and graphics processing units (GPU) and other hardware.

[0083] It should be understood that in some embodiments, the layers implementing the same function may be called by other names, or the layers that can implement the functions of multiple layers may be regarded as one layer, or the layers that can implement the functions of multiple layers may be divided into multiple layers. The embodiments of the present application are not limited to this.

[0084] The following describes the image processing method 400 provided in the embodiment of the present application from the perspective of the interaction between the modules in the software architecture of the electronic device described above.

[0085] In a possible scenario, at time 1, the user clicks the camera application icon on the electronic device desktop, triggering the launch process of the camera application in the electronic device. In some implementations, when the user clicks the camera application icon, the Launcher (Launcher is a special application in the Android system that is responsible for displaying and managing all application icons on the device) captures the click event, generates a camera application launch event, and sends it to the camera application.

[0086] Figure 4 The flowchart of the image processing method 400 provided in the embodiment of the present application is exemplarily shown. The method 400 includes the following steps:

[0087] S401: A camera application creates a camera application process in response to a camera application start event, and loads an encoder in the created camera application process.

[0088] In a possible implementation, the camera application transmits a camera application start event to the Activity Manager Service (AMS). The AMS searches for the main Activity of the camera application and creates an instance of the Activity. During the creation process, the AMS loads files related to the camera application, which may include the encoder.

[0089] In one possible implementation, an electronic device may include multiple camera algorithm models, any of which may include an encoder and a decoder. The functions of the encoders in the multiple camera algorithm models may be similar and may all be used to extract features of the image to be processed, while the decoders have differences due to different output purposes (for example, the ability to adapt to different modes of camera applications based on mode parameter training of different camera applications). Therefore, in an embodiment of the present application, the encoder is loaded during the process of starting the camera application process. In this way, after the user subsequently presses the press button, the electronic device only needs to load the decoder of the selected camera algorithm model in response to the press event, which is conducive to reducing the number of times the encoder is loaded during the use of the camera application, simplifying the workflow of the camera application, and avoiding the repeated loading of the encoder from bringing operating burden to the electronic device.

[0090] Optionally, the camera application can load the encoder when the camera application has been started for a preset time. This can help avoid the operating burden caused by pulling up the encoder during the startup process and reduce the pressure to create the camera application process. On the other hand, in most scenarios, the user still needs to focus on the target object after the camera application is started. There is some transition time in between. The preset duration can be set to be less than this transition time so that the encoder loading process can be completed before the user clicks the photo button. Subsequently, after the user presses the button, the electronic device only needs to load the decoder of the selected camera algorithm model in response to the press event. This can help disperse the operating burden of the camera application and avoid electronic device freezes due to large instantaneous memory usage.

[0091] Optionally, the camera application can also load the encoder in response to the setting operation of the camera application mode. In one possible scenario, after the camera application is started, the user can manually set the camera application mode based on their own shooting purpose (for example, using portrait mode for shooting people, using night mode for shooting at night, etc.), or the camera application can automatically set the camera application mode based on the current environment (setting shooting parameters such as sensitivity, aperture, exposure compensation, or switching shooting modes based on the target in the viewfinder), and the camera application can load the encoder in response to this setting operation, and then, after the user presses the press button, the electronic device only needs to load the decoder of the selected camera algorithm model in response to the press event, which is conducive to simplifying the workflow of the camera application and also helps to avoid the repeated loading of the decoder from bringing operating burden to the electronic device.

[0092] In some implementations, the types of image frames to be processed that are input to the encoder may include one or more of long frames, short frames, and normal frames. In related technologies, long frames, short frames, and normal frames can be spliced ​​together and input into an encoder with comprehensive analysis capabilities. Since such an encoder model needs to have the ability to extract features from the three types of frames, the number of channels of the decoder is also large, and the number of parameters is usually large, which occupies a large amount of read-only memory (ROM) memory and runtime memory of the electronic device.

[0093] The encoder in the embodiment of the present application may include one or more of a long frame encoder, a short frame encoder, and a normal frame encoder. Among them, the long frame encoder is used to perform feature extraction processing on the long frames in the image frames to be processed, the short frame encoder is used to perform feature extraction processing on the short frames in the image frames to be processed, and the normal frame encoder is used to perform feature extraction processing on the normal frames in the image frames to be processed. Since any one of the long frame encoder, the short frame encoder, and the normal frame encoder only needs to perform feature extraction on one type of image frame, in this way, the scene that a single encoder needs to face becomes simpler. On the one hand, it is conducive to reducing the difficulty of encoder training and development. On the other hand, it is conducive to reducing the number of encoder parameters, which is conducive to reducing the occupation of the encoder's memory in the electronic device and the operating burden of the electronic device.

[0094] S402: The camera application transmits a sensor start instruction to the image sensor.

[0095] In a possible implementation, the camera application may transmit a sensor start instruction to the image sensor through a camera access interface, a camera hardware abstraction layer, and a camera device driver in sequence.

[0096] It should be understood that the camera access interface can convert instructions from the application layer into instructions that can be recognized by the hardware abstraction layer, and the camera application can send the instruction to start the camera to the camera device driver of the kernel layer by calling the camera hardware abstraction layer. The camera device driver can drive the corresponding camera sensor to collect images.

[0097] S403: The image sensor collects images.

[0098] It should be understood that the process of image sensor capturing images can be understood as the process of exposing the pixel array of the image sensor. After the exposure of the pixel array of the image sensor is completed, the image sensor generates a frame of image. The image captured by the image sensor and not processed by the image signal processor can be called a RAW image.

[0099] In a possible implementation, the electronic device transmits the RAW image acquired by the image sensor and not processed by the image signal processor to the image signal processor, and the image signal processor processes the RAW image into a YUV image, and then transmits it back to the camera application through the camera device driver, the camera hardware abstraction layer, and the camera access interface, and then the camera application transmits it to the display synthesis module through the view system. The display synthesis module synthesizes a display interface containing a preview image based on the YUV image, and displays it on the display screen through the display driver, for example Figure 1 In the interface 101 in a, the preview image can be understood as the image that the user can see through the camera capture frame.

[0100] S404: The image sensor transmits the RAW image to the frame queue module.

[0101] It should be understood that the frame queue module is used to store images captured by the image sensor of the electronic device. These images may include relevant information used to represent image clarity or image quality, such as gyro information, optical image stabilization ois information, etc.

[0102] In a possible scenario, at time 2 after time 1, the camera application executes S405 in response to event 1 of the user pressing a photo button.

[0103] S405 : The camera application loads decoder A in response to the press event 1 .

[0104] In an embodiment of the present application, the electronic device includes multiple camera algorithm models, and each camera algorithm model may correspond to at least one decoder. Exemplarily, at the above moment 2, the camera application responds to the user's operation on the photo button, and decides based on the mode A of the current camera application that the camera algorithm model A should be used, that is, the decoder A corresponding to the camera algorithm model A is loaded. In some implementations, the decoder A may include a layer that adapts the capabilities of the encoder (for example, adapts the number of output modes).

[0105] It should be understood that after a press event occurs, the specific camera algorithm model adopted by the camera application is related to the mode of the camera application (the shooting mode selected by the user, the user's settings for the camera parameters, or the parameters automatically adjusted by the camera application, etc.). The camera application can use a decision tree model based on the mode of the camera application to decide at least one camera algorithm model for fusing image frames to obtain a photographed image.

[0106] Optionally, the camera algorithm model involved in the embodiments of the present application can be obtained by training based on a twin network model, but the present application is not limited to this.

[0107] In a possible implementation, in response to pressing event 1 , the camera application may load encoder A and decoder A based on mode A of the camera application, where encoder A is used to process image features output by the encoder in mode A to adapt decoder A.

[0108] Optionally, the preceding encoder may be understood as an encoder common to all decoders included in the electronic device, and the function of encoder A is to receive the output of the preceding decoder and adapt the input of the subsequent decoder A. Encoder A may be set separately from decoder A and trained separately, or may be set together with decoder A and trained in combination. Encoder A may also have other names, such as decoder A1 or any other arbitrary name, which is not specifically limited in the present application.

[0109] S406: The frame queue module transmits X image frames to be processed to the encoder.

[0110] Optionally, the X image frames to be processed may include a long frame, b short frames and c ordinary frames. The present application does not limit the specific number of long frames, short frames and ordinary frames in the X image frames to be processed.

[0111] In a possible implementation, the encoder may include a long frame encoder, a short frame encoder, and a normal frame encoder. The long frame encoder may process a-frame long frames frame by frame or in batches, the short frame encoder may process b-frame short frames frame by frame or in batches, and the normal frame encoder may process c-frame normal frames frame by frame or in batches, and this application does not make specific limitations on this.

[0112] When the processing mode of the long frame encoder, the short frame encoder and the normal frame encoder for the image frames is frame by frame, any one of the long frame encoder, the short frame encoder and the normal frame encoder can retain only three modes, namely, the three channels of red (red, R), green (green, G), and blue (blue, B). The number of modes can no longer be matched according to the number of input image frames, which is beneficial to further reduce the memory occupied by the encoder.

[0113] S407: The encoder extracts image features M from the X frames of images to be processed.

[0114] Optionally, the encoder may perform a splicing process on the image feature M, obtain the spliced ​​image feature and then input it into the decoder, but the present application is not limited to this.

[0115] S408 : The encoder transmits the image feature M to the decoder A.

[0116] S409, decoder A processes the image feature M output by the encoder to obtain a photographed image A.

[0117] S410. Decoder A transmits photographed image A to the camera application.

[0118] In one possible implementation, decoder A can transmit the photo image A back to the camera application through the camera hardware abstraction layer and the camera access interface in sequence. In one implementation, the camera application transmits it to the display synthesis module through the view system. The display synthesis module synthesizes a display interface containing the photo image A based on the photo image A, and displays it on the display screen through the display driver, such as the interface displayed after the user clicks the photo button and the photo image A is generated.

[0119] S411, the camera application process releases decoder A.

[0120] Optionally, the camera application process may release the encoder A after the decoder A outputs the photographed image A, and the decoder A may also release it automatically after a preset time interval, which is not limited in the present application.

[0121] So far, for the press event 1 at the above moment 2, the photographed image A has been obtained. In a possible scenario, at moment 3 after moment 2, after the user adjusts the photographing mode or camera parameters, when the camera application is in mode B, the user presses the photographing button again, and the camera application executes S412 in response to the user's press event 2 for the photographing button.

[0122] S412 : The camera application loads decoder B in response to the press event 2 .

[0123] S413. The frame queue module transmits the Y frame image to be processed to the encoder.

[0124] Optionally, X in S406 and Y in S413 may be the same or different, and this application does not make any specific limitation thereto.

[0125] S414: The encoder extracts image features N from the Y frame images to be processed.

[0126] S415. The encoder transmits image feature N to decoder B.

[0127] S416. Decoder B processes the image feature N output by the encoder to obtain a photographed image B.

[0128] S417. Decoder B transmits photographed image B to the camera application.

[0129] S418: The camera application process releases decoder B.

[0130] The processing methods of the above S407~S411 and S414~S418 may be similar and will not be repeated here.

[0131] In the embodiment of the present application, the decoder can be released during the process of closing the camera application process.

[0132] In the embodiment of the present application, the encoder only needs to be loaded once when the camera application is started, which is helpful in simplifying the workflow of the camera application, saving the computing power overhead of the electronic device, reducing the operating burden of the electronic device, and improving the user experience.

[0133] Moreover, the loading method that decouples the encoder and the decoder allows the training process of the camera algorithm model to be decoupled into training for the encoder and training for the decoder. When the electronic device is updated, its encoder and / or decoder can also be updated, increasing the flexibility of the electronic device update configuration.

[0134] Furthermore, the encoder in the embodiment of the present application may include one or more of a long frame encoder, a short frame encoder and a normal frame encoder. The long frame encoder is used to perform feature extraction processing on the long frame in the image frame to be processed, the short frame encoder is used to perform feature extraction processing on the short frame in the image frame to be processed, and the normal frame encoder is used to perform feature extraction processing on the normal frame in the image frame to be processed.

[0135] Since any of the long frame encoders, short frame encoders and normal frame encoders only needs to extract features from one type of image frame, when using a frame-by-frame extraction method, it is no longer necessary for the encoder to match the number of modes and only needs to retain three mode numbers (which can be understood as the number of R, G, and B channels). This method, on the one hand, is conducive to reducing the difficulty of encoder training and development, and on the other hand, is conducive to reducing the number of encoder parameters. When the encoder is running, it can occupy a smaller running memory and reduce the running burden of electronic equipment. On the other hand, since the scenarios that a single encoder needs to face become simpler, it is conducive to improving its performance.

[0136] The following describes an image processing method proposed in an embodiment of the present application. The method can be executed by an electronic device, and its software architecture can be as follows: Figure 3 As shown, the hardware structure can be as follows Figure 6As shown, this application does not make any specific limitation to this.

[0137] The image processing method includes: at a first moment, the mode of the first application is the first mode, in response to a first operation on the first application, a first photographed image is obtained by using a first encoder and a first decoder, the first application is an application with a photographing function, the first operation is used to trigger the first application to shoot an image, the first encoder is used to extract features of multiple frames of images collected by the first application, the first decoder is used to obtain the first photographed image based on image features output by the first encoder in the first mode, and the mode of the first application is related to the shooting mode and shooting parameters; at a second moment, the mode of the first application is the second mode, in response to the first operation, a second photographed image is obtained by using the first encoder and the second decoder, the second decoder is used to obtain the second photographed image based on image features output by the first encoder in the second mode, the second mode is different from the first mode, the second moment is after the first moment, and between the first moment and the second moment, the electronic device does not release the first encoder.

[0138] Optionally, the first application may be the camera application described above, or any other application having a shooting function or any application that can call a camera. When the first application is a camera application, the first operation may be the operation of the photo button in the camera application interface described above, or any other operation that can trigger the first application to capture an image. The first moment may be moment 2 in the above method 400, and the second moment may be moment 3 in the above method 400. The first decoder may be decoder A in the above method 400, and the second decoder may be decoder B in the above method 400. The present application does not limit this.

[0139] In an embodiment of the present application, after the electronic device obtains the first photographed image using the first encoder and the first decoder at a first moment, at a second moment after the first moment, in response to a first operation that triggers the first application to take an image, the electronic device obtains the second photographed image using the first encoder and the second decoder, wherein between the first moment and the second moment, the electronic device does not release the first encoder, so that the first encoder does not need to be repeatedly loaded during the running of the first application, which is conducive to simplifying the workflow of the electronic device, improving the image processing efficiency of the electronic device, reducing the operating burden of the electronic device, and improving the user experience.

[0140] Optionally, the first encoder may be a common encoder for all decoders in the electronic device described above, or an encoder applicable to a part of the decoders included in the electronic device. Other targeted decoders may also be set for another part of the decoders included in the electronic device. This application does not make any specific limitations on this.

[0141] In a possible implementation, when the first encoder is a common encoder for all decoders included in the electronic device described above, the camera algorithm model needs to be updated, and online rapid modification can be achieved. For example, if the number of frames input to the first decoder needs to be increased (for example, two short frames need to be added), the first encoder part does not need to be modified or iterated, and only two short frame calls need to be added. After the first decoder part modifies the number of input channels of the first layer, it can inherit the historical training parameters, quickly complete the algorithm modification through R&D training, and push it to the user-side electronic device in the form of online updates.

[0142] As an optional embodiment, when the first encoder and the first decoder are not completely matched, the above-mentioned obtaining the first photographed image using the first encoder and the first decoder includes: obtaining the first photographed image using the first encoder, the second encoder and the first decoder, and the second encoder is used to process the image features output by the first encoder in the first mode to adapt to the first decoder.

[0143] Among them, the second encoder can have similar capabilities to the encoder A described above, which will not be repeated here.

[0144] The method provided in the embodiment of the present application is advantageous in that, when the encoder and decoder are decoupled, each decoder has an encoder adapted thereto, which is advantageous in simplifying the workflow of electronic devices, improving the image processing efficiency of electronic devices, and reducing the operating burden of electronic devices, while maintaining the working performance of the encoder and / or decoder in the electronic device.

[0145] As an optional embodiment, the first decoder is loaded at a first moment in response to a first operation on a first application.

[0146] Optionally, the second encoder may also be loaded at a first moment in response to a first operation on a first application, but this application is not limited to this.

[0147] In the method provided in the embodiment of the present application, the first decoder can be loaded at a first moment in response to a first operation for a first application, and the timing of loading the first encoder and the first decoder can be different. This approach is conducive to reducing the operating pressure of the electronic device in the first application during the image capturing and image processing stages, and is conducive to improving image processing efficiency.

[0148] As an optional embodiment, before the first moment, the method further includes: loading a first encoder.

[0149] In the method provided in the embodiment of the present application, the first encoder can be loaded at a first moment, and the first decoder can be loaded at a first moment in response to a first operation for a first application. The timing of loading the first encoder and the first decoder can be different. This method is conducive to reducing the operating pressure of the electronic device in the first application during the image capturing and image processing stages, and is conducive to improving image processing efficiency.

[0150] As an optional embodiment, before the first moment, the method of loading the first encoder may include the following three: loading the first encoder in response to a second operation, the second operation being used to start the first application; or, loading the first encoder in response to a third operation, the third operation being used to set the mode of the first application; or, loading the first encoder when the first application has been started for a preset duration.

[0151] As an optional embodiment, the method further includes: at a third moment, in response to a fourth operation, releasing the first encoder, the fourth operation being used to close the first application.

[0152] In the embodiment of the present application, the first encoder is released only when the first application is closed. In this way, the first encoder only needs to be loaded once when the first application is run, which is conducive to simplifying the workflow of the electronic device, saving computing power of the electronic device and improving efficiency.

[0153] As an optional embodiment, the first encoder includes one or more of S encoders trained respectively for S types of frames in the multi-frame image, where S is a non-negative integer.

[0154] Optionally, the types of frames in the multi-frame images include one or more of long frames, short frames or ordinary frames.

[0155] In the related technology, long frames, short frames and ordinary frames can be spliced ​​together and input into an encoder with comprehensive analysis capabilities. Since such an encoder model needs to have the ability to extract features from the three frames, the encoder has a large number of channels and its parameters are usually large, which occupies a lot of memory in the electronic device.

[0156] The first encoder in the embodiment of the present application may include one or more of a long frame encoder, a short frame encoder, and a normal frame encoder. Among them, the long frame encoder is used to perform feature extraction processing on the long frames in the image frames to be processed, the short frame encoder is used to perform feature extraction processing on the short frames in the image frames to be processed, and the normal frame encoder is used to perform feature extraction processing on the normal frames in the image frames to be processed. Since any of the long frame encoder, the short frame encoder, and the normal frame encoder only needs to perform feature extraction on one type of image frame, in this way, the scene that a single encoder needs to face becomes simpler. On the one hand, it is conducive to reducing the difficulty of encoder training and development. On the other hand, it is conducive to reducing the number of encoder parameters, which is conducive to reducing the occupation of the encoder's memory for electronic equipment operation, and is conducive to reducing the operating burden of electronic equipment.

[0157] In some implementations, the long frame encoder can process the long frames in the image frames to be processed frame by frame or in batches, the short frame encoder can process the short frames in the image frames to be processed frame by frame or in batches, and the normal frame encoder can process the normal frames in the image frames to be processed frame by frame or in batches. This application does not make any specific limitations on this.

[0158] Figure 5 The operation flow when the first encoder includes a long frame encoder, a short frame encoder and a normal frame encoder is shown as an example. Figure 5 As shown, after the long frame, short frame and normal frame in the image frame to be processed are respectively subjected to feature extraction by the long frame encoder, the short frame encoder and the normal frame encoder, the image features extracted by the long frame encoder, the short frame encoder and the normal frame encoder can be spliced ​​and input into the first decoder to obtain the first photographed image.

[0159] In one implementation, Figure 5 In the process shown, the first encoder can be loaded before the user clicks the photo button and released when the first application is closed. The first decoder can be loaded when the user clicks the photo button and released after the first photo image is output.

[0160] The image processing method of the embodiment of the present application has been described above, and the related device for performing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the related device provided by the embodiment of the present application can perform the steps in the above method.

[0161] Figure 6 FIG. 6 is a schematic diagram showing the hardware structure of an electronic device 600 provided in an embodiment of the present application. Figure 6As shown, the electronic device may include a processor 610, a memory 620, a display screen 630, a sensor module 640, a universal serial bus (USB) interface 650, a charging management module 660, a power management module 661, a battery 662, a camera 670, and the like.

[0162] The processor 610 may include one or more processing units, for example, the processor 610 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0163] The memory 620 may be used to store computer executable program codes, where the executable program codes include instructions, such as program codes corresponding to the methods in the embodiments of the present application.

[0164] The camera 670, which may also be referred to as an image sensor, is used to capture static images or videos. In some embodiments, the electronic device may include 1 or P cameras, where P is a positive integer greater than 1. In the embodiment of the present application, the camera may be used to capture image frames to implement a photo taking function.

[0165] The display screen 630 is used to display images, videos, etc. The display screen 630 includes a display panel. In some embodiments, the electronic device may include 1 or Q display screens 630, where Q is a positive integer greater than 1. The electronic device implements the display function through a GPU, a display screen 630, and an application processor. In the embodiment of the present application, the display screen 630 can be used to display a preview interface of a camera application, a captured image, etc.

[0166] It should be understood that in order to implement the functions described in the above embodiments, the electronic device may include hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the method steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0167] The embodiment of the present application can divide the functional modules of the device implementing the method according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0168] The image processing method provided in the embodiment of the present application can be applied to an electronic device with a camera function. The software structure of the electronic device can be as follows: Figure 3 As shown in , the hardware structure can be Figure 6 As shown in , the specific form of the electronic device can refer to the above description and will not be repeated here.

[0169] Figure 7 The schematic diagram of the structure of a chip 700 provided by the embodiment is exemplarily shown. The chip 700 includes one or more than two (including two) processors 701 , a communication line 702 , a communication interface 703 and a memory 704 .

[0170] In some implementations, the memory 704 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0171] The method described in the above embodiment of the present application can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 701 or an instruction in the form of software. The above processor 701 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 701 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0172] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 704, and the processor 701 reads the information in the memory 704 and completes the steps of the above method in combination with its hardware.

[0173] The processor 701 , the memory 704 , and the communication interface 703 may communicate with each other via the communication line 702 .

[0174] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0175] In the embodiment of the present application, the chip 700 may also be a chip system, such as a system on chip (SOC), which is not limited in the present application.

[0176] An embodiment of the present application provides an electronic device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory stores computer program code, and the computer program code includes computer instructions; the one or more processors call the computer instructions to enable the electronic device to execute the method in the above embodiment.

[0177] The embodiment of the present application provides a chip or a chip system. The chip or chip system is applied to an electronic device, and the chip or chip system includes one or more processors, and the one or more processors are used to call computer instructions to enable the electronic device to execute the method in the above embodiment. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, and will not be repeated here.

[0178] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are run on an electronic device, the electronic device executes the method in the above embodiment. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0179] In one possible implementation, a computer readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile discs (DVD), floppy disks and Blu-ray discs, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer readable media.

[0180] An embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on an electronic device, the electronic device executes the method in the above embodiment.

[0181] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: At a first moment, the mode of the first application is the first mode. In response to a first operation on the first application, a first photographed image is obtained by using a first encoder and a first decoder. The first application is an application with a photographing function. The first operation is used to trigger the first application to photograph an image. The first encoder is used to extract features of multiple frames of images collected by the first application. The first decoder is used to obtain the first photographed image based on image features output by the first encoder in the first mode. The mode of the first application is related to a photographing mode and a photographing parameter. At a second moment, the mode of the first application is a second mode. In response to the first operation, a second photographic image is obtained by using the first encoder and the second decoder. The second decoder is used to obtain the second photographic image based on the image features output by the first encoder in the second mode. The second mode is different from the first mode. The second moment is after the first moment. Between the first moment and the second moment, the electronic device does not release the first encoder. Before the first moment, the method further includes: Load the first encoder.

2. The method according to claim 1, characterized in that: The loading of the first encoder comprises: In response to a second operation, loading the first encoder, wherein the second operation is used to start the first application; or, In response to a third operation, loading the first encoder, the third operation being used to set a mode of the first application; or, When the first application has been started for a preset time, the first encoder is loaded.

3. The method according to claim 2, characterized in that The method further comprises: At a third moment, in response to a fourth operation, the first encoder is released, and the fourth operation is used to close the first application.

4. The method according to claim 3, characterized in that The method of obtaining a first photographed image by using a first encoder and a first decoder includes: The first photographed image is obtained by using the first encoder, the second encoder and the first decoder, and the second encoder is used to process the image features output by the first encoder in the first mode to adapt to the first decoder.

5. The method according to claim 4, characterized in that The first decoder is loaded at the first time in response to a first operation for a first application.

6. The method according to any one of claims 1 to 5, characterized in that The first encoder includes one or more of S encoders trained respectively for S types of frames in the multi-frame images, where S is a non-negative integer.

7. The method according to claim 6, characterized in that The types of frames in the multi-frame images include one or more of long frames, short frames or ordinary frames.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 7.

9. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 7.

Images