Image enhancement method and electronic device

Abstract

The application provides an image enhancement method and an electronic device, and relates to the technical field of image processing. The model for image enhancement of a mode is obtained through a base model and a branch network suitable for the mode. In this way, the storage pressure of the device for storing the model is reduced while the image enhancement effect is ensured. The image enhancement method comprises the following steps: obtaining a first image in a first mode; processing the first image based on a first model to obtain a second image; wherein the first model is obtained based on a base model and a first branch network; the base model is a general model for image enhancement in multiple modes, and the first branch network is suitable for feature extraction of images in the first mode; and outputting the second image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image enhancement method and electronic device. Background Technology

[0002] Processing raw images using artificial intelligence (AI) models can improve their display quality. Images captured in different modes exhibit different display characteristics. For example, images captured in night scene mode generally contain noise, while images captured in wide aperture mode often show a blurred background. To enhance image quality, targeted processing of images from different modes is necessary.

[0003] In related technologies, the same model is pre-trained separately to obtain models corresponding to each mode. The device deploys models corresponding to each mode. After acquiring the original image, the model corresponding to the mode to which the original image belongs can be obtained. Subsequently, the device uses this model to enhance the original image, thereby improving its quality.

[0004] However, deploying multiple models simultaneously on the device consumes a large amount of storage space, resulting in significant storage pressure on the device. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an image enhancement method and electronic device. By utilizing shared model parameters across different modes, the total number of model parameters used to process images in various modes is reduced, thus helping to lower the storage pressure on the device. Furthermore, it does not significantly impact the image enhancement effect across different modes, exhibiting good universality.

[0006] Firstly, this application provides an image enhancement method that can be applied to electronic devices or servers. This application uses the application of the image enhancement method to an electronic device as an example for illustration. The image enhancement method can also be applied to a server; for details, please refer to the description on the electronic device side, which will not be elaborated upon here.

[0007] The image enhancement method includes: an electronic device acquiring a first image under a first mode. Here, "mode" refers to the functional mode or scene mode used when acquiring the image; images under different modes are acquired using different acquisition parameters (such as aperture, focal length, etc.). The electronic device processes the first image based on a first model to obtain a second image; wherein the first model is based on a base model and a first branch network; the base model is a general model used for image enhancement under multiple modes, and the first branch network is suitable for feature extraction of images under the first mode. The electronic device outputs the second image.

[0008] In summary, this embodiment sets up corresponding branch networks for different modes, with multiple branch networks sharing a single base model. Different branch networks are loaded into the base model to obtain models corresponding to different modes. By decoupling the mode-specific models into the base model and the branch networks corresponding to each mode, the number of model parameters that need to be stored is reduced, helping to alleviate the storage pressure on the device. Furthermore, since different models correspond to different branch networks, good image enhancement effects can be obtained for images in different modes.

[0009] In one possible implementation, the base model is an artificial intelligence model. Optionally, the base model is a deep learning model, such as a generative model. Exemplarily, the base model includes an encoder, an intermediate network layer, and a decoder. The encoder encodes a first image to obtain input data for the intermediate network layer; the intermediate network layer performs feature processing on the first image; and the decoder decodes the second image based on the output data of the intermediate network layer.

[0010] For example, the intermediate network layer is a collection of network layers connected in series. The intermediate network layer is used to extract features in the implicit space to learn the pixel distribution characteristics of the first image (such as texture, contours, edges, etc.), thereby better performing image enhancement.

[0011] In one possible implementation, before the electronic device processes the first image based on the first model to obtain the second image, the method further includes: the electronic device replacing the original branch network in the base model based on the first branch network to obtain the first model.

[0012] For example, after the electronic device replaces the original branch network with the first branch network: during the image enhancement process in the first mode, the first branch network performs feature extraction based on the input data of the original branch network to generate the output data of the first branch network; the output data of the first branch network is used to replace the output data of the original branch network as the input data of other network layers in the first model.

[0013] In some embodiments, after determining the first mode to which the first image belongs, the electronic device obtains the configuration information corresponding to the first mode; subsequently, the electronic device obtains the first branch network based on the configuration information corresponding to the first mode.

[0014] Optionally, the first mode to which the first image belongs can be acquired synchronously with the first image.

[0015] For example, in a shooting scenario, when the first image is acquired by the image acquisition device, the acquisition device synchronously sends the first image and the mode to which the first image belongs to the electronic device. The acquisition device can be an external image acquisition device connected to the electronic device.

[0016] For example, the detailed information of the first image includes a first mode; after receiving the first image sent by another device, the electronic device reads the detailed information of the first image to obtain the first mode.

[0017] Optionally, a mapping relationship exists between the mode and the configuration information corresponding to the mode. After determining the first mode, the electronic device determines the storage location of the first branch network and the first identifier of the original branch network to be replaced based on the configuration information corresponding to the first mode. The first identifier is used to identify the original branch network waiting to be replaced. For example, the first identifier may be the name, sequence number, etc. of the original branch network waiting to be replaced. For example, the storage location of the first branch network is the storage address of the first branch network in the electronic device.

[0018] By replacing the original branch network in the base model with the branch network corresponding to the model, the network structure of the base model is changed, resulting in a model corresponding to each mode. This allows different modes to correspond to different models, improving the image enhancement method's ability to process images under different modes. Furthermore, since multiple mode-corresponding models are based on the same base model, they share common model parameters, thus reducing the total number of parameters.

[0019] In one possible implementation, the electronic device replaces the original branch network in the base model with a first branch network to obtain a first model, including: the electronic device loading the base model; the electronic device loading the first branch network based on a first mode; and the electronic device replacing the original branch network in the base model with the first branch network to obtain the first model.

[0020] For example, the input of the first branch network is connected to the input of the original branch network, and the output of the first branch network is connected to the output of the original branch network, so that the input data of the first branch network is consistent with the input data of the original branch network, and the output data of the first branch network replaces the output data of the original branch network and participates in the subsequent calculation process in the first model.

[0021] In one possible implementation, the electronic device loads the base model and the branch networks corresponding to m modes into memory, and pre-mounts the branch networks corresponding to the m modes onto the base model, where m is a positive integer. This helps to acquire the first model in real time and shortens the waiting time for the first image to undergo image enhancement.

[0022] Optionally, a first switch is provided between the base model and the branch networks corresponding to each mode. The first switch is used to determine whether the branch network is attached to the base model. For example, after being attached to the base model, the branch network participates in feature extraction for image enhancement. For example, when the first switch is on, the input data of the original branch network is transmitted to the first branch network, and the output data of the first branch network replaces the output data of the original branch network.

[0023] In one example, the first switch is implemented based on a logical judgment statement. Each first switch includes a logical judgment statement to determine whether the current image enhancement process is performed on an image in the target mode. The target mode is the mode corresponding to the branch network to which the first switch is connected. For example, for first switch 1 connected to the first branch network, the target mode is the first mode.

[0024] In another example, after determining the first mode, the electronic device sends a first control message to each first switch, with each first switch corresponding to one control message.

[0025] In one possible implementation, the intermediate network layer of the pedestal model includes a first part and a second part; the intermediate network layer is used to extract feature information of the first image, the input data of the intermediate network layer is obtained based on the encoding of the first image, and the output data of the intermediate network layer is used to decode to obtain the second image; loading the pedestal model includes: loading the first part; during the processing of the first image, after the feature extraction task for the first part is completed, the second part is loaded.

[0026] In one possible implementation, the base model includes an intermediate network layer; the intermediate network layer is used for feature extraction, the input data of the intermediate network layer is obtained based on the encoding of a first image, and the output data of the intermediate network layer is used for decoding to obtain a second image; wherein, the intermediate network layer includes a first network layer, and the first model is obtained by replacing the first original branch network in the first network layer with the first branch network.

[0027] In one possible implementation, the base model is a generative model, and the first branch network is used to replace the original branch networks within the residual blocks of the base model. That is, the replaced original branch networks come from the intermediate network layer. For example, the intermediate network layer includes multiple residual blocks connected in series. Each residual block includes n original branch networks, and the first network layer is used to replace at least one of the n original branch networks, where n is a positive integer less than or equal to the total number of residual blocks in the intermediate network layer.

[0028] For example, the first branch networks corresponding to the multiple residual blocks are different; the first branch networks corresponding to the multiple residual blocks are obtained by image pre-training in the corresponding mode.

[0029] In one possible implementation, the first network layer includes a first original branch network and a second original branch network; during the first model's processing of the first image, the output data of the first network layer is obtained based on the output data of the second original branch network and the output data of the first branch network; the output data of the first network layer is used as input data for the next network layer in the intermediate network layer, or for decoding to obtain the second image.

[0030] In one example, the intermediate network layer is a U-shaped convolutional network (U-Net), the first network layer is a residual block in the U-Net, the first original branch network is the network structure in the residual block used for feature extraction, and the second original branch network is used to directly use the input data of the residual block as the output data to realize skip connections between the residual blocks and reduce the gradient decay problem.

[0031] By keeping the second original branch network in the first network layer unchanged and replacing the first original branch network with the first branch network, the feature extraction capability of the image in the first mode is improved, while the structure of the replaced first network layer remains largely unchanged. This helps to reduce the impact of attaching the first branch network to the base model on the overall model structure of the base model.

[0032] In one possible implementation, the first branch network includes at least two linear network layers, where for any two adjacent linear layers, the dimension of the output data of one linear layer is equal to the dimension of the input data of the other linear layer. Optionally, the first branch network includes two linear layers.

[0033] Optionally, the first branch network has a simpler structure than the original branch network; it also includes fewer model parameters. On the one hand, fewer model parameters in the first branch network help reduce the storage space occupied by the first branch network, thus reducing the storage pressure on the device.

[0034] On the other hand, fewer model parameters mean that the electronic device needs to perform less computation, which helps to reduce the computational complexity of feature extraction in the first branch network, reduce the computational pressure on the electronic device, and shorten the time it takes for the first model to process the first image.

[0035] In one possible implementation, the method further includes: determining a second network layer in the first model based on a first mode, wherein the second network layer does not participate in image enhancement of the image in the first mode; replacing the second network layer in the first model with a third network layer to obtain a second model; wherein the function of the third network layer is the same as that of the second network layer, and the total number of model parameters included in the third network layer is less than the total number of model parameters included in the second network layer; and processing the first image based on the first model to obtain a second image, including: processing the first image based on the second model to obtain a second image.

[0036] In this way, by replacing the second network layer with the third network layer, the total number of model parameters of the second model is less than the total number of model parameters of the first model, which helps to reduce the computational complexity in the image enhancement process.

[0037] In one possible implementation, the method further includes: obtaining image attributes based on a first image; wherein the image attributes include at least one of the following: the focal length of the first image or the type of a first object; and processing the first image based on a first model to obtain a second image, including: processing the first image based on the first model to obtain a second image when the image attributes meet the processing conditions.

[0038] In one example, the image attributes include the focal length of the first image, and the processing condition includes: the focal length of the first image is within a preset focal length. For example, if the focal length of the first image is within the preset focal length, the electronic device performs image enhancement; if the focal length of the first image is outside the preset focal length, the electronic device does not perform the image enhancement method.

[0039] The focal length at which the first image was captured reflects its degree of blur, allowing for a determination of whether image enhancement processing is needed. For first images with low blur levels, the difference between the enhanced second image and the first image is minimal. In such cases, skipping the image enhancement process reduces the total number of images requiring enhancement by the electronic device, thus lowering its computational load. When the first application is a camera app, this approach ensures the quality of the captured image while minimizing the user's waiting time for image enhancement, thereby improving the user's photography experience.

[0040] In another example, the image attributes include the type of the first object, and the processing condition includes: the type of the first object is not included in the preset types. The preset types are pre-set, such as including at least one of the following: sensitive icons, etc.

[0041] In yet another example, when the image attributes include the shooting focal length of the first object and the type of the first object, the processing conditions include: the shooting focal length of the first image is within a preset focal length, and the preset type includes the type of the first object.

[0042] This helps avoid enhancing images containing sensitive content and reduces unnecessary image enhancement.

[0043] In one possible implementation, the method further includes: fusing the first image and the second image to generate a third image; and outputting the second image, including: outputting the third image.

[0044] This helps to improve image clarity while preserving the image style of the first image, which helps meet the image enhancement needs in scenarios such as live-action shooting.

[0045] In one possible implementation, a third image is generated by fusing a first image and a second image, including: acquiring first information of the first image; wherein the first information is used to indicate the pixel values ​​of a first region in the first image, and the pixel values ​​in the first region change smoothly; acquiring second information of the second image; wherein the second information is used to indicate the pixel values ​​of the first region in the second image; and adjusting the pixel values ​​of the first region in the first image based on the first information and the second information to obtain the third image.

[0046] In this embodiment, image enhancement is used to change the pixel values ​​of some pixels in the original image (first image), thereby reducing the impact of image enhancement on the image style of clearer areas in the first image and helping to compensate for color distortion caused by image enhancement.

[0047] Secondly, this application provides an electronic device, which includes a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory, causing the electronic device to perform the methods described in the first aspect and any possible implementation thereof. Specifically, the electronic device acquires a first image in a first mode; the electronic device processes the first image based on a first model to obtain a second image; wherein the first model is obtained based on a base model and a first branch network; the base model is a general model for image enhancement in multiple modes, and the first branch network is suitable for feature extraction of images in the first mode; the electronic device outputs the second image.

[0048] In one possible implementation, when the processor reads computer instructions from memory, it also causes the electronic device to perform: replacing the original branch network in the base model with a first branch network to obtain a first model.

[0049] In one possible implementation, the electronic device replaces the original branch network in the base model with a first branch network to obtain a first model, including: the electronic device loading the base model; the electronic device loading the first branch network based on a first mode; and the electronic device replacing the original branch network in the base model with the first branch network to obtain the first model.

[0050] In one possible implementation, the electronic device performs image enhancement on a first image and a second image in a second mode in parallel, wherein the first mode and the second mode are different. The electronic device loads a first branch network and a second branch network. At this time, a first model and a third model are loaded in memory, wherein the third model is obtained based on the base model and the second branch network. Assuming that the first branch network and the second branch network are used to replace the first branch structure in the same first network layer, then the first network layer is simultaneously connected to both the first branch network and the second branch network.

[0051] In one possible implementation, the intermediate network layer of the pedestal model includes a first part and a second part; the intermediate network layer is used to extract feature information of the first image, the input data of the intermediate network layer is encoded based on the first image, and the output data of the intermediate network layer is used to decode to obtain the second image; when the processor reads computer instructions from the memory, it also causes the electronic device to execute: loading the first part; during the processing of the first image, after the feature extraction task for the first part is completed, loading the second part.

[0052] Optionally, there is an execution order between the first part and the second part. During image enhancement, the feature extraction task of the first part is executed before the feature extraction task of the second part; wherein, the feature extraction task is to extract features sequentially from the network layers included in a certain part to obtain feature information of the first image at a deeper level.

[0053] In the base model, the intermediate network layers have a large number of model parameters, requiring significant memory during runtime. To reduce runtime memory consumption, this embodiment supports splitting the intermediate network layers into multiple independent parts, allowing the network layers within the intermediate layers to be loaded into memory in batches, thereby reducing memory usage during model execution. This helps prevent stuttering during image enhancement on electronic devices.

[0054] In one possible implementation, the base model includes an intermediate network layer; the intermediate network layer is used to extract feature information of the first image, the input data of the intermediate network layer is encoded based on the first image, and the output data of the intermediate network layer is used to decode to obtain the second image; wherein, the intermediate network layer includes a first network layer, and the first model is obtained by replacing the first original branch network in the first network layer with the first branch network.

[0055] In one possible implementation, the first network layer includes a first original branch network and a second original branch network; during the first model's processing of the first image, the output data of the first network layer is obtained by merging the output data of the second original branch network and the output data of the first branch network; the output data of the first network layer is used as input data for the next network layer in the intermediate network layer, or for decoding to obtain the second image.

[0056] By keeping the second original branch network in the first network layer unchanged and replacing the first original branch network with the first branch network, the feature extraction capability of the image in the first mode is improved, while the structure of the replaced first network layer remains largely unchanged. This helps to reduce the impact of attaching the first branch network to the base model on the overall model structure of the base model.

[0057] In one possible implementation, the first branch network includes at least two linear network layers, where for two adjacent linear layers, the dimension of the output data of one linear layer is equal to the dimension of the input data of the other linear layer.

[0058] Optionally, the first branch network has a simpler structure than the original branch network; it also includes fewer model parameters. On the one hand, fewer model parameters in the first branch network help reduce the storage space occupied by the first branch network, thus reducing the storage pressure on the device.

[0059] On the other hand, fewer model parameters mean that the electronic device needs to perform less computation, which helps to reduce the computational complexity of feature extraction in the first branch network, reduce the computational pressure on the electronic device, and shorten the time it takes for the first model to process the first image.

[0060] In one possible implementation, the base model is a generative model, and the first branch network is used to replace the original branch network within the residual block in the base model.

[0061] In one possible implementation, when the processor reads computer instructions from memory, it also causes the electronic device to perform: determining a second network layer in the first model based on a first mode, wherein the second network layer does not participate in image enhancement of the image in the first mode; replacing the second network layer in the first model with a third network layer to obtain a second model; wherein the function of the third network layer is the same as that of the second network layer, and the total number of model parameters included in the third network layer is less than the total number of model parameters included in the second network layer. In this case, the electronic device processes the first image based on the first model to obtain the second image, including: processing the first image based on the second model to obtain the second image.

[0062] In one possible implementation, when the processor reads computer instructions from memory, it also causes the electronic device to perform: acquiring image attributes based on a first image; wherein the image attributes include at least one of the following: the focal length of the first image or the type of a first object. In this case, the electronic device processes the first image based on a first model to obtain a second image, including: processing the first image based on the first model to obtain the second image if the image attributes satisfy processing conditions.

[0063] In one possible implementation, when the processor reads computer instructions from memory, it also causes the electronic device to perform the following: merging the first and second images to generate a third image. In this case, the electronic device outputting the second image can be implemented as outputting the third image.

[0064] In one possible implementation, the electronic device fuses a first image and a second image to generate a third image, including: the electronic device acquiring first information of the first image; wherein the first information is used to indicate the pixel values ​​of a first region in the first image, and the pixel values ​​in the first region change smoothly; the electronic device acquiring second information of the second image; wherein the second information is used to indicate the pixel values ​​of the first region in the second image; and the electronic device adjusting the pixel values ​​of the first region in the first image based on the first information and the second information to obtain the third image.

[0065] Thirdly, a server is provided for executing the methods described in the first aspect and any implementation thereof.

[0066] Fourthly, an image enhancement system is provided, comprising a server and an electronic device. The electronic device sends a first image in a first mode to the server; the server processes the first image based on a first model to obtain a second image. The server also sends the first image to the electronic device.

[0067] Fifthly, a computer-readable storage medium is provided, on which computer instructions are stored, which, when executed by a processing circuit, implement the method as described in any possible embodiment of the first aspect.

[0068] In a sixth aspect, a chip system is provided, the chip system including a processing circuit and a storage medium, the storage medium storing computer instructions; when the computer instructions are executed by the processing circuit, they implement the method as described in any possible embodiment of the first aspect.

[0069] In a seventh aspect, a computer program product comprising instructions is provided, which, when run on a computer, causes the computer to perform the method as described in any of the possible embodiments of the first aspect.

[0070] The technical effects corresponding to any of the implementation methods in aspects two through seven can be found in the first aspect and the technical effects corresponding to any of the implementation methods in the first aspect, and will not be repeated here. Based on the implementation methods provided in the above aspects, this application can further combine them to provide more implementation methods. Attached Figure Description

[0071] Figure 1 This is a schematic diagram illustrating the effects of image enhancement methods in related technologies;

[0072] Figure 2 This is a schematic diagram illustrating the effect of the image enhancement method provided in the embodiments of this application;

[0073] Figure 3 This is a block diagram of the architecture of the communication system provided in the embodiments of this application;

[0074] Figure 4 This is one of the structural schematic diagrams of the electronic device provided in the embodiments of this application;

[0075] Figure 5 This is one of the connection diagrams of the functional modules provided in the embodiments of this application;

[0076] Figure 6 This is the second schematic diagram showing the connection of the functional modules provided in the embodiments of this application;

[0077] Figure 7 A schematic diagram of the software structure of the electronic device provided in the embodiments of this application;

[0078] Figure 8 One of the schematic flowcharts of the image enhancement method provided in the embodiments of this application;

[0079] Figure 9 A schematic diagram of the mounting of the base model and multiple branch networks provided in the embodiments of this application;

[0080] Figure 10 A schematic diagram of the structure of the first model provided in the embodiments of this application;

[0081] Figure 11 This is a schematic diagram of the distributed loading of the intermediate network provided in an embodiment of this application;

[0082] Figure 12 A second schematic flowchart illustrating the image enhancement method provided in this application embodiment;

[0083] Figure 13 This is a second schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0084] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0085] The terms "comprising" and "having," and any variations thereof, used in the description of the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0086] Hereinafter, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature.

[0087] In this application, the terms "exemplarily" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0088] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone.

[0089] Artificial intelligence generated content (AIGC) is a key technology for enhancing image sharpness and is increasingly being applied to the field of image enhancement. AIGC utilizes artificial intelligence models for image enhancement. Before applying the AI ​​model, the AIGC model can be fine-tuned based on images in different modes to obtain models corresponding to different modes.

[0090] Here, "mode" refers to the scene mode and / or function mode used to acquire the image. For example, mode types include one of the following: standard mode, large aperture mode, panorama mode, sports mode, night scene mode, etc. Different modes are adapted to different shooting parameters, resulting in images with different display characteristics.

[0091] The standard mode is an automatic shooting mode suitable for everyday scenes. The large aperture mode uses a large aperture to reduce depth of field and blur the background; images taken in this mode may have a blurred background. The panorama mode captures multiple images and stitches them together to create an ultra-wide image; images taken in this mode may have blur at the seams between the images. The sports mode uses a high-speed shutter to capture moving objects; images taken in this mode may have horizontal background blur. The night mode is specifically designed for shooting in low-light conditions such as at night; images taken in this mode may have some noise.

[0092] As mentioned above, since images in different modes have different display characteristics, they need to be processed in a targeted manner.

[0093] In one possible implementation, the image enhancement system maintains a fine-tuned model for each mode, meaning that different modes correspond to different models, and the different models are independent of each other. For example, different models do not share model parameters, or the different models are stored independently.

[0094] like Figure 1 As shown, images in mode 1 are enhanced using model 1; images in mode 2 are enhanced using model 2; and so on, images in mode N are enhanced using model N, where N is a positive integer.

[0095] However, each model contains a large number of model parameters, which causes the image enhancement system composed of multiple models to expand rapidly when different modes are deployed on the device at the same time, occupying a large storage area and increasing the storage pressure on the device.

[0096] Furthermore, the model needs to be loaded from the hard drive into memory before it can be run, and memory has limited storage capacity, typically unable to store multiple modes simultaneously. This results in frequent loading and unloading of the model from memory in the above method. For example, there might be image 1 in mode 1 and image 2 in mode 2, where modes 1 and 2 are different. The device uses the model corresponding to mode 1 to process image 1. Subsequently, the electronic device unloads the model corresponding to mode 1 from memory, loads the model corresponding to mode 2 into memory, and uses the model corresponding to mode 2 to process image 2.

[0097] However, the loading process takes a certain amount of time, which makes image enhancement time-consuming and is not conducive to improving the efficiency of image enhancement.

[0098] To address the aforementioned issues, this application provides an image enhancement method. Specifically, models corresponding to different modes are assembled based on a base model and a branch network corresponding to each mode. In other words, the base model can support multiple sets of model parameters, each set of model parameters being the model parameters in the branch network corresponding to a specific mode.

[0099] In this application, the pattern corresponding to one pattern is obtained based on the base model and the branch network corresponding to that pattern. For example... Figure 2 As shown, images in Mode 1 are enhanced based on the pedestal model and branch network 1; images in Mode 2 are enhanced based on the pedestal model and branch network 2. Similarly, images in Mode N are enhanced based on the pedestal model and branch network N.

[0100] In this way, the base model and the branch networks corresponding to each mode can be stored independently in the storage space. This also allows models corresponding to different modes to share a single base model.

[0101] Compared to related technologies that store different models separately in the storage space, this solution only needs to store the common base model once in the storage space. Therefore, the storage space only needs to store one base model and the branch networks corresponding to multiple modes. Since the branch networks have fewer network parameters, the total data volume of the image enhancement system is reduced, which helps to reduce the storage space occupied. This helps to reduce the storage pressure on the device.

[0102] Furthermore, the branch networks corresponding to different modes require fewer parameters, and these branch networks can be stored in memory simultaneously. This eliminates the need for loading and unloading models from memory when the mode of an image changes. This reduces model loading time and helps improve image enhancement efficiency.

[0103] When multiple branch networks corresponding to different modes cannot be stored in memory simultaneously, the time taken to load a branch network is shorter than the time taken to load a complete mode corresponding to a certain mode, since the number of parameters required by the branch network is smaller. Therefore, the time taken to load the model can also be shortened.

[0104] like Figure 3 As shown, the image enhancement method can be applied to a communication system 100, which includes an electronic device 10.

[0105] In some embodiments, the image enhancement method can be applied to an electronic device. The electronic device 10 is also referred to as at least one of the following: an access terminal, a user unit, a user station, a mobile station, a mobile station, a remote station, a remote terminal, a mobile device, a wireless communication device, a user agent, or a user apparatus.

[0106] The electronic devices 10 in the embodiments of this application include, but are not limited to: mobile phones, personal computers, tablets, computers with wireless transceiver capabilities, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, vehicle-mounted terminals, and roadside units (RSUs) with terminal functions.

[0107] In some embodiments, the operating system installed on the electronic device 10 includes, but is not limited to, those described above. Or other operating systems.

[0108] Optionally, the electronic device 10 has a first application installed, which provides image processing functions. Optionally, the first application may be a camera application, an image processing application, a social application, etc. The first application may take the form of, but is not limited to, applications installed on the electronic device 10, web pages, or sub-applications within a parent application.

[0109] In one example, image enhancement is performed by an electronic device 10, which deploys a base model and multiple branch networks corresponding to the models. After acquiring an image in a first mode, the electronic device 10 obtains a first model based on the base model and the branch networks corresponding to the first mode (referred to as "first branch networks" in the following embodiments). Subsequently, the first image is processed based on the first model to obtain an enhanced second image.

[0110] In some embodiments, such as Figure 3 As shown, the communication system 100 also includes a server 20.

[0111] Optionally, server 20 is a computing device or network device such as a cloud server or network server. Server 20 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Optionally, server 20 is used to provide backend services for the primary application.

[0112] In another example, image enhancement is performed collaboratively by electronic device 10 and server 20. A first application in electronic device 10 sends a first image in a first mode to server 20. Server 20 processes the first image based on the first model to obtain a second image. Subsequently, server 20 sends the second image back to electronic device 10. In this case, a base model and branch models corresponding to multiple modes are deployed on server 20.

[0113] The structure of the electronic equipment included in the communication system 100 is described below. For example... Figure 4 As shown, the electronic device 10 includes at least the following components (the devices mentioned in the claims include all of these components): processor 110, memory 120, and communication module 130.

[0114] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0115] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0116] Memory 120 includes external memory and internal memory, the internal memory being used to store executable program code. The executable program code includes instructions. Processor 110 executes various functional applications and data processing of electronic device 10 by running the instructions stored in the internal memory. Exemplarily, the internal memory includes a program storage area and a data storage area. Memory 120 may be disposed within processor 110 for storing instructions and data. External memory is a memory card connected to electronic device 10 via an external memory interface.

[0117] Optionally, the processor 110 reads instructions from the memory 120 to acquire a first model in response to acquiring a first image; and processes the first image based on the first model to obtain a second image.

[0118] The communication module 130 is used to implement the communication function of the electronic device 10. The communication module 130 includes functional modules such as an antenna, a mobile communication module, a wireless communication module, a modem processor, and a baseband processor. Optionally, at least some of the functional modules of the mobile communication module can be housed in the processor 110. The wireless communication module is used to support the electronic device 10 in wireless communication.

[0119] Optionally, the communication module 130 is used to send the first image to the server 20, or to receive the first image sent by another device. This other device may include an image acquisition device, such as a camera or webcam.

[0120] In some embodiments, the electronic device 10 further includes an image acquisition unit 140 for acquiring images from the actual environment. Optionally, the image acquisition unit supports multiple functional models and scene modes so that the processor 110, in response to an image acquisition operation, controls the image acquisition unit 140 to acquire images in a certain mode.

[0121] In some embodiments, the image enhancement method uses an electronic device (such as...) Figure 3 The functional modules are implemented in electronic devices 10). Figure 5 As shown, the functional modules include at least: an acquisition module, an enhancement module, and an output module.

[0122] The acquisition module is used to acquire the first image in the first mode.

[0123] An enhancement module is used to process the first image based on the first model to obtain the second image.

[0124] The output module is used to output a second image.

[0125] Optionally, in order to improve image processing quality, such as Figure 6 As shown, the electronic device also includes at least one of the following: a loading module or a stacking module.

[0126] The loading module is used to obtain the first model based on the first branch network and the base model.

[0127] The overlay module is used to obtain a third image based on the first and second images. In this case, an output model is used to output the third image. For a description of each of the above functional modules, please refer to the method-side embodiment below.

[0128] In some embodiments, Figure 3The communication system 100 includes an electronic device 10 equipped with a first application, which is located in the application architecture layer of the software structure. The software structure is described below using the electronic device 10 as an example.

[0129] The software system of electronic device 10 adopts a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. The following example uses the layered architecture Android system to illustrate the software structure of electronic device 10.

[0130] like Figure 7 As shown, the software architecture of an electronic device can be divided into several layers. In some embodiments, from bottom to top, these layers are: kernel layer, system service layer, framework layer, and application layer. Layers communicate with each other through software interfaces. System functions can be tailored, added, or combined at the subsystem granularity in different device deployment scenarios, and each subsystem can also be tailored, added, or combined at the functional granularity.

[0131] The kernel layer is called the kernel abstract layer (KAL), which provides basic kernel capabilities to the upper layers by shielding the differences between multiple kernels. These capabilities include, but are not limited to, process / thread management, memory management, file system, network management, and peripheral device management.

[0132] Kernel Subsystem: Supports the selection of a suitable OS kernel for different resource-constrained devices, including but not limited to Linux kernel, HarmonyOS kernel, LiteOS (lite operating system), etc.

[0133] Driver Subsystem: The driver framework is the foundation for the open system hardware ecosystem, providing unified peripheral access capabilities and a framework for driver development and management. The driver framework includes: display drivers, camera drivers, audio drivers, Bluetooth drivers, sensor drivers, etc.

[0134] The system service layer comprises the core capabilities of the system, providing services to applications through the framework layer. This layer includes, but is not limited to, the following subsystems:

[0135] The system's basic capability subsystem set provides fundamental capabilities for the operation, scheduling, and migration of distributed applications across multiple devices. This set may include distributed soft bus, distributed data management, distributed task scheduling, and Ark multi-language runtime; it may also include multi-modal input subsystem, graphics subsystem, security subsystem, and AI business subsystem.

[0136] Basic software service subsystem set: provides public and general software services; the basic software service subsystem set may include event notification subsystem, telephone service subsystem, multimedia subsystem, etc.

[0137] Enhanced software service subsystem suite: Provides differentiated, capability-enhancing software services for different devices; the enhanced software service subsystem suite may include proprietary business subsystems for smart screens, wearable devices, and the Internet of Things (IoT), etc. Optionally, Figure 6 The enhancement module needs to run the AI ​​model, and the enhancement module is set in the enhancement software service subsystem set in the system service layer.

[0138] Hardware service subsystem set: Provides hardware services; the hardware service subsystem set may include location service subsystem, unified identity and access management (IAM) subsystem, wearable proprietary hardware service subsystem, biometric identification, IoT proprietary hardware service subsystem, etc.

[0139] Distributed task scheduling enables distributed service management (discovery, synchronization, registration, and invocation), supporting remote startup, remote invocation, remote connection, and migration of applications across devices.

[0140] Distributed data management enables data synchronization, data storage, data sharing, and data access across all scenarios and devices.

[0141] The distributed soft bus provides communication-related capabilities for seamless interconnection between multiple devices, including: wireless local area network (WLAN) service capabilities, Bluetooth service capabilities, soft bus, inter-process communication, remote procedure call (RPC), and StarFlash communication capabilities.

[0142] Ark Multilingual Runtime is a unified compilation runtime platform designed to support the joint compilation and execution of multiple programming languages ​​and multiple chip platforms.

[0143] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer includes: the ArkUI framework (which provides a complete infrastructure for UI development of system applications, including UI functions such as components, layouts, animations, and interactive events, as well as a real-time interface preview tool), the user application framework, and the Ability framework (an Ability is a lightweight application; the Ability framework schedules and manages the operation and lifecycle of Abilities). Different devices may have different operating systems, and the APIs they support may also differ.

[0144] The HarmonyOS API is a series of open capabilities provided to support HarmonyOS application development. The HarmonyOS API can be set at the framework layer or independently of the framework layer. The HarmonyOS API includes the Audio API (audio service), Push API (push service), and Account API (account service), among others.

[0145] The application may include system applications and extended / third-party applications. System applications may include the desktop, control bar, settings, contacts, phone, camera, etc., while extended / third-party applications may include social applications, travel applications, etc. Optionally, in this embodiment, a first application is installed in the electronic device. For example, the first application is a camera application among system applications, or an image processing application among third-party applications. Image enhancement methods can be used with the first application in the electronic device.

[0146] The following uses electronic devices (such as...) Figure 3 The electronic device 10 in this embodiment acts as the execution subject, and specifically describes the image enhancement method provided in this application. Optionally, this embodiment is executed by an electronic device. This embodiment uses an electronic device including an acquisition module, an enhancement module, and an output module as an example for illustration. Exemplarily, the electronic device may also include a loading module.

[0147] For example, the image enhancement method can also be implemented via a server (such as...) Figure 3 The server 20 in the middle executes the process. For specific steps, please refer to the embodiment where the electronic device is the execution subject. It will not be repeated here.

[0148] For example, this embodiment uses image enhancement under the first model as an example for illustration. The first mode can be any functional mode or scene mode for acquiring images, and other modes will not be described in detail.

[0149] Please refer to the following: Figure 8 , Figure 8 This is a schematic flowchart of the image enhancement method provided in the embodiments of this application.

[0150] like Figure 8 As shown, the image enhancement method provided in this application embodiment may include some or all of the steps in steps 810 to 840.

[0151] Step 810: The electronic device acquires the first image in the first mode.

[0152] In some embodiments, patterns are used to categorize images with similar display characteristics. For example, images in different patterns have different display characteristics. Display characteristics include blurred objects (such as the subject, background, etc.) and the distribution of noise in the image.

[0153] Optionally, "mode" refers to the image acquisition mode used during image acquisition, such as scene mode or function mode. Mode types include, but are not limited to: standard mode, night scene mode, large aperture mode, and motion mode. For a detailed description of the above modes, please refer to the background section of the invention description above; further details will not be repeated here.

[0154] The first mode is at least one of the above-mentioned standard mode, night mode, large aperture mode, and sports mode, or a combination of at least two of them.

[0155] For example, the first mode is night scene mode. Another example is that the first mode includes a large aperture mode and a motion mode; images in both of these modes suffer from background blurring.

[0156] In some embodiments, the first image in the first mode refers to an image acquired from the actual environment through the first mode.

[0157] Optionally, the first image displays at least one object. The object includes at least one of the following: a complete or partial person, animal, vehicle, building, plant, or landscape, etc. For example, the first image is obtained from an image captured from a real-world environment.

[0158] The first image may be acquired by an electronic device, or it may be sent to the electronic device by another device.

[0159] In one example, the first image is acquired by an electronic device. Exemplarily, the electronic device acquires an image in a first mode by: the electronic device responding to an image acquisition operation to obtain acquisition information; and the electronic device obtaining the first image based on the acquisition information. The acquisition information is acquired by an image acquisition device in the electronic device. The image acquisition device may include a color camera, an infrared camera, etc. The acquisition information is the first image, or the electronic device performs image encoding based on the acquisition information to obtain the first image.

[0160] Image acquisition operations are used to control electronic devices to acquire images. Image acquisition operations include, but are not limited to, triggering operations on controls and buttons; such as clicking a control or pressing the shutter button.

[0161] For example, the first application is a camera application. In response to a click on a first control, the first application drives the image sensor in the electronic device to acquire information based on the parameters corresponding to the first mode. In this case, the first mode is the shooting mode currently in which the first application is operating. The first mode supports adjustment and switching based on user actions.

[0162] In another example, the first image is sent to the electronic device by another device. Exemplarily, the other device sends the first image and detailed information within the first image to the electronic device; based on the detailed information, the electronic device determines that the first image belongs to a first mode. The detailed information of the first image includes the first mode to which the first image belongs.

[0163] Compared to related technologies, this application decouples the models corresponding to different modes into: a common base model and branch networks corresponding to different modes. Since the model corresponding to the first mode needs to be used to process the model corresponding to the first mode, it is also necessary to obtain the first model based on the first branch network and the base model corresponding to the first mode.

[0164] The process of obtaining the first model is described below through step 820.

[0165] Optionally, after acquiring the first image, the electronic device determines the first branch network based on the first pattern; subsequently, the electronic device executes step 820 to obtain the first model.

[0166] For example, the electronic device includes configuration information corresponding to multiple modes; wherein, the configuration information corresponding to the first mode is used to indicate the first branch network. Accordingly, the electronic device determines the first branch network based on the first mode, which can be implemented as follows: the electronic device obtains the configuration information corresponding to the first mode; the electronic device determines the first branch network based on the configuration information. If the first branch network is not stored in storage, the electronic device reads the first branch network from the hard disk into memory. For a detailed description of the first branch network, refer to the embodiment corresponding to step 820.

[0167] Step 820: The electronic device obtains the first model based on the base model and the first branch network.

[0168] The first model is the model corresponding to the first mode, and the first model is used to enhance the image in the first mode.

[0169] In some embodiments, the first model includes a first branch network and a portion of the network structure in the base model.

[0170] Optionally, the pedestal model is a general-purpose model for image enhancement across multiple modes. That is, the pedestal model can also enhance the first image. However, because images in different modes have different display characteristics, the pedestal model cannot differentiate between images in different modes, resulting in low image enhancement quality for images in specific modes.

[0171] Image enhancement is an image processing technique that alters the display effect of an image. Compared to before image enhancement, after image enhancement, parameters such as brightness, contrast, and saturation of the image change, resulting in improved image clarity.

[0172] For example, image 1 is enhanced to obtain image 2. In region 1 of image 1, the pixel values ​​of multiple adjacent pixels change relatively smoothly, resulting in a somewhat blurry visual effect in region 1. In image 2 after image enhancement, details such as edges and textures are added to region 1, making the visual effect of region 1 in image 2 clearer.

[0173] For example, the base model is an artificial intelligence model, such as a generative model. The base model includes an encoder, intermediate network layers, and a decoder. The encoder encodes the image to obtain input data for the intermediate network layers, which then extract features based on their input data to obtain output data. The decoder decodes the output data of the intermediate network layers to obtain the image-enhanced image.

[0174] In the field of image enhancement, feature extraction combines techniques such as convolution, activation, and pooling to learn high-frequency details and low-frequency information in a first image (the image to be enhanced). This allows for the adjustment of pixel values ​​in blurred areas, thereby improving the quality of the enhanced image. For example, high-frequency information includes edges and textures in the image. Low-frequency information includes the overall brightness and color of objects displayed in the image.

[0175] In contrast, the first branch network is dedicated to image enhancement of images in the first mode; that is, the first branch network is suitable for feature extraction of images in the first mode.

[0176] In some embodiments, the branch networks corresponding to multiple modes are obtained through pre-training.

[0177] The following example illustrates the pre-training of the branch network corresponding to the pattern, using the first branch network as an example.

[0178] For example, before applying the first branch network for image enhancement, the model training device pre-trains the first branch network; wherein the model training device is a server or other computer device, and the training data includes multiple images under the first model.

[0179] Specifically, the first branch network is attached to the base model to form the first model before training; the model training device processes the image under the first mode based on the first model before training to obtain the enhanced image; the model training device determines the model loss based on the enhanced image. Subsequently, the model training device adjusts the model parameters of the first branch network based on the model loss until the model loss converges, and the pre-training process ends.

[0180] During pre-training, the model training device keeps the model parameters of the base model fixed and only changes the model parameters of the branch networks corresponding to each mode. This helps the branch networks corresponding to each mode to fully learn the display characteristics of the image under that mode, thereby improving the feature extraction ability of that branch network for the image under that mode. This, in turn, helps improve the quality of image enhancement.

[0181] For example, the model parameters of the first branch network are carried in the application file package. The electronic device obtains the application file package from the server to obtain the model parameters of the first branch network. The application file package is used to install or update the first application on the electronic device.

[0182] In some embodiments, the network structure of the first branch network is simpler than that of the base model; the function of the first branch network is similar to that of some of the original branch networks included in the base model. For example, during image enhancement in the first mode, the first branch network replaces the original branch networks in the base model to extract features from the first image. That is, the first branch network participates in feature extraction of the image in the first mode.

[0183] In some embodiments, the model parameters of the first branch network and the model parameters of the base model are stored independently in the hard drive of the electronic device; the loading of the base model into memory by the electronic device is independent of the loading of the first branch network into memory. For example, the electronic device mounts the first branch network to the base model to obtain the first model.

[0184] In some embodiments, step 820 is a selectively executed step in the image enhancement method. For example, after the first application starts running, the electronic device loads the base model and the branch networks corresponding to N modes into memory; the electronic device pre-mounts the branch networks corresponding to the N modes onto the base model, where N is a positive integer.

[0185] like Figure 9 As shown, N branch networks are mounted on the base model. For example, branch network 910 corresponding to mode 1 includes two linear layers (such as...). Figure 9 The dimension reduction matrix corresponding to the 920th linear layer is A = N(0, σ). 2 (The linear layer 930 corresponds to the upsizing matrix B = 0). For example... Figure 9 As shown, in the image enhancement process under mode 1, the electronic device obtains the model corresponding to mode 1 based on the base model and the branch network 910 corresponding to mode 1. In this way, step 820 does not need to be executed after acquiring the first image, which helps to acquire the first model in real time and shortens the waiting time for the first image to be enhanced.

[0186] Optionally, a first switch is provided between the base model and the branch networks corresponding to the N modes. The first switch is used to determine whether the branch network is attached to the base model.

[0187] In one example, the first switch is implemented based on a logical judgment statement. Each first switch includes a logical judgment statement to determine whether the current image enhancement process is performed on an image in the target mode. The target mode is the mode corresponding to the branch network to which the first switch is connected. For example, for first switch 1 connected to the first branch network, the target mode is the first mode.

[0188] In another example, after determining the first mode, the electronic device sends a first control message to each first switch, with each first switch corresponding to one control message.

[0189] For example, for the first switch 1 connected to the first branch network, the control message is used to control the switching state of the first switch 1. When the target mode is the first mode, the control information corresponding to the first switch 1 is used to control the first switch 1 to turn on, so that the input data of the replaced original branch network is transmitted to the first branch network, and the output data of the first branch network is output to the first data.

[0190] Correspondingly, the control messages corresponding to the other first switches are used to disconnect the first switches between the branch networks corresponding to other modes and the base model, so as to prevent the branch networks corresponding to other modes from participating in the image enhancement process of the first image. Among them, the other modes are the modes other than the first mode among the N modes.

[0191] Since the total number of model parameters included in the branch networks corresponding to each mode is relatively small, storing the branch networks corresponding to N modes in memory simultaneously only requires a small amount of memory. Moreover, this method eliminates the need to load and unload the branch networks corresponding to a mode multiple times in memory, which helps to shorten the time spent acquiring the models corresponding to the modes, thereby improving the efficiency of image enhancement.

[0192] For example, there is no strict execution order between step 820 and step 810. Step 820 may be executed before step 810, or after step 810, or steps 810 and 820 may be executed in parallel. After determining the first mode to which the first image belongs, the electronic device can execute step 820 to obtain the first model.

[0193] After the electronic device acquires the first image and obtains the first model, it performs image enhancement on the first image. The image enhancement process is described below through several embodiments.

[0194] Step 830: The electronic device processes the first image based on the first model to obtain the second image.

[0195] The first model is derived based on the base model and the first branch network. Optionally, the display effect of the first image is related to the display of the second image. For example, for region 1 in the first image and region 2 in the second image, the clarity of region 2 is higher than that of region 1, wherein the position of region 1 in the first image overlaps with the position of region 2 in the second image. The image enhancement process supplements the content of the blurred areas in the first image.

[0196] In one possible implementation, the image enhancement method further includes: the electronic device acquiring image attributes based on the first image; wherein the image attributes include at least one of the following: the shooting focal length of the first image or the type of the first object. In this case, step 830, where the electronic device processes the first image based on the first model to obtain the second image, can be implemented as follows: if the image attributes meet the processing conditions, the first image is processed based on the first model to obtain the second image.

[0197] The shooting focal length refers to the focal length of the image acquisition device's lens during the acquisition of the first image. Optionally, the shooting focal length of the first image is obtained based on detailed information about the first image, or, if the first image is captured by an electronic device, the electronic device can read the shooting focal length of the first image from the first application.

[0198] The first image displays a first object, and the type of the first object is used to characterize the kind of the first object. Optionally, the type of the first object includes: people, animals, plants, etc.

[0199] In this case, the electronic device processes the first image based on the first model to obtain the second image, including: when the image attributes meet the processing conditions, the electronic device processes the first image based on the first model to obtain the second image.

[0200] In other words, after acquiring the first image, the electronic device determines whether to perform image enhancement on the first image based on its image attributes. For example, if image enhancement is required for the first image, the electronic device executes step 830; if image enhancement is not required for the first image, the electronic device does not execute the image enhancement method.

[0201] The processing conditions are used to determine whether image enhancement should be triggered.

[0202] In one example, the image attributes include the focal length of the first image, and the processing condition includes: the focal length of the first image is within a preset focal length. For example, if the focal length of the first image is within the preset focal length, the electronic device executes step 830; if the focal length of the first image is outside the preset focal length, the electronic device does not execute the image enhancement method.

[0203] The preset focal lengths are pre-set. For example, before releasing the first application, developers test the blurriness of images taken at various focal lengths and set the focal lengths with the greatest blurriness as preset focal lengths. Preset focal lengths may include 50mm, 55mm, 60mm, etc.

[0204] The preset focal length range includes multiple preset focal length ranges. The shooting focal length of the first image includes any of the predicted focal lengths, which can be used to determine that the shooting focal length of the first image is within the preset focal length range.

[0205] In this example, the focal length at which the first image was captured reflects its blur level, allowing for a determination of whether image enhancement processing should be applied. For first images with low blur levels, the difference between the enhanced second image and the first image is minimal. In such cases, skipping the image enhancement process helps reduce the total number of images requiring enhancement by the electronic device, thereby reducing its computational load. For applications where the first application is a camera app, this approach ensures the display quality of the captured image without requiring the user to wait excessively for image enhancement, thus improving the user's photography experience.

[0206] In another example, the image attributes include the type of the first object, and the processing condition includes: the preset type does not include the type of the first object. For example, if the preset type does not include the type of the first object, the electronic device performs step 830; if the preset type includes the type of the first object, the electronic device does not perform the image enhancement method.

[0207] The preset type is set in advance, and the preset type includes at least one of the following: sensitive icon, etc.

[0208] Optionally, before performing step 830, the electronic device performs image detection on the first image to obtain the type of the first object. For example, the electronic device is equipped with an image detection model, which is an artificial intelligence model, used to predict the type of the first image; after obtaining the first image, the electronic device inputs the first image into the image detection model to obtain the type of the first object.

[0209] In this example, the type of the first object determines whether to perform image enhancement on the first image. This helps avoid enhancing images displaying sensitive content and reduces unnecessary image enhancement.

[0210] In yet another example, when the image attributes include the shooting focal length of the first object and the type of the first object, the processing conditions include: the shooting focal length of the first image is within a preset focal length, and the preset type includes the type of the first object.

[0211] Step 840: The electronic device outputs the second image.

[0212] In one example, after enhancing the first image, the electronic device outputs a second image. This can be implemented by displaying the second image. For example, if the first application is a camera app or an image processing app, after obtaining the second image, the electronic device displays the second image to demonstrate the effect of image enhancement.

[0213] In another example, after enhancing the first image, the electronic device outputs a second image. This can be implemented by the electronic device outputting an image file containing the second image. If the first application is an image processing application, after obtaining the second image, the electronic device outputs the image file for the user to use in other applications and different scenarios.

[0214] In summary, the base model and the branch networks corresponding to each mode in this application are decoupled and can be stored independently in the storage space. This allows models corresponding to different modes to be obtained based on the same base model.

[0215] Compared to related technologies that store different models separately in the storage space, this solution only needs to store the common base model once in the storage space. Therefore, the storage space only needs to store one base model and multiple branch networks corresponding to each mode. The branch networks have fewer network parameters and occupy less storage space, reducing the total data volume of the image enhancement system. This helps to reduce the storage space occupied and lower the storage pressure on the device.

[0216] Furthermore, the branch networks corresponding to different modes require fewer parameters, and these branch networks can be stored in memory simultaneously. This eliminates the need for loading and unloading the model from memory when the mode of an image changes. This reduces model loading time and improves image enhancement efficiency. Even when multiple branch networks for different modes cannot be stored in memory simultaneously, the smaller parameter requirements of each branch network mean that loading the branch network takes less time than loading the complete mode for a given mode, thus also reducing overall model loading time.

[0217] The following describes the process of obtaining the first model.

[0218] In one possible implementation, step 820 can be implemented as: replacing the original branch network in the base model with the first branch network to obtain the first model.

[0219] In one possible implementation, the base model includes an intermediate network layer; the intermediate network layer is used for feature extraction, the input data of the intermediate network layer is obtained based on the encoding of a first image, and the output data of the intermediate network layer is used for decoding to obtain a second image; wherein, the intermediate network layer includes a first network layer, and the first model is obtained by replacing the first original branch network in the first network layer with the first branch network.

[0220] The input data of the first original branch network has the same dimension as the input data of the first branch network.

[0221] Optionally, the first branch network functions similarly to the original branch network, both used for feature extraction from the image. For example, unlike the original branch network, the first branch network is pre-trained based on images from the first mode. Therefore, the first branch network is better able to learn high-frequency details and low-frequency information in the first image, helping to improve the image enhancement effect.

[0222] Optionally, the first branch network has a simpler structure than the original branch network it replaces. That is, the first branch network has fewer model parameters than the original branch network it replaces.

[0223] For example, the first branch network is set up as a low-rank adaptation of large language (LoRA) based on a large language model.

[0224] In one possible implementation, the first branch network includes multiple linear layers, where for any two adjacent linear layers, the dimension of the output data of one linear layer is the same as the dimension of the input data of the other linear layer.

[0225] Optionally, multiple linear layers are connected in series, with the output data of one linear layer serving as the input data of the next linear layer. For the first linear layer among the multiple linear layers, the dimension of the input data of the first linear layer is the same as the dimension of the input data of the original branch network used by the first branch network for replacement; for the last linear layer among the multiple linear layers, the dimension of the output data of the last linear layer is the same as the dimension of the input data of the original branch network used by the first branch network for replacement.

[0226] On the one hand, the first branch network has fewer model parameters, which helps to save storage space occupied by the first branch network and reduce the storage pressure on the device.

[0227] On the other hand, fewer model parameters mean that the electronic device needs to perform less computation, which helps to reduce the computational complexity of feature extraction in the first branch network, reduce the computational pressure on the electronic device, and shorten the time it takes for the first model to process the first image.

[0228] For electronic devices such as mobile phones and other user terminals, due to the limited computing power of these terminals, replacing the structurally complex original branch network in the base model with a simpler branch network can significantly shorten the image enhancement time. This allows the user terminal to respond quickly to the user's image enhancement needs, preventing device lag and improving the user experience.

[0229] In one possible implementation, the electronic device replaces the original branch network in the base model with a first branch network to obtain a first model, including: the electronic device loading the base model; the electronic device loading the first branch network based on a first mode; and the electronic device replacing the original branch network in the base model with the first branch network to obtain the first model.

[0230] In one example, the electronic device preloads the base model. For instance, since multiple mode-corresponding models share the base model, the electronic device can preload the base model before step 810. This helps to reduce the waiting time for loading the base model.

[0231] In another example, the electronic device could also load the pedestal model after acquiring the first image and confirming that image enhancement is being performed.

[0232] For example, loading a pedestal model is implemented by reading the pedestal model from the hard drive into memory. For instance, an electronic device reads the model structure and parameters of the pedestal model into memory.

[0233] Optionally, the electronic device obtains configuration information corresponding to the first mode based on the first mode. The configuration information corresponding to the first mode is used to indicate the first branch network. For example, the configuration information corresponding to the first mode includes an identifier of the first branch network, and the electronic device loads the first branch network into memory based on the identifier of the first branch network.

[0234] For example, the configuration information is also used to indicate the identifier of the original branch network that the first branch network is used to replace.

[0235] For example, in the case of multiple first branch networks, the configuration information may also indicate the correspondence between the multiple first branch networks and the multiple original branch networks, so that the electronic device can replace the corresponding original branch network with a first branch network based on the correspondence, thereby avoiding errors in the process of replacing the original branch network with a first branch network.

[0236] In one example, the multiple first-branch networks are all different. For instance, the network structures of the multiple first-branch networks are different. Or, the network structures of the multiple first-branch networks are the same, but the model parameters included in the multiple first-branch networks are different. Having multiple first-branch networks with the same network structure helps to compress the storage space required to store multiple first-branch networks, alleviating the storage pressure on electronic devices.

[0237] In one possible implementation, the base model includes an intermediate network layer; the intermediate network layer is used for feature extraction, the input data of the intermediate network layer is obtained based on the encoding of a first image, and the output data of the intermediate network layer is used for decoding to obtain a second image; wherein, the intermediate network layer includes a first network layer, and the first model is obtained by replacing the first original branch network in the first network layer with the first branch network.

[0238] Optionally, the intermediate network layer includes multiple network layers, and the first network layer is any one or more of the multiple network layers included in the intermediate network layer. For example, all of the multiple network layers are the first network layer.

[0239] For example, such as Figure 10 As shown, the first model is a generative model, which includes an encoder, an intermediate network layer (U-Net), and a decoder. The encoder and decoder are used to encode and decode the image in pixel space. The intermediate network layer is used to extract features in implicit space. The first image x is encoded by the encoder to obtain the input data Z of the intermediate network layer, and the input data Z is used for feature extraction by the intermediate network layer. For example, noise can be added during feature extraction (specifically, a diffusion process) to obtain noisy feature information ZT. Subsequently, back-diffusion can be used to predict and remove the noise, obtaining the denoised feature information x(T-1). Here, the noisy feature information ZT is either the output data or the input data of each network layer.

[0240] Specifically, the replaced first network layer includes a first branch network for feature extraction based on an attention mechanism (Q, K, V matrices). Subsequently, the decoder decodes the second image based on the output data of the intermediate network layers.

[0241] For example, the network layers also include a fourth network layer, which remains unchanged in the first model. In this case, only a portion of the intermediate network layers are replaced using the first branch network, reducing the total number of original branch networks that need to be replaced, which helps improve the efficiency of obtaining the first model.

[0242] In one possible implementation, the first network layer includes a first original branch network and a second original branch network; the output data of the first network layer is obtained based on the output data of the second original branch network and the output data of the first branch network; the output data of the first network layer is used as input data for the next network layer in the intermediate network layer, or for decoding to obtain the second image.

[0243] Optionally, the input data of the first original branch network and the second original branch network are the same, and the input data of the first original branch network is the output data of the network layer adjacent to the first network layer.

[0244] Optionally, the first original branch network has the same function as the first branch network, which is used for feature extraction.

[0245] In one example, the first network layer is a residual block in U-Net, the first original branch network is the network structure in the residual block used for feature extraction, and the second original branch network is used to directly use the input data of the residual block as the output data to realize skip connections between the residual blocks and reduce the gradient decay problem.

[0246] In this way, the first model has a good image enhancement capability for images in the first mode, while maintaining the characteristics of the base model in image enhancement, and avoiding significant changes in the performance of the first model due to the addition of the first branch network, which would affect the image enhancement effect.

[0247] Optionally, the intermediate network layer includes multiple first network layers, each of which has its own corresponding first branch network; the electronic device replaces the first original branch network in the multiple first network layers with the multiple branch networks respectively to obtain the first model.

[0248] To further reduce the computational burden on electronic devices for image enhancement, in addition to replacing the more complex original branch network with a first branch network containing linear network layers in the above embodiments, at least one network layer in the pedestal model can also be simplified.

[0249] The following section introduces a simplification method for the network layers in the base model.

[0250] Optionally, this embodiment can be combined with an embodiment that replaces the original branch network in the base model with a first branch network to obtain a first model. For example, after obtaining the first model, the electronic device executes this embodiment to further reduce the network parameters in the first model to obtain a second model.

[0251] In one possible implementation, the electronic device determines a second network layer in the first model based on a first mode, the second network layer not participating in image enhancement of the image in the first mode; the electronic device replaces the second network layer in the first model with a third network layer to obtain a second model.

[0252] The third network layer has the same function as the second network layer, and the total number of model parameters included in the third network layer is less than the total number of model parameters included in the second network layer.

[0253] Optionally, the second network layer can be any one of the multiple network layers included in the intermediate network layers.

[0254] For example, the second network layer can be the replaced first network layer, or any network layer other than the replaced first network layer.

[0255] The replaced first network layer is obtained by replacing the first original branch network with the first branch network. For example, the replaced first network layer includes the second original branch network and the first branch network.

[0256] Optionally, the third network layer includes at least one linear layer. For example, the second network layer differs in models corresponding to different modes, and the third network layer also differs for each mode.

[0257] For example, the third network layer in the first mode is obtained by pre-training on images in the first mode. The pre-training method of the third network layer is similar to that of the first branch network, and specific details are provided in the embodiment corresponding to step 820.

[0258] For example, the total number of linear layers in the third network layer is less than the total number of linear layers in the first branch network. Thus, after replacing the second network layer with the third network layer, the total number of model parameters in the second model is less than the total number of model parameters in the first model, which helps reduce the computational complexity in the image enhancement process.

[0259] Optionally, in this case, step 830, where the electronic device processes the first image based on the first model to obtain the second image, can be implemented as follows: the electronic device processes the first image based on the second model to obtain the second image.

[0260] In other words, the electronic device uses a second model instead of the first model to process the first image. Since the second model includes fewer model parameters than the first model, the computational complexity of image enhancement using the second model is lower, which helps to avoid lag in the electronic device.

[0261] The following describes how to reduce memory usage during the operation of the first model of an electronic device.

[0262] In some embodiments, the total number of model parameters in the intermediate network layers of the base model is large, requiring a significant amount of memory during runtime. To reduce runtime memory consumption, this embodiment supports splitting the intermediate network layers into multiple independent parts, allowing the network layers included in the intermediate network layers to be loaded into memory in batches, thereby reducing memory consumption during model execution.

[0263] For example, the multiple network layers included in the intermediate network layer can be divided into multiple parts, and each part includes at least one network layer; in this embodiment, the first part and the second part, which are adjacent in execution order, are used as examples for illustrative explanation. For example, the first part and the second part are any two parts that are adjacent in execution order among the multiple parts.

[0264] In some possible implementations, the intermediate network layer of the pedestal model includes a first part and a second part; the image enhancement method further includes: the electronic device loading the first part; and during the processing of the first image, after the feature extraction task for the first part is completed, loading the second part.

[0265] Optionally, the first part and the second part are relatively independent portions of the intermediate network layer, and the first part and the second part each include different network layers. For example, the number of network layers included in the first part and the second part may be the same or different.

[0266] For example, the first part and the second part each contain k residual blocks, where k is a positive integer.

[0267] Optionally, there is an execution order between the first part and the second part. During image enhancement, the feature extraction task of the first part is executed before the feature extraction task of the second part; wherein, the feature extraction task is to extract features sequentially from the network layers included in a certain part to obtain feature information of the first image at a deeper level.

[0268] In one example, such as Figure 11As shown, the intermediate network layer is divided into two parts. During image enhancement, the electronic device loads the encoder into the memory pool, runs the encoder, and obtains the input data for the intermediate network layer. Then, the electronic device unloads the encoder and loads the first part into memory; the electronic device runs the first part to process the input data of the intermediate network layer, obtaining the output data of the first part. Then, the electronic device unloads the first part and loads the second part into memory. This process continues until the decoder generates the second image.

[0269] This helps reduce memory usage during model operation and avoids the rapid expansion of the image enhancement system causing electronic devices to lag.

[0270] In some embodiments, where the intermediate network layer is divided into multiple parts and these parts are loaded in batches, the electronic device also replaces the original branch network with the first branch network and replaces the second network layer with the third network in batches. The process of replacing the original branch network and the second network in batches is described below using the first part as an example.

[0271] Optionally, if the first part includes a first network layer 1, then after loading the first network layer 1 into memory, the electronic device obtains the first network layer 1 based on the first mode.

[0272] For example, when the first part includes multiple first network layers, the loading of the first branch network corresponding to the first network layer and the process of replacing the original branch network with the loaded first branch network can be performed in parallel.

[0273] The corresponding first branch network 1; subsequently, the electronic device replaces the first branch network in the first network layer 1 with the first branch network 1.

[0274] For example, when the first part includes a second network layer 1, the electronic device replaces the second network layer 1 with a third network layer 1, the second network layer 1 having the same function as the third network layer 1.

[0275] To avoid the image enhancement masking the image style of the first image, after obtaining the second image, regions in the first image can be adjusted based on the second image to maintain the overall style of the first image while improving its clarity. In one possible implementation, the electronic device fuses the first and second images to generate a third image. In this case, the electronic device outputs the second image, which can be implemented as the electronic device outputting the third image.

[0276] Optionally, for each pixel in the first image, the electronic device performs a weighted average based on the pixel values ​​of pixels at the same location in the first and second images to obtain the pixel value at that location in the third image.

[0277] In one possible implementation, the electronic device fuses a first image and a second image to generate a third image, including: the electronic device acquiring first information from the first image, wherein the first information indicates pixel values ​​in a first region of the first image, and the pixel values ​​in the first region change smoothly; the electronic device acquiring second information from the second image, wherein the second information indicates pixel values ​​in the first region of the second image; and the electronic device adjusting the pixel values ​​in the first region of the first image based on the first information and the second information to obtain the third image.

[0278] Optionally, the first region is a visually blurred area in the first image. The first region has low pixel variation, resulting in a hazy, grayish appearance that makes it difficult to discern textures, edges, and other details within the region.

[0279] For example, the first information includes the pixel values ​​of each pixel in the first region of the first image; the second information includes the pixel values ​​of each pixel in the first region of the second image.

[0280] For example, an electronic device adjusts the pixel values ​​in a first region of a first image based on first information and second information to obtain a third image, including: for any pixel in the first region, the electronic device obtains a first pixel value for that pixel based on the first information; obtains a second pixel value for that pixel based on the second information; and obtains a third pixel value for that pixel by weighted summation of the first and second pixel values. The electronic device then uses the third pixel value as the pixel value for that pixel. For a second region in the first image other than the first region, the pixel values ​​of the pixels in the second region remain unchanged, still being the original pixel values ​​of the pixels in the first image. The electronic device then obtains the third image based on the third pixel values ​​of each pixel in the first region and the original pixel values ​​of each pixel in the second region.

[0281] In this embodiment, keeping the pixels in the second region of the first image unchanged reduces the impact of image enhancement on the image style of the clearer regions in the first image, which helps to compensate for color distortion caused by image enhancement.

[0282] The following is a combination Figure 12 Examples illustrating the combination of image enhancement and image overlay.

[0283] For example, in this example, the first application is a camera application, and image enhancement and image fusion are used to adjust the display effect of the image taken by the user for illustration.

[0284] For example, the base model includes an encoder, an intermediate network model, and a decoder.

[0285] like Figure 12 As shown, this embodiment includes steps 12010-12120, and the execution subject in this embodiment is an electronic device (such as...). Figure 3 Electronic devices 10).

[0286] Step 12010: Acquire the first image. Optionally, the electronic device, in response to the image capture operation, acquires the first image in the first mode.

[0287] For example, the image acquired by the image acquisition device is in RAW format, which is an image file format in which the electrical signal obtained by the image acquisition device is directly stored after analog-to-digital conversion without any further processing. The electronic device performs format conversion on the first image (such as Raw2RGB) to obtain the first image in RGB format.

[0288] For example, the format of the first image is adapted to the first model. If the first model can process RAW format images, then no format conversion is required.

[0289] Step 12020: Determine whether to perform image enhancement. Optionally, the electronic device acquires the image attributes of the first image; if the image attributes meet the processing conditions, the electronic device executes step 12030; if the image attributes do not meet the processing conditions, the electronic device executes step 12120 and outputs the first image. For a description of the processing conditions, refer to the embodiment corresponding to step 830.

[0290] Step 12030: While performing image enhancement, resize the first image.

[0291] Optionally, if the first image is captured at a magnification of 's', the electronic device reduces the size of the first image to the capture size / s, where 's' is a positive number, to improve the clarity of the first image. This helps to enhance the effect of subsequent image enhancement.

[0292] Optionally, step 820 can be implemented through steps 12040-12060.

[0293] Step 12040: Run the encoder.

[0294] Step 12050: Run the intermediate network layer. Optionally, the intermediate network layer can be divided into at least two parts: a first part and a second part. The electronic device can load the first and second parts in batches; please refer to [reference needed]. Figure 11 Corresponding implementation examples.

[0295] Step 12060: Run the decoder. The electronic device runs the decoder to generate a second image. For example, the second image has the same size as the first image input to the coding layer.

[0296] Step 12070, Size Adjustment. Optionally, the electronic device restores the size of the second image to the shooting size.

[0297] Step 12080, Information Extraction. Optionally, the electronic device obtains the first information of the first image and the second information of the second image through wavelet transform, fast Fourier transform, or other means.

[0298] Step 12090, noise processing. Optionally, the electronic device performs image denoising on the second image to remove noise from the second image.

[0299] Step 12100: Determine whether to perform image fusion. Optionally, the electronic device determines the pixel difference based on the first information and the second information. If the pixel difference is greater than or equal to a threshold, the electronic device executes step 12120; if the pixel difference is less than the threshold, the electronic device executes step 12120 and outputs the second image.

[0300] Step 12110, Fusion Processing. For example, the electronic device fuses the first image and the second image to obtain and output a third image.

[0301] Step 12120: Output the image.

[0302] This embodiment improves the quality of captured images while helping to maintain image style, thus enhancing the usability of image enhancement methods in image capturing scenarios.

[0303] The above combination Figures 3-12 The image enhancement method provided in the embodiments of this application is described in detail below. Figure 13 This application provides a detailed description of the electronic device provided in its embodiments. All or part of any feature in any embodiment of this application can be freely combined. The resulting combined technical solutions also fall within the scope of this application.

[0304] In one possible design, Figure 13 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 13 As shown, the electronic device 1300 may include a transceiver unit 1301 and a processing unit 1302. The electronic device 1300 can be used to implement the functions of the electronic device involved in the above method embodiments.

[0305] Optionally, the processing unit 1302 is used to support the electronic device 1300 in performing operations. Figure 8 Steps 810-840 in the text.

[0306] The transceiver unit may include a receiving unit and a transmitting unit, and may be implemented by a transceiver or transceiver-related circuit components, and may be a transceiver or transceiver module. The operation and / or function of each unit in the electronic device 1300 are respectively for implementing the corresponding process of the image enhancement method described in the above method embodiments. All relevant content of each step involved in the above method embodiments can be referred to the functional description of the corresponding functional unit, and for the sake of brevity, it will not be repeated here.

[0307] Optionally, Figure 13 The illustrated electronic device 1300 may also include a storage unit ( Figure 13 (not shown in the image), this storage unit stores a program or instruction. When the processing unit 1302 executes the program or instruction, it causes... Figure 13 The electronic device 1300 shown can perform the image enhancement method described in the above method embodiments.

[0308] Figure 13 The technical effects of the electronic device 1300 shown can be referred to the technical effects of the image enhancement method described in the above method embodiments, and will not be repeated here.

[0309] In addition to being in the form of electronic device 1300, the technical solution provided in this application can also be a functional unit or chip in an electronic device, or a device used in conjunction with an electronic device.

[0310] This application also provides a chip system, including: a processor coupled to a memory, the memory being used to store programs or instructions, wherein when the program or instructions are executed by the processor, the chip system implements the methods in any of the above method embodiments.

[0311] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0312] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0313] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0314] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0315] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, it causes the computer to perform the aforementioned steps to implement the image enhancement method described in the above embodiments.

[0316] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the image enhancement method described above.

[0317] In addition, this application also provides an apparatus. Specifically, the apparatus may be a component or module, and may include one or more processors and a memory connected together. The memory is used to store a computer program. When the computer program is executed by one or more processors, the apparatus performs the image enhancement methods described in the above method embodiments.

[0318] In this application, the computer-readable storage medium, computer program product, or chip provided in the embodiments are all used to execute the corresponding methods described above. Therefore, the beneficial effects they can achieve can be referred to in the beneficial effects of the corresponding methods described above, and will not be repeated here.

[0319] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, optical discs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an application-specific integrated circuit (ASIC).

[0320] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the above functional modules is only used as an example. In practical applications, the above functions can be assigned to different functional modules as needed; that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0321] In the several embodiments provided in this application, it should be understood that the disclosed methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of modules or units may be electrical, mechanical or other forms.

[0322] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. Computer-readable storage media include, but are not limited to, any of the following: USB flash drive, portable hard drive, read-only memory, random access memory, magnetic disk, or optical disk, and other media capable of storing program code.

[0323] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image enhancement method, characterized in that, The method includes: Get the first image in the first mode; The first image is processed based on the first model to obtain the second image; wherein the first model is obtained based on the base model and the first branch network; the base model is a general model for image enhancement in multiple modes, and the first branch network is suitable for feature extraction of the image in the first mode; Output the second image.

2. The method according to claim 1, characterized in that, Before processing the first image based on the first model to obtain the second image, the method further includes: The first model is obtained by replacing the original branch network in the base model with the first branch network.

3. The method according to claim 2, characterized in that, The step of replacing the original branch network in the base model with the first branch network to obtain the first model includes: Load the base model; Based on the first mode, load the first branch network; The first model is obtained by replacing the original branch network in the base model with the first branch network.

4. The method according to any one of claims 1-3, characterized in that, The base model includes an intermediate network layer; the intermediate network layer is used for feature extraction, the input data of the intermediate network layer is encoded based on the first image, and the output data of the intermediate network layer is used for decoding to obtain the second image; The intermediate network layer includes a first network layer, and the first model is obtained by replacing the first original branch network in the first network layer with the first branch network.

5. The method according to claim 4, characterized in that, The first network layer includes a first original branch network and a second original branch network; The output data of the first network layer is obtained by merging the output data of the second original branch network and the output data of the first branch network; the output data of the first network layer is used as the input data of the next network layer in the intermediate network layer, or it is used to decode and obtain the second image.

6. The method according to any one of claims 1-5, characterized in that, The first branch network includes at least two linear network layers. For two adjacent linear layers, the dimension of the output data of one linear layer is equal to the dimension of the input data of the other linear layer.

7. The method according to any one of claims 1-6, characterized in that, The base model is a generative model, and the first branch network is used to replace the original branch network in the residual block of the base model.

8. The method according to any one of claims 1-7, characterized in that, The intermediate network layer of the base model includes a first part and a second part; the intermediate network layer is used to extract feature information of the first image, the input data of the intermediate network layer is encoded based on the first image, and the output data of the intermediate network layer is used to decode to obtain the second image. The method further includes: Load the first part; During the processing of the first image, after the feature extraction task for the first part is completed, the second part is loaded.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Based on the first mode, a second network layer is determined in the first model, and the second network layer does not participate in image enhancement of the image in the first mode; The second network layer in the first model is replaced with the third network layer to obtain the second model; wherein the function of the third network layer is the same as that of the second network layer, and the total number of model parameters included in the third network layer is less than the total number of model parameters included in the second network layer. The process of processing the first image based on the first model to obtain the second image includes: The second image is obtained by processing the first image based on the second model.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: Based on the first image, image attributes are obtained; wherein the image attributes include at least one of the following: the shooting focal length of the first image or the type of the first object, wherein the first image displays the first object; The process of processing the first image based on the first model to obtain the second image includes: If the image attributes meet the processing conditions, the first image is processed based on the first model to obtain the second image.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: A third image is generated by fusing the first and second images. The output of the second image includes: Output the third image.

12. The method according to claim 11, characterized in that, The step of fusing the first image and the second image to generate a third image includes: Obtain first information of the first image; wherein, the first information is used to indicate the pixel value of a first region in the first image, and the pixel value in the first region changes smoothly; Obtain second information from the second image; wherein the second information is used to indicate the pixel value in the first region of the second image; Based on the first information and the second information, the pixel values ​​in the first region of the first image are adjusted to obtain the third image.

13. An electronic device, characterized in that, The electronic device includes: A display screen is used to display the interface. A transceiver is used to send and receive radio signals. Memory is used to store computer program instructions; A processor for executing the computer program instructions to support the electronic device in implementing the method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the method as described in any one of claims 1 to 12.

15. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 12.

Images