Image processing method and related device

Abstract

The present application provides an image processing method and related apparatus, relating to the field of terminal technology. The method includes: in response to a photo-taking operation, acquiring a first image, the first image including N regions; obtaining N second features corresponding to the N first features from a preset feature, the N first features being used to characterize the N regions, each second feature in the N second features being similar to the first feature to which each second feature corresponds, wherein the distortion of the image obtained by decoding the N second features is greater than or equal to the distortion of the image obtained by decoding the N first features; fusing the N first features and the N second features to generate a second image, wherein the fusion weight of any one of the N first features is positively correlated with the distortion of the image obtained by decoding the second feature corresponding to any one of the first features. This helps reduce the probability of image distortion.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and related devices. Background Art

[0002] With the development of terminal technology, electronic devices have become increasingly versatile and can be equipped with one or more cameras to take photos.

[0003] In a photo-taking scenario, an electronic device may capture images through one or more configured cameras.

[0004] However, in the above-mentioned photo-taking scenario, images taken by the electronic device are prone to image distortion. Summary of the Invention

[0005] The embodiments of the present application provide an image processing method and related devices, which are applied in the field of terminal technology and are beneficial to reducing the probability of image distortion.

[0006] In a first aspect, an embodiment of the present application provides an image processing method that can be applied to an electronic device. The method includes: in response to a photo-taking operation, acquiring a first image, the first image including N regions, where N is a positive integer greater than or equal to 1; obtaining N second features corresponding to the N first features from a preset feature, the N first features being used to characterize the N regions, each second feature in the N second features being similar to the first feature to which each second feature corresponds, wherein the clarity of an image obtained by decoding the N second features is greater than the clarity of an image obtained by decoding the N first features, and the distortion of an image obtained by decoding the N second features is greater than or equal to the distortion of an image obtained by decoding the N first features; and fusing the N first features and the N second features to generate a second image, wherein the fusion weight of any one of the N first features is positively correlated with the distortion of the image obtained by decoding the second feature corresponding to the first feature; and the fusion weight of any one of the N second features is negatively correlated with the distortion of the image obtained by decoding the second feature.

[0007] When the distortion of the image obtained by decoding the second feature corresponding to any one of the N first features is large, the fusion weight of any one of the first features is larger, and correspondingly, the fusion weight of the second feature corresponding to any one of the first features is smaller. Since the fidelity of any one of the N first features is greater than the second feature corresponding to any one of the first features, it is beneficial to reduce the probability of distortion of the second image.

[0008] In one possible implementation, before fusing the N first features and the N second features, the method further includes: inputting the N second features into a semantic prediction model to obtain the semantics of each of the N regions and the rejection rate of each region when predicting the semantics, the rejection rate is positively correlated with the distortion of the image obtained by decoding each second feature, and the rejection rate is positively correlated with the fusion weight of the first feature corresponding to each second feature.

[0009] In this way, by identifying semantics and rejection rates through a semantic prediction model, the implicitly expressed second feature can be visualized through semantics, and the fusion weight of the first feature can be influenced by the rejection rate. The generation of the second image can be indirectly guided by the rejection rate, thus realizing semantic-guided implicitization.

[0010] In one possible implementation, the semantic prediction model is trained in the following manner: M third features are obtained, and the M third features are used to characterize M regions in an image sample, where the image sample includes multiple semantics, and M is a positive integer greater than 1; the M third features are input into a source model to obtain the semantics of each region in the M regions and the rejection rate of each region when predicting the semantics, wherein the source model is used to: calculate the prediction probability that the semantics of the i-th region in the M regions is each semantic in the multiple semantics, obtain multiple prediction probabilities, and obtain a target probability that meets preset conditions from the multiple prediction probabilities, and based on the inverse correlation between the target probability, the prediction probability and the rejection rate, calculate the rejection rate of the i-th region in the M regions when predicting the semantics, and output the semantics corresponding to the target probability and the rejection rate of the i-th region in the M regions when predicting the semantics; wherein i is an integer traversing from 1 to M; when the loss function meets the preset model convergence condition, a semantic prediction model is obtained, and the loss function is related to the difference between the predicted probability and the true probability.

[0011] This helps the semantic prediction model to predict semantics and output the rejection rate when predicting semantics.

[0012] In one possible implementation, when the rejection rate of any one of the N regions when predicting semantics is less than or equal to the first value and greater than or equal to the second value, the fusion weight of any one of the N first features is the first weight; when the rejection rate of any one of the N regions when predicting semantics is greater than or equal to the third value and less than the second value, the fusion weight of any one of the N first features is the second weight; wherein the second value is less than the first value, the third value is less than the second value, and the second weight is less than the first weight.

[0013] When the rejection rate is within a certain range, the weight is the same; when the rejection rate is within different ranges, the weight is different. This is conducive to flexibly adjusting the weight according to the rejection rate.

[0014] In one possible implementation, an electronic device includes an encoding module and a decoding module; N first features are extracted from a first image by the encoding module, the encoding module includes L layers of downsampling, a first layer of downsampling in the L layers is used to extract N fourth features representing N regions from the first image, the resolution of an image obtained by decoding the N fourth features being greater than the resolution of an image obtained by decoding the N first features, where L is a positive integer greater than or equal to 1; the N first features and N second features are fused to generate a second image, including: after the decoding module decodes the N second features, fusing the N fourth features to generate the second image. In this way, fusing the outputs of layers with the same resolution facilitates the generation of the second image.

[0015] In a possible implementation, N fourth features are represented by Fe, the output after decoding N second features is represented by Fd, and the second image is represented by Fm. Then, the second image satisfies the following formula:

[0016] Fe*W+Fd=Fm

[0017] Among them, W is used to represent the matrix composed of the fusion weights of any fourth feature among the N fourth features, and the matrix composed of the fusion weights of any first feature among the N first features is obtained by transforming W.

[0018] In this way, when the distortion rate is greater, the value of the weight is greater, the larger the element in W is, the greater the role played by Fe, and correspondingly, the smaller the role played by Fd, which is conducive to reducing the distortion rate of the image.

[0019] In one possible implementation, after the decoding module decodes the N second features and before fusing the N fourth features, the method also includes: performing feature parallel connection on the output after decoding the N second features and the N fourth features; inputting the output result after the feature parallel connection into the residual dense block RRDB to obtain the output of RRDB; convolving the output of RRDB to obtain the output result; after the decoding module decodes the N second features, fusing the N fourth features, including: after the decoding module decodes the N second features, fusing the output result.

[0020] In this way, the output of the encoder can be successfully applied to the input of the decoder, reducing the probability of distortion.

[0021] In one possible implementation, obtaining N second features corresponding one-to-one to the N first features from the preset features includes: finding a feature having the smallest Euclidean distance to each of the N first features from the preset features to obtain the N second features.

[0022] This is helpful for finding features that can be used to decode a higher-definition image, thereby improving the clarity of the second image.

[0023] In one possible implementation, in response to a photographing operation, obtaining a first image includes: obtaining multiple frames of original images in response to the photographing operation; calculating a registration matrix and / or AE parameters based on the multiple frames of original images, where the registration matrix is ​​a matrix representing a geometric transformation relationship between the multiple frames of original images, and the AE parameters include at least one of exposure time, brightness, or sensitivity of the multiple frames of original images; and fusing the multiple frames of original images based on the registration matrix and / or AE parameters to obtain the first image.

[0024] In this way, fusing multiple frames of images is beneficial to reducing noise in the first image, reducing the phenomenon of motion blur, and further beneficial to improving the quality of the second image.

[0025] In one possible implementation, after fusing N first features and N second features to generate a second image, the method further includes: processing the brightness of the second image through gamma correction, global tone mapping GTM, sharpening, or at least one of a first curve, wherein the first curve is used to represent the correspondence between brightness and contrast.

[0026] This helps to adjust the brightness of the second image to better meet the needs.

[0027] In a second aspect, an embodiment of the present application provides an image processing device, which may be an electronic device or a chip or chip system within an electronic device. The image processing device may include an acquisition unit and a processing unit. When the image processing device is an electronic device, the processing unit may be a processor. The image processing device may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing device is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (for example, a register, a cache, etc.), or a storage unit within the electronic device that is located outside the chip (for example, a read-only memory, a random access memory, etc.).

[0028] Exemplarily, an acquisition unit is used to acquire a first image in response to a photographing operation, where the first image includes N areas, where N is a positive integer greater than or equal to 1; a processing unit is used to obtain N second features corresponding one-to-one to the N first features from preset features, where the N first features are used to characterize the N areas, and each second feature of the N second features is similar to the first feature corresponding to each second feature, wherein the clarity of the image obtained by decoding the N second features is greater than the clarity of the image obtained by decoding the N first features, and the distortion of the image obtained by decoding the N second features is greater than or equal to the distortion of the image obtained by decoding the N first features; the N first features and the N second features are fused to generate a second image, and the fusion weight of any one of the N first features is positively correlated with the distortion of the image obtained by decoding the second feature corresponding to any one of the first features; and the fusion weight of any one of the N second features is negatively correlated with the distortion of the image obtained by decoding any one of the second features.

[0029] In one possible implementation, the processing unit is further used to: input N second features into a semantic prediction model to obtain the semantics of each of the N regions and the rejection rate of each region when predicting the semantics, the rejection rate is positively correlated with the distortion of the image obtained by decoding each second feature, and the rejection rate is positively correlated with the fusion weight of the first feature corresponding to each second feature.

[0030] In one possible implementation, the acquisition unit is further used to: acquire M third features, the M third features are used to characterize M regions in an image sample, the image sample includes multiple semantics, and M is a positive integer greater than 1; the processing unit is further used to: input the M third features into a source model to obtain the semantics of each region in the M regions and the rejection rate of each region when predicting the semantics, wherein the source model is used to: calculate the predicted probability that the semantics of the i-th region in the M regions is each semantic in the multiple semantics, obtain multiple predicted probabilities, and obtain a target probability that meets preset conditions from the multiple predicted probabilities, and based on the inverse correlation between the target probability, the predicted probability and the rejection rate, calculate the rejection rate of the i-th region in the M regions when predicting the semantics, and output the semantics corresponding to the target probability and the rejection rate of the i-th region in the M regions when predicting the semantics; wherein i is an integer traversing from 1 to M; when the loss function meets the preset model convergence condition, a semantic prediction model is obtained, and the loss function is related to the difference between the predicted probability and the true probability.

[0031] In one possible implementation, when the rejection rate of any one of the N regions when predicting semantics is less than or equal to the first value and greater than or equal to the second value, the fusion weight of any one of the N first features is the first weight; when the rejection rate of any one of the N regions when predicting semantics is greater than or equal to the third value and less than the second value, the fusion weight of any one of the N first features is the second weight; wherein the second value is less than the first value, the third value is less than the second value, and the second weight is less than the first weight.

[0032] In one possible implementation, the image processing device also includes an encoding module and a decoding module; the N first features are extracted from the first image by the encoding module, the encoding module includes L layers of downsampling, the first layer of downsampling in the L layers of downsampling is used to extract N fourth features for representing N areas from the first image, the resolution of the image obtained by decoding the N fourth features is greater than the resolution of the image obtained by decoding the N first features, and L is a positive integer greater than or equal to 1; the decoding module is used to: after decoding the N second features, fuse the N fourth features to generate a second image.

[0033] In a possible implementation, N fourth features are represented by Fe, the output after decoding N second features is represented by Fd, and the second image is represented by Fm. Then, the second image satisfies the following formula:

[0034] Fe*W+Fd=Fm

[0035] Among them, W is used to represent the matrix composed of the fusion weights of any fourth feature among the N fourth features, and the matrix composed of the fusion weights of any first feature among the N first features is obtained by transforming W.

[0036] In one possible implementation, the processing unit is further used to: perform feature parallel connection on the output after decoding the N second features and the N fourth features; input the output result after feature parallel connection into the residual dense block RRDB to obtain the output of RRDB; convolve the output of RRDB to obtain the output result; the decoding module is also used to: after decoding the N second features, fuse the output results.

[0037] In a possible implementation, the processing module is further configured to: find, among the preset features, a feature having the smallest Euclidean distance to each of the N first features, to obtain N second features.

[0038] In one possible implementation, the acquisition unit is further used to: acquire multiple frames of original images in response to a photo-taking operation; the processing unit is further used to: calculate a registration matrix and / or AE parameters based on the multiple frames of original images, where the registration matrix is ​​a matrix representing the geometric transformation relationship between the multiple frames of original images, and the AE parameters include at least one of the exposure time, brightness, or sensitivity of the multiple frames of original images; and fuse the multiple frames of original images based on the registration matrix and / or AE parameters to obtain a first image.

[0039] In one possible implementation, the processing unit is further used to: process the brightness of the second image through at least one of gamma correction, global tone mapping GTM, sharpening, or a first curve, wherein the first curve is used to represent the correspondence between brightness and contrast.

[0040] In a third aspect, an embodiment of the present application provides an electronic device comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, and the one or more processors calling the computer instructions to enable the electronic device to execute the method described in the first aspect or any possible implementation of the first aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0043] In the sixth aspect, the present application provides a chip or chip system, which is applied to an electronic device, and the chip or chip system includes at least one or more processors, and the one or more processors are used to call computer instructions to execute the method described in the first aspect or any possible implementation of the first aspect.

[0044] In one possible implementation, the chip or chip system described above in this application further includes at least one memory, in which instructions are stored. The memory may be a storage unit within the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0045] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a semantic guidance provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a VQVAE algorithm flow provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a technical concept provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of a generative network provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of a network structure in a decoder provided in an embodiment of the present application;

[0054] Figure 9 A schematic diagram of weight calculation provided in an embodiment of the present application;

[0055] Figure 10 A schematic diagram of a network structure in a decoder provided in an embodiment of the present application;

[0056] Figure 11 A schematic flowchart of another image processing method provided in an embodiment of the present application;

[0057] Figure 12 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0059] 1. Semantic guidance

[0060] Semantic guidance can guide an algorithm to generate a specific object by providing semantic information. In some examples, the semantic information can be a semantic segmentation map, the algorithm can be a neural network algorithm, and the specific object can be a scene.

[0061] For example, Figure 1 A schematic diagram of semantic guidance is shown. Figure 1 As shown in the figure, the semantic segmentation map can be used as a guiding framework for generating images, guiding the generation model to generate multiple images with similar semantic structures by using the pixels in the semantic segmentation image and the semantics corresponding to the pixels.

[0062] Semantic guidance can be implemented in two ways: implicit semantic guidance and explicit semantic guidance.

[0063] Implicit semantic guidance means that the generative model does not need to explicitly know the semantics of each pixel. Instead, it uses the semantics of each pixel as an implicit guiding signal to indirectly control the semantic structure and details of the generated image. This implicit approach allows the generative model to learn the semantic structure of images without explicit semantic labels, thereby generating multiple images with similar semantic information.

[0064] Explicit semantic guidance means that the generative model needs to know the semantics of each pixel explicitly, and uses the semantics of each pixel to control the semantic structure and details of the generated image. This explicit approach allows the generative model to learn the semantic structure of images in the presence of explicit semantic labels, thereby generating multiple images with similar semantic information.

[0065] 2. Vector quantized variational auto encoder (VQVAE) algorithm

[0066] The VQVAE algorithm is a method that combines vector quantization (VQ) and variational auto encoder (VAE).

[0067] VAE is a generative model that can include an encoder, a latent space, and a decoder. The encoder can encode the input data of the generative model into the latent space, where the data in the latent space satisfies the prior distribution, and the decoder can reconstruct the data in the latent space.

[0068] VQ is used to discretize the data in the latent space of VAE.

[0069] For example, Figure 2 A schematic diagram of the flow of the VQVAE algorithm is shown. Figure 2 As shown, VQ discretizes the data in the latent space of VAE through the embedding space. The embedding space includes an embedding table, which can include k vectors, each of which has a dimension of D. The k vectors can be represented as e1, e2, ..., e k .

[0070] The processing of the VQVAE algorithm can include:

[0071] 1) After passing an image through the encoder, a H×W×D feature map can be obtained. The feature map can be represented by z e (x) indicates.

[0072] 2) These H×W D-dimensional vectors can be found in the embedding table respectively. The closest (for example, the closest Euclidean distance) e i and use e i The index (index) represents that we can get q(z|x). Among them, the numbers in q(z|x) are discrete, and each number can be regarded as a probability distribution. There are a total of K dimensions, and each dimension represents the corresponding embedding table e i The probability of (i=1, 2, ..., k).

[0073] 3) Feature map z e (x) can be obtained by embedding the closest e in the table i After substitution, we get z q (x). Among them, from z e (x) to z q The change of (x) can be regarded as a cluster, that is, the feature obtained by the encoder is represented by a vector closest to it in the embedding table.

[0074] Among them, 2) and 3) can be called discrete feature matching.

[0075] 4)z q (x) is the input of the decoder, z q (x) After the decoder, the processed image p(x|z q ).

[0076] It can be understood that the VQVAE algorithm includes three processing steps: encoding, discrete feature matching, and decoding.

[0077] 3. Other terms

[0078] In the embodiments of this application, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same function or effect. For example, the first image and the second image are used solely to distinguish between the different images and do not define their order. Those skilled in the art will understand that terms such as "first" and "second" do not define the quantity or order of execution, and that terms such as "first" and "second" do not necessarily define differences.

[0079] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0080] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0081] 4. Electronic devices

[0082] The electronic devices of the embodiments of the present application may include handheld devices, vehicle-mounted devices, etc. with processing functions. For example, some electronic devices include: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 5G networks or future evolved public land mobile communication networks (PLMNs), and so on. The terminal equipment in the network (PLMN), etc., is not limited to this in the embodiments of the present application.

[0083] In addition, in the embodiments of the present application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0084] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0085] In a photo-taking scenario, an electronic device may capture images through one or more configured cameras. However, in the above-mentioned photo-taking scenario, the images captured by the electronic device are prone to image distortion.

[0086] In some implementations, the electronic device may be configured with a complementary metal oxide semiconductor (CMOS) camera. In a photo-taking scenario, the electronic device may capture a raw image (Raw Image) through the CMOS camera in response to a photo-taking operation. The data format of the raw image may adopt a Bayer arrangement. The electronic device may be deployed with an image signal processing (ISP) algorithm, which may be used to denoise, enhance, and optimize the raw image captured by the CMOS camera to provide a clear and accurate image to the user.

[0087] However, ISP algorithms are limited by various technical bottlenecks in processing algorithms, especially for solving irreversible problems such as super-resolution. It is difficult to restore high-definition images at high magnification using traditional upsampling algorithms or deterministic learning methods. This results in poor clarity of images captured by electronic devices.

[0088] In some examples, to improve the clarity of images captured by an electronic device, the electronic device may be equipped with a generative model. In a photo-taking scenario, the generative model can be used to process the raw images captured by a CMOS camera to produce a high-definition image. For example, the generative model may be a VQVAE.

[0089] However, the generative model is limited by the technical bottleneck of its own algorithm, which can lead to image distortion.

[0090] For example, in the generative model VQVAE, the feature map z e (x) can be obtained by embedding the closest e in the table i After substitution, we get z q (x), for z q (x) is decoded, so that the features used for decoding are not the features output by the encoder, but the features embedded in the table i , e embedded in the table i The clarity of the image obtained after decoding is greater than the clarity of the image obtained after decoding the features output by the encoder, but the e embedded in the table iThe fidelity of the decoded image is lower than that of the image obtained by decoding the features output by the encoder. Therefore, although the image processed by the generative model has improved clarity, it will also be distorted. It is understandable that fidelity and distortion are inversely correlated: the lower the fidelity, the greater the distortion, and conversely, the higher the fidelity, the lower the distortion.

[0091] In view of this, an embodiment of the present application provides an image processing method and related devices. When the fidelity of the vectors in the embedding table is poor or the distortion is high, in the process of obtaining the desired image by using the features output by the encoder and the features after fusing the vectors in the embedding table, the fusion weight of the features output by the encoder is larger. In this way, compared with generating images using the vectors in the embedding table, it is beneficial to reduce the probability of image distortion.

[0092] In order to better understand the method provided in the embodiment of the present application, Figure 3 Introduce the technical concept of the embodiments of this application.

[0093] Figure 3 A schematic diagram of a technical concept is shown. Figure 3 As shown in Figure 2, the purpose of semantic guidance is to guide generation. The generation model includes the process of vector discretization, in which the vector discretization is implicitly expressed, that is, the embedding table e is unknown. i (i=1, 2, ..., k) represents the meaning.

[0094] The embodiment of the present application combines semantic guidance implicitization with implicit expression concretization, realizes implicit expression concretization through the meaning of the vectors in the semantic expression embedding table, and then realizes the discretization representation of the spatial region, and calculates the rejection rate when predicting semantics. The rejection rate is positively correlated with the distortion. The distortion of the vectors in the embedding table is reflected by the rejection rate. The fusion weight of the features output by the encoder is calculated through the rejection rate and the correlation between the rejection rate and the weight. The generation of the image is guided by the weight to realize semantic guidance implicitization (that is, the generation of the image is indirectly guided by semantics).

[0095] When the rejection rate is large and the distortion is large, the fusion weight of the features output by the encoder is large. When the rejection rate is small and the distortion is small, the fusion weight of the features output by the encoder is small. This is beneficial to reduce the probability of image distortion.

[0096] The method provided in the embodiment of the present application can be executed by an electronic device. To facilitate understanding, the hardware structure of the electronic device provided in the embodiment of the present application is first introduced.

[0097] For example, Figure 4 FIG1 shows a hardware structure diagram of an electronic device provided by an embodiment of the present application. Figure 4As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, and a display screen 194, etc.

[0098] Optionally, the above-mentioned sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0099] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0100] In an embodiment of the present application, the processor 110 may be equipped with the image processing method provided in an embodiment of the present application. In a photo shooting scenario, the processor 110 may obtain an image captured by the camera 193, process the captured image using the method provided in an embodiment of the present application, and then output the processed image. The image processed by the processor 110 may be displayed on the display screen 194.

[0101] The camera 193 may include one or more cameras, which is not limited in the present embodiment. In some examples, the camera 193 may include a CMOS camera and / or a charge coupled device (CCD) camera.

[0102] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a microservice architecture, or a cloud architecture. The layered architecture can adopt the Android system, the Apple (IOS) system, or other operating systems, and the embodiments of the present application are not limited to this. The following uses the Android system with a layered architecture as an example to illustrate the software architecture of the electronic device provided in the embodiments of the present application.

[0103] Figure 5A schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application is shown. Figure 5 As shown, the layered architecture divides the software architecture of an electronic device into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers: from top to bottom, the application layer (applications), the application framework layer (application framework), the hardware abstraction layer (HAL), the kernel layer (kernel), and the hardware layer.

[0104] The application layer can include a series of application packages, and the application layer runs applications by calling the application programming interface (API) provided by the application framework layer. Figure 5 As shown, the application package can include applications such as camera and gallery.

[0105] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 5 As shown, the application framework layer may include a camera access interface and a view system. The camera access interface may be used to provide an application programming interface and a programming framework for camera applications.

[0106] like Figure 5 As shown, the HAL layer may include a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer may provide virtual hardware for the camera device. The camera algorithm library may include the operating code and data for implementing the image processing method provided in the embodiments of the present application. In other implementations, the camera algorithm library may be deployed in the camera hardware abstraction layer, which is not limited in the embodiments of the present application.

[0107] The kernel layer is the layer between hardware and software. Figure 5 As shown, the kernel layer may include a camera device driver.

[0108] The hardware layer may include hardware such as cameras and image signal processors.

[0109] It should be understood that in some embodiments, layers that implement the same function may be referred to by other names, or a layer that can implement the functions of multiple layers may be considered as one layer, or a layer that can implement the functions of multiple layers may be divided into multiple layers. This embodiment of the present application does not limit this.

[0110] The following combines the above Figure 5The software structure shown is used to specifically describe the image processing method in the embodiment of the present application:

[0111] In response to a user opening the camera application, such as by clicking the camera application icon, the camera application invokes the camera access interface of the application framework layer to start the camera application. The camera hardware abstraction layer then sends a command to start the camera. The camera hardware abstraction layer sends this command to the camera device driver in the kernel layer. The camera device driver then activates the corresponding camera to capture images.

[0112] The camera can transmit the collected image light signal to the image signal processor for preprocessing to obtain the image electrical signal, that is, the original image, and transmit the original image to the camera hardware abstraction layer through the camera device driver.

[0113] The camera hardware abstraction layer can send the original image to the camera algorithm library. The camera algorithm library stores the degree code for implementing the image processing method provided by the embodiment of the application. Executing this code can reduce the probability of image distortion while improving image clarity.

[0114] The camera algorithm library can send the processed image to the camera hardware abstraction layer, and the camera hardware abstraction layer will display the processed image.

[0115] Combined with the above Figure 4 and Figure 5 , introduces the software and hardware structure of the electronic device in the embodiment of the present application, and then introduces the method applied to the electronic device.

[0116] In order to better understand the method provided in the embodiment of the present application, the deployment location of the core algorithm provided in the embodiment of the present application is first introduced.

[0117] For example, Figure 6 FIG. 1 is a schematic diagram showing an image processing method provided by an embodiment of the present application. Figure 6 As shown, in response to a camera operation, the electronic device enters a camera mode and can obtain multiple frames of original images captured by the camera and calculate a registration matrix and automatic exposure (AE) parameters based on these multiple frames of original images. The registration matrix, which can also be called a transformation matrix, is a matrix used to describe the geometric transformation relationship between multiple frames of original images. The AE parameters may include parameters such as brightness, aperture size, exposure time, and sensitivity of each frame of the original image.

[0118] The electronic device can perform denoising on each of the multiple original image frames and convert the format of each frame. For example, the electronic device can convert the format of each frame to RGB format using a demosaic algorithm. The electronic device can also perform high dynamic range fusion (HDR fusion) on the converted images based on the registration matrix and AE parameters to obtain a fused image.

[0119] The electronic device can use the generative network provided in the embodiment of the present application to process the fused image to obtain a processed image. The clarity of the processed image is greater than the clarity of the image before processing. The generative network provided in the embodiment of the present application can combine the semantic guidance implicitization with the implicit expression concreteness, embed the meaning of the vector in the table through semantic expression, and calculate the rejection rate when predicting semantics. In the case of a large rejection rate, in the process of obtaining the desired image by using the features output by the encoder and the features fused by the vectors in the embedding table, the fusion weight of the features output by the encoder is large. In the case of a small rejection rate, in the process of obtaining the desired image by using the features output by the encoder and the features fused by the vectors in the embedding table, the fusion weight of the features output by the encoder is small.

[0120] Electronic devices can perform gamma correction and global tone mapping (GTM) on processed images. Both GTM and gamma correction can be used to adjust the brightness and contrast of images. GTM focuses more on global color and brightness optimization, while gamma correction focuses more on correcting the nonlinear response of the device.

[0121] Electronic devices can also sharpen processed images and adjust image brightness using a brightness curve. Sharpening adjusts image clarity and detail. A brightness curve represents the relationship between brightness and contrast, allowing for image brightness adjustment.

[0122] The electronic device may also perform a color look-up table (LUT) on the processed image, wherein the color LUT is used to adjust the color and brightness of the image.

[0123] Electronic devices can compress images processed by color LUT. Figure 6 As shown, the electronic device can perform JPEG compression on the image after applying the color LUT processing.

[0124] The electronic device may output the compressed image, which may be displayed in a gallery application or in a preview of a camera application.

[0125] The structure and function of the generative network are introduced below.

[0126] For example, Figure 7 Figure 2 shows a schematic diagram of the structure of a generative network. Figure 7 As shown in Figure 1, the generative network includes an encoder, discrete feature matching, a decoder, and a discrete vector semantic prediction network. The encoder is used to extract features from the input image. The embedding space includes multiple sets of discrete implicit feature bases (or multiple vectors). Discrete feature matching is used to find the discrete implicit feature base with the smallest Euclidean distance from each feature extracted by the encoder in the embedding table. The index corresponding to the discrete implicit feature base is then assigned to each feature, and the features extracted by the encoder are replaced with the discrete implicit feature base.

[0127] The specific implementation can refer to the following formula:

[0128]

[0129] z q (x) = e i , k = argmin i ‖z e (x)-e i ‖2

[0130] Among them, z e (x) is the feature extracted by the encoder, e i is a discrete implicit eigenvalue, and the argmin function is used to calculate the minimum value. There are k sets of discrete implicit eigenvalues ​​in the embedding table. The k sets of discrete implicit eigenvalues ​​are traversed. If the discrete implicit eigenvalues ​​e in the k sets of discrete implicit eigenvalues ​​are i With z e (x) is closest, then the probability q(z=k|x) of the discrete implicit feature basis is 1. If the discrete implicit feature basis e in the k sets of discrete implicit feature basis i Not with z e (x) is closest, then the probability q(z=k|x) of the discrete implicit feature basis is 0. Use the closest discrete implicit feature basis to replace the features extracted by the encoder, and get z q (x).

[0131] like Figure 7 As shown, 1, 2, 3, 4, ..., P are P groups of discrete implicit feature bases. After discrete feature matching, the P groups of discrete implicit feature bases are input to the decoder.

[0132] In the embodiment of the present application, the encoder and the discrete feature matching are the same as above. Figure 2 The process shown is the same.

[0133] The discrete vector semantic prediction network can generate a set of two-channel two-dimensional spaces based on a set of discrete implicit feature bases. If the above P sets of discrete implicit feature bases are input into the discrete vector semantic prediction network, P sets of two-channel two-dimensional spaces can be obtained. This application uses a set of two-channel two-dimensional spaces as an example for illustration.

[0134] In the dual-channel 2D space, one channel represents the pixel-wise semantics of the semantic segmentation task, while the other channel represents the rejection rate. The rejection rate reflects the confidence of the semantic prediction. A higher rejection rate indicates a lower confidence level, indicating a lower probability that the image fidelity obtained by decoding the discrete implicit feature base meets the requirements. A lower rejection rate indicates a higher confidence level, indicating a higher probability that the image fidelity obtained by decoding the discrete implicit feature base meets the requirements.

[0135] Pixel-level semantics means that each pixel in the image has a corresponding semantic meaning. Rejection rates correspond to pixels, so each pixel has a corresponding rejection rate.

[0136] The embodiment of the present application presets a corresponding relationship between rejection rate and weight. The greater the rejection rate, the greater the weight value. Conversely, the smaller the rejection rate, the smaller the weight value.

[0137] In some examples, the rejection rate may have a value between 0 and 1, and the weight may have a value between 0 and 1. The correspondence between the rejection rate and the weight may include:

[0138] If the rejection rate is greater than or equal to T1 and less than or equal to 1, the weight is a1. If the rejection rate is greater than or equal to T2 and less than T1, the weight is a2. If the rejection rate is greater than or equal to 0 and less than T2, the weight is a3. Here, T1 is greater than T2, a1 is greater than a2, and a2 is greater than a3.

[0139] The input of the discrete vector semantic prediction network is P groups of discrete implicit feature bases. Each group of discrete implicit feature bases corresponds to pixel-level semantics and rejection rates. The output of the discrete vector semantic prediction network is P groups of pixel-level semantics and P groups of rejection rates. A group of discrete implicit feature bases can correspond to one or more pixels. If a group of discrete implicit feature bases includes one pixel, the pixel-level semantics obtained based on the group of discrete feature bases includes the semantics of the pixel and a rejection rate. If a group of discrete implicit feature bases includes multiple pixels, the pixel-level semantics obtained based on the group of discrete feature bases includes the semantics of the multiple pixels and multiple rejection rates.

[0140] Based on the P group rejection rate and the correspondence between the rejection rate and the weight, the P group weight can be obtained. The decoder can fuse the output of the encoder based on the P group weight in the process of decoding the output of discrete feature matching. The P group weight can be represented in the form of a matrix or array, which is not limited in the embodiment of the present application.

[0141] The number of weights is the same as the number of pixels in the decoder's input image. In other words, multiple weights can be represented by a weight matrix whose dimension is the same as the resolution of the decoder's input image.

[0142] First, the network structure of the decoder in the embodiment of the present application is introduced. Figure 8 A schematic diagram of a network structure in a decoder is shown. Figure 8 As shown in , the decoder network structure can include multiple upsampling processes. Figure 8 In the example shown, the decoder includes four upsampling processes. The decoder processing may include:

[0143] 1) Upsample the input information four times in sequence to obtain Fd.

[0144] 2) Obtain the output of the first layer network in the encoder. In this embodiment of the present application, Fe is used to represent the output of the first layer network in the encoder. The first layer network of the encoder may be a downsampling layer. The number of layers included in the encoder may be the same as the number of upsampling layers included in the decoder.

[0145] 3) After fusing Fe and Fd, we get Fa: Fa*W+Fd=Fm, where W is the matrix and Fm is the processed image. The elements in W represent the fusion weights of each pixel in Fe.

[0146] The fusion processing of Fe and Fd may include: performing feature concatenation on Fd and Fe, and inputting the result of the feature concatenation into a fusion network to obtain Fa. The fusion network may include multiple possible implementations.

[0147] In one possible implementation, the fusion network may include two RRDB (residual in residual dense block) networks and a convolutional network.

[0148] In this case, the fusion processing of Fe and Fd may include: performing feature parallel connection on Fd and Fe, inputting the processing results after the feature parallel connection into two RRDB (residual in residual dense block) networks, and convolving the outputs of the two RRDB networks to obtain the output result Fa.

[0149] In another possible implementation, the fusion network may include one RRDB network and a convolutional network.

[0150] In this case, the fusion processing of Fe and Fd may include: performing feature parallel connection on Fd and Fe, inputting the processing result after the feature parallel connection into the RRDB network, and convolving the output of the RRDB network to obtain the output result Fa.

[0151] In this way, fusing Fe and Fd allows the encoder's output to successfully affect the decoder's input, reducing the probability of distortion. Furthermore, as the rejection rate increases, the weight increases, the element values ​​in W increase, and Fe becomes more influential. Correspondingly, Fd becomes less influential, which helps reduce image distortion.

[0152] The dimension of W is the same as the resolution of the image obtained by the last upsampling of the decoder. If the resolution of the image obtained by the last upsampling of the decoder is the same as the resolution of the decoder input image, then the elements of W are the weights obtained by the rejection rate. If the resolution of the image obtained by the last upsampling of the decoder is smaller than the resolution of the decoder input image, then the elements of W can be obtained by transforming the weights obtained by the rejection rate.

[0153] For example, the dimension of the weight matrix formed by the weights obtained by the rejection rate is H*W. The resolution of the image obtained by the last upsampling of the decoder is H / 2*W / 2, so the dimension of W should be H / 2*W / 2. The dimension of the weight matrix is ​​4 times the dimension of W, so each element in W can be obtained by averaging every 4 elements in the weight matrix.

[0154] For example, Figure 9 A schematic diagram of weight calculation is shown. Figure 9 As shown, the dimension of W is H / 2*W / 2, and the dimension of the weight matrix is ​​H*W. Then the average value of each of the four elements above, below, left and right in these H*W elements is one element in W, which can be expressed by the formula:

[0155] (a m,n +a m,n+1 +a m+1,n +a m+1,n+1 ) / 4=A m,n

[0156] Among them, A m,n is the element in row m and column n in W, a m,n is the element in the mth row and nth column of the weight matrix, a m,n+1 is the element in the mth row and (n+1)th column in the weight matrix, a m+1,n is the element in the (m+1)th row and nth column of the weight matrix, am+1,n+1 is the element in the (m+1)th row and (n+1)th column in the weight matrix.

[0157] exist Figure 9 In the example shown, the average values ​​of the four upper, lower, left and right elements in the weight matrix may include 0.9, 0.5 and 0.1, and the elements in W may include 0.9, 0.5 and 0.1, etc.

[0158] above Figure 8 In the processing process of the decoder shown, after the last upsampling, the output of the first layer network of the encoder is fused. In the embodiment of the present application, the output of the network corresponding to the encoder can also be fused after each layer of upsampling, or the output of the network corresponding to the encoder can be fused after partial upsampling, and the output of the network corresponding to the encoder cannot be fused after partial upsampling. The embodiment of the present application does not limit this. Among them, the last upsampling of the decoder corresponds to the first layer network of the encoder, and the penultimate upsampling of the decoder corresponds to the second layer network of the encoder. Other similarities are not listed one by one in the embodiment of the present application. In the processing process of the decoder, the more levels of the output of the encoder are fused, the more conducive it is to reducing the probability of image distortion, and the fewer levels of the output of the encoder are fused, the less computing power is required.

[0159] For example, Figure 10 A schematic diagram of a network structure in a decoder is shown. Figure 10 As shown in , the decoder network structure can include multiple upsampling processes. Figure 10 In the example shown, the decoder includes four upsampling processes. The decoder processing may include:

[0160] 1) Upsample the input information once to obtain Fd1.

[0161] 2) Obtain the output of the last network layer in the encoder. In the embodiment of the present application, Fe1 is used to represent the output of the last network layer in the encoder.

[0162] 3) After Fd1 and Fe1 are fused, Fa1 is obtained, Fa1*W1+Fd1=Fm1. The fusion process can be as described above. Figure 8 As shown, no further details are given here.

[0163] 4) Upsample Fm1 three times in sequence to obtain Fd2.

[0164] 5) Obtain the output of the first layer of the network in the encoder. In the embodiment of the present application, Fe2 is used to represent the output of the last layer of the network in the encoder.

[0165] 6) After fusion processing of Fd2 and Fe2, Fa2 is obtained, Fa2*W2+Fd2=Fm2. Fm2 is the processed image. The fusion processing method can be as described above. Figure 8 As shown, no further details are given here.

[0166] W1 and W2 are both matrices with different dimensions. The dimension of W1 is the same as the resolution of the image obtained by the first upsampling of the decoder, and the dimension of W2 is the same as the resolution of the image obtained by the last upsampling of the decoder.

[0167] If the resolution of the image obtained by the last upsampling of the decoder is the same as the resolution of the decoder input image, then W2 is the above weight matrix. If the resolution of the image obtained by the last upsampling of the decoder is smaller than the resolution of the decoder input image, then W2 can be obtained by transforming the above weight matrix. For details, please refer to the above Figure 9 W1 can be obtained from W2 or from the above weight matrix transformation, which will not be described here.

[0168] Through the above Figures 7 to 10 From the description, we can see that the image processing method provided by the embodiment of the present application reflects the distortion of the image obtained by decoding the discrete implicit feature basis through the rejection rate during semantic prediction. The higher the rejection rate, the higher the distortion, the higher the fusion weight of the features output by the encoder, and correspondingly, the smaller the fusion weight of the discrete implicit feature basis. This helps reduce the probability of image distortion.

[0169] In the embodiment of the present application, semantic prediction is performed by a discrete vector semantic prediction network. The training process of the discrete vector semantic prediction network may include the following steps:

[0170] 1) Obtain an image sample, which may include multiple pixels. The image sample may correspond to multiple semantics. In some examples, the semantics in the image sample may include human hair, human eyes, human nose, human eyebrows, human mouth, human arms, human legs, cat eyes, cat nose, cat mouth, dog eyes, dog nose, dog mouth, etc.

[0171] 2) Input the image sample into the encoder to obtain a feature map, which includes multiple features 1, and find the feature closest to each feature in the multiple features 1 in the embedding table to obtain multiple features 2.

[0172] 3) Inputting the plurality of features 2 into a source model to obtain an output of the source model. The output of the source model may include the semantics of each pixel in the plurality of pixels and a rejection rate when predicting the semantics. The source model may be a neural network model, for example, a Unet network.

[0173] The semantics of each pixel and the rejection rate when predicting semantics are obtained by the source model in the following way:

[0174] The source model calculates the semantics of each pixel point as the predicted probability of each semantic, obtains multiple predicted probabilities, and obtains the predicted probability with the largest probability value from the multiple predicted probabilities. 1 minus the predicted probability with the largest probability value is used to obtain the rejection rate when predicting the semantics. The semantics corresponding to the predicted probability with the largest probability value is the semantics of the pixel point.

[0175] For example, an image sample includes 10,000 pixels, corresponding to 10 semantics. Taking a single pixel as an example, the source model can calculate the predicted probabilities of the pixel being one of the 10 semantics. If the predicted probabilities of these 10 semantics are 0.8, 0.1, 0.05, 0.01, 0.01, 0.01, 0.01, 0, and 0, respectively, and the predicted probability with the highest probability value is 0.8, then the semantic corresponding to this predicted probability is the semantic of the pixel. 1-0.8=0.2, then the rejection rate for predicting the semantic for this pixel is 0.2.

[0176] 4) Calculate the loss function of the source model and obtain the discrete vector semantic prediction network when the loss function meets the preset model convergence conditions. The loss function is calculated by calculating the predicted probability and the true probability.

[0177] For example, in the above example, taking a pixel as an example, the predicted probabilities of the 10 semantics are 0.8, 0.1, 0.05, 0.01, 0.01, 0.01, 0.01, 0, and 0, respectively. If the true probabilities of these 10 semantics are 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, and 0, respectively. The loss function is related to these 20 probability values. The loss function can be determined by means of mean squared error, average mean squared error, or cross entropy loss. These 20 probability values ​​are input into the formula for calculating the loss function to obtain the loss function.

[0178] The above combination Figures 7 to 10 The method provided in the embodiment of the present application is introduced from the perspective of a network, and the method provided in the embodiment of the present application is introduced from the perspective of an electronic device.

[0179] For example, Figure 11 FIG1 shows a schematic flow chart of an image processing method provided by an embodiment of the present application. Figure 11 As shown, the method includes the following steps:

[0180] S1101. In response to a photographing operation, a first image is acquired, where the first image includes N regions, where N is a positive integer greater than or equal to 1.

[0181] Each of the N regions may include one pixel or multiple pixels, which is not limited in this embodiment of the present application.

[0182] S1102. Obtain N second features corresponding one-to-one to the N first features from the preset features, the N first features are used to characterize N areas, each second feature in the N second features is similar to the first feature corresponding to each second feature, wherein the clarity of an image obtained by decoding the N second features is greater than the clarity of an image obtained by decoding the N first features, and the distortion of an image obtained by decoding the N second features is greater than or equal to the distortion of an image obtained by decoding the N first features.

[0183] The preset feature is used to represent the vector in the above embedding space. The vector in the above embedding space is obtained during the model training process and preset in the model. The preset feature is only an example name and is not limited to this embodiment of the present application.

[0184] The N first features are used to characterize the N regions, which can be understood as being obtained by extracting features from the N regions. In one example, the N first features can be extracted from the first image by an encoder.

[0185] Each of the N second features is similar to the first feature corresponding to each second feature. It can be understood that each second feature and the first feature corresponding to each second feature belong to the same category, or in other words, each second feature and the first feature corresponding to each second feature can be clustered into one category.

[0186] The distortion of the image obtained by decoding N second features is greater than or equal to the distortion of the image obtained by decoding N first features. It can also be described as: the fidelity of the image obtained by decoding N second features is less than or equal to the fidelity of the image obtained by decoding N first features. This embodiment of the present application does not limit this.

[0187] In the above example, N second features corresponding to N first features are obtained from the preset features, and the above discrete feature matching can be referred to. e (x), N second features can refer to the above z q (x).

[0188] S1103. Fuse the N first features and the N second features to generate a second image, wherein the fusion weight of any one of the N first features is positively correlated with the distortion of an image obtained by decoding a second feature corresponding to any one of the first features; and the fusion weight of any one of the N second features is negatively correlated with the distortion of an image obtained by decoding any one of the second features.

[0189] When the distortion of the image obtained by decoding the second feature corresponding to any one of the N first features is large, the fusion weight of any one of the first features is larger, and correspondingly, the fusion weight of the second feature corresponding to any one of the first features is smaller. Since the fidelity of any one of the N first features is greater than the second feature corresponding to any one of the first features, it is beneficial to reduce the probability of distortion of the second image.

[0190] Optionally, before fusing the N first features and the N second features, the above method also includes: inputting the N second features into a semantic prediction model to obtain the semantics of each of the N regions and the rejection rate of each region when predicting the semantics, the rejection rate is positively correlated with the distortion of the image obtained by decoding each second feature, and the rejection rate is positively correlated with the fusion weight of the first feature corresponding to each second feature.

[0191] The semantic prediction model can refer to the above-mentioned discrete vector semantic prediction model. The model name is just an example and is not limited to this embodiment of the present application.

[0192] If each of the N regions includes a pixel, the semantics of each pixel and the rejection rate of each pixel when predicting the semantics can be obtained through the semantic prediction model.

[0193] The rejection rate is positively correlated with the distortion of the image obtained by decoding each second feature. A higher rejection rate indicates a greater distortion.

[0194] In this way, by identifying semantics and rejection rates through a semantic prediction model, the implicitly expressed second feature can be visualized through semantics, and the fusion weight of the first feature can be influenced by the rejection rate. The generation of the second image can be indirectly guided by the rejection rate, thus realizing semantic-guided implicitization.

[0195] Optionally, the semantic prediction model is trained in the following manner: M third features are obtained, and the M third features are used to characterize M regions in the image sample, where the image sample includes multiple semantics, and M is a positive integer greater than 1; the M third features are input into the source model to obtain the semantics of each region in the M regions and the rejection rate of each region when predicting the semantics, wherein the source model is used to: calculate the prediction probability that the semantics of the i-th region in the M regions is each semantic in the multiple semantics, obtain multiple prediction probabilities, and obtain a target probability that meets preset conditions from the multiple prediction probabilities, and based on the inverse correlation between the target probability, the prediction probability and the rejection rate, calculate the rejection rate of the i-th region in the M regions when predicting the semantics, and output the semantics corresponding to the target probability and the rejection rate of the i-th region in the M regions when predicting the semantics; wherein i is an integer traversing from 1 to M; when the loss function meets the preset model convergence condition, a semantic prediction model is obtained, and the loss function is related to the difference between the predicted probability and the true probability.

[0196] Each of the M regions may include one pixel or multiple pixels. The M third features may refer to the above-mentioned multiple features 2. The preset condition may be the maximum prediction probability, and the target probability is the maximum prediction probability among the multiple prediction probabilities.

[0197] The anti-correlation relationship between the prediction probability and the rejection probability can be: the sum of the prediction probability and the rejection probability is 1. In this way, the rejection rate of the i-th region in the M regions when predicting semantics and the sum of the prediction probability of the i-th region in the M regions when predicting semantics is 1.

[0198] The training process of the semantic prediction model can refer to the training process of the discrete vector semantic prediction network mentioned above, which will not be repeated here.

[0199] This helps the semantic prediction model to predict semantics and output the rejection rate when predicting semantics.

[0200] Optionally, when the rejection rate of any one of the N regions in predicting semantics is less than or equal to the first value and greater than or equal to the second value, the fusion weight of any one of the N first features is the first weight; when the rejection rate of any one of the N regions in predicting semantics is greater than or equal to the third value and less than the second value, the fusion weight of any one of the N first features is the second weight; wherein the second value is less than the first value, the third value is less than the second value, and the second weight is less than the first weight. In one example, the first value is 1, the second value can refer to the above T1, and the first weight can refer to the above a1. The third value can refer to the above T2, and the second weight can refer to the above a2.

[0201] In another example, the first value may refer to T1, the second value may refer to T2, the first weight may refer to a2, the third value may be 0, and the second weight may refer to a3.

[0202] When the rejection rate is within a certain range, the weight is the same; when the rejection rate is within different ranges, the weight is different. This is conducive to flexibly adjusting the weight according to the rejection rate.

[0203] Optionally, the electronic device includes an encoding module and a decoding module; the N first features are extracted from the first image by the encoding module, the encoding module includes L layers of downsampling, the first layer of downsampling in the L layers of downsampling is used to extract N fourth features for representing N areas from the first image, the resolution of the image obtained by decoding the N fourth features is greater than the resolution of the image obtained by decoding the N first features, and L is a positive integer greater than or equal to 1; the N first features and the N second features are fused to generate a second image, including: after the decoding module decodes the N second features, the N fourth features are fused to generate the second image.

[0204] If the encoding module includes one layer of downsampling, the N fourth features are the same as the N first features. The resolution of an image obtained by decoding the N fourth features is the same as the resolution of an image obtained by decoding the N first features.

[0205] If the encoding module includes multi-layer downsampling, the N fourth features are different from the N first features, and a resolution of an image obtained by decoding the N fourth features is greater than a resolution of an image obtained by decoding the N first features.

[0206] The resolution of the image obtained by decoding the N second features is the same as the resolution of the image obtained by decoding the N fourth features. Therefore, after the decoding module decodes the N features, the N fourth features are fused to generate the second image.

[0207] In this way, the outputs of layers with the same resolution are fused to facilitate the generation of the second image.

[0208] Optionally, the N fourth features are represented by Fe, the output after decoding the N second features is represented by Fd, and the second image is represented by Fm, then the second image satisfies the following formula:

[0209] Fe*W+Fd=Fm

[0210] Among them, W is used to represent the matrix composed of the fusion weights of any fourth feature among the N fourth features, and the matrix composed of the fusion weights of any first feature among the N first features is obtained by transforming W.

[0211] If the N fourth features are the same as the N first features, then the dimension of W is the same as the dimension of the matrix formed by the fusion weights of any one of the N first features, and W is the matrix formed by the fusion weights of any one of the N first features.

[0212] If the N fourth features are different from the N first features, the dimension of W is greater than the dimension of the matrix composed of the fusion weights of any of the N first features, and the matrix composed of the fusion weights of any of the N first features can be transformed by W. The changes between matrices can refer to the above Figure 9 , I will not go into details here.

[0213] In this way, when the distortion rate is greater, the value of the weight is greater, the larger the element in W is, the greater the role played by Fe, and correspondingly, the smaller the role played by Fd, which is conducive to reducing the distortion rate of the image.

[0214] Optionally, after the decoding module decodes the N second features and before fusing the N fourth features, the method also includes: performing feature parallel connection on the output after decoding the N second features and the N fourth features; inputting the output result after the feature parallel connection into the residual dense block RRDB to obtain the output of RRDB; convolving the output of RRDB to obtain the output result; after the decoding module decodes the N second features, fusing the N fourth features, including: after the decoding module decodes the N second features, fusing the output result.

[0215] This method can refer to the above Figure 8 In this way, the output of the encoder can be successfully applied to the input of the decoder, reducing the probability of distortion.

[0216] Optionally, obtaining N second features corresponding one-to-one to the N first features from the preset features includes: finding a feature having the smallest Euclidean distance with each of the N first features from the preset features to obtain the N second features.

[0217] The specific implementation of this method can refer to the above formula, which will not be described in detail here. In this way, it is helpful to find features that can be used to decode a high-definition image and improve the clarity of the second image.

[0218] Optionally, in response to a photographing operation, obtaining a first image includes: in response to the photographing operation, obtaining multiple frames of original images; calculating a registration matrix and / or AE parameters based on the multiple frames of original images, the registration matrix being a matrix used to represent the geometric transformation relationship between the multiple frames of original images, the AE parameters including at least one of the exposure time, brightness or sensitivity of the multiple frames of original images; and fusing the multiple frames of original images based on the registration matrix and / or AE parameters to obtain the first image.

[0219] This method can refer to the above Figure 6 In this way, fusing multiple frames of images is beneficial to reducing noise in the first image, reducing motion blur, and thus improving the quality of the second image.

[0220] Optionally, after fusing N first features and N second features to generate a second image, the method further includes: processing the brightness of the second image by gamma correction, global tone mapping GTM, sharpening, or at least one of the first curves, wherein the first curve is used to represent the correspondence between brightness and contrast.

[0221] This method can refer to the above Figure 6 This helps to adjust the brightness of the second image to better meet the needs.

[0222] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.

[0223] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0224] The image processing method according to the embodiment of the present application has been described above. The following describes the apparatus for performing the above method provided in the embodiment of the present application. Those skilled in the art will appreciate that the method and apparatus may be combined and referenced with each other, and the relevant apparatus provided in the embodiment of the present application may perform the steps in the above image processing method.

[0225] Figure 12 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Figure 12 As shown, the chip 120 includes one or more (including two) processors 1201 , a communication line 1202 , a communication interface 1203 and a memory 1204 .

[0226] In some embodiments, the memory 1204 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0227] The image processing method described in the above embodiment of the present application can be applied to the processor 1201, or implemented by the processor 1201. The processor 1201 may be an integrated circuit chip with signal processing capabilities. During the implementation process, the various steps of the above image processing method can be completed by the hardware integrated logic circuit in the processor 1201 or the instructions in the form of software. The above-mentioned processor 1201 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1201 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiments of the present application.

[0228] The steps of the image processing method disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read-only memory (EEPROM). The storage medium is located in the memory 1204, and the processor 1201 reads the information in the memory 1204 and performs the steps of the above method in conjunction with its hardware.

[0229] The processor 1201 , the memory 1204 , and the communication interface 1203 may communicate with each other via the communication line 1202 .

[0230] In the above embodiment, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0231] The image processing method provided in the embodiment of the present application can be applied to electronic devices with a camera function. The electronic device includes a terminal device. The specific device form of the terminal device can refer to the above-mentioned related description and will not be repeated here.

[0232] An embodiment of the present application provides a terminal device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the terminal device to execute the above method.

[0233] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program or instructions. When the computer program or instructions are executed by an electronic device, the above method is implemented. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0234] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium intended to carry or store the desired program code in the form of instructions or data structures and accessible by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and optical disc as used herein include optical disc, laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0235] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program code runs on an electronic device, the electronic device executes the above method.

[0236] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to produce a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0237] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: Used in electronic equipment, including: In response to a photographing operation, acquiring a first image, where the first image includes N regions, where N is a positive integer greater than or equal to 1; Obtaining N second features corresponding one-to-one to the N first features from the preset features, the N first features being used to characterize the N regions, each of the N second features being similar to the first feature corresponding to each second feature, wherein clarity of an image obtained by decoding the N second features is greater than clarity of an image obtained by decoding the N first features, and distortion of an image obtained by decoding the N second features is greater than or equal to distortion of an image obtained by decoding the N first features; Extracting N fourth features for representing the N regions from the first image based on first-layer downsampling; Performing feature parallel connection on outputs after decoding the N second features and the N fourth features; Input the output result after the feature parallel connection into the residual dense block RRDB to obtain the output of the RRDB; Convolving the output of the RRDB to obtain an output result; After decoding the N second features, the output results are fused to generate a second image; the fusion weight of any second feature among the N second features is negatively correlated with the distortion of the image obtained by decoding any second feature.

2. The method according to claim 1, characterized in that After decoding the N second features and before fusing the output results, the method further includes: The N second features are input into a semantic prediction model to obtain the semantics of each of the N regions and the rejection rate of each region when predicting the semantics, the rejection rate is positively correlated with the distortion of the image obtained by decoding each second feature, and the rejection rate is positively correlated with the fusion weight of the first feature corresponding to each second feature.

3. The method according to claim 2, characterized in that The semantic prediction model is trained in the following way: Obtaining M third features, where the M third features are used to characterize M regions in an image sample, where the image sample includes multiple semantics, and M is a positive integer greater than 1; The M third features are input into the source model to obtain the semantics of each region in the M regions and the rejection rate of each region when predicting the semantics, wherein the source model is used to: calculate the prediction probability that the semantics of the i-th region in the M regions is each of the multiple semantics to obtain multiple prediction probabilities, and obtain a target probability that meets the preset conditions from the multiple prediction probabilities, and based on the inverse correlation between the target probability, the prediction probability and the rejection rate, calculate the rejection rate of the i-th region in the M regions when predicting the semantics, and output the semantics corresponding to the target probability and the rejection rate of the i-th region in the M regions when predicting the semantics; wherein i is an integer traversing from 1 to M; When the loss function satisfies a preset model convergence condition, the semantic prediction model is obtained, and the loss function is related to the difference between the predicted probability and the true probability.

4. The method according to claim 2 or 3, characterized in that The rejection rate of any one of the N regions when predicting semantics is less than or equal to the first value and greater than or equal to the second value, and the fusion weight of any one of the N first features is the first weight; When the rejection rate of any one of the N regions in semantic prediction is greater than or equal to the third value and less than the second value, the fusion weight of any one of the N first features is the second weight; The second value is smaller than the first value, the third value is smaller than the second value, and the second weight is smaller than the first weight.

5. The method according to any one of claims 1 to 3, characterized in that The electronic device includes an encoding module and a decoding module; the N first features are extracted by the encoding module from the first image, the encoding module includes L layers of downsampling, the first layer of downsampling in the L layers of downsampling is used to extract N fourth features for representing the N regions from the first image, the resolution of the image obtained by decoding the N fourth features is greater than the resolution of the image obtained by decoding the N first features, and L is a positive integer greater than or equal to 1.

6. The method according to any one of claims 1 to 3, characterized in that The obtaining of N second features corresponding one-to-one to the N first features from the preset features includes: Find the feature with the smallest Euclidean distance to each of the N first features among the preset features to obtain the N second features.

7. The method according to any one of claims 1 to 3, characterized in that The acquiring of the first image in response to the photographing operation includes: In response to a photographing operation, acquiring multiple frames of original images; Calculating a registration matrix and / or AE parameters based on the multiple frames of original images, where the registration matrix is ​​a matrix representing a geometric transformation relationship between the multiple frames of original images, and the AE parameters include at least one of exposure time, brightness, or sensitivity of the multiple frames of original images; The multiple frames of original images are fused based on the registration matrix and / or the AE parameters to obtain the first image.

8. The method according to any one of claims 1 to 3, characterized in that After decoding the N second features and fusing the output results to generate a second image, the method further includes: The brightness of the second image is processed by at least one of gamma correction, global tone mapping (GTM), sharpening, or a first curve, wherein the first curve is used to represent a corresponding relationship between brightness and contrast.

9. An electronic device, characterized in that: The electronic device includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1 to 8.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 8.

Images