Image processing method and related device

Abstract

The embodiment of the invention provides an image processing method and a related device, and relates to the technical field of terminals. The method comprises the steps that in response to a photographing operation, a first image is acquired, and the first image comprises N areas; n second features in one-to-one correspondence with the N first features are obtained from preset features, the N first features are used for representing the N regions, and each second feature in the N second features is similar to the first feature corresponding to each second feature, the distortion degree of an image obtained by decoding the N second features is greater than or equal to the distortion degree of an image obtained by decoding the N first features; the N first features and the N second features are fused to generate a second image, and the fusion weight of any first feature in the N first features is positively correlated with the distortion degree of the image obtained by decoding the second feature corresponding to any first feature. Therefore, the probability of image distortion can be reduced.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and related devices. Background Art

[0002] With the development of terminal technology, the functions of electronic devices are becoming more and more abundant. Electronic devices can be configured with one or more cameras, and the one or more cameras can be used to realize the photo taking function.

[0003] In a photo-taking scenario, an electronic device may capture images through one or more configured cameras.

[0004] However, in the above-mentioned photo-taking scenarios, images taken by electronic devices are prone to image distortion. Summary of the invention

[0005] The embodiments of the present application provide an image processing method and related devices, which are applied in the field of terminal technology and are helpful in reducing the probability of image distortion.

[0006] In a first aspect, an embodiment of the present application provides an image processing method that can be applied to an electronic device. The method includes: in response to a photo-taking operation, acquiring a first image, the first image including N regions, N being a positive integer greater than or equal to 1; obtaining N second features corresponding to the N first features in a preset feature, the N first features being used to characterize the N regions, each second feature in the N second features being similar to the first feature corresponding to each second feature, wherein the clarity of the image obtained by decoding the N second features is greater than the clarity of the image obtained by decoding the N first features, and the distortion of the image obtained by decoding the N second features is greater than or equal to the distortion of the image obtained by decoding the N first features; fusing the N first features and the N second features to generate a second image, the fusion weight of any one of the N first features being positively correlated with the distortion of the image obtained by decoding the second feature corresponding to any one of the first features; and the fusion weight of any one of the N second features being negatively correlated with the distortion of the image obtained by decoding any one of the second features.

[0007] When the distortion of the image obtained by decoding the second feature corresponding to any one of the N first features is large, the fusion weight of any one of the first features is large, and accordingly, the fusion weight of the second feature corresponding to any one of the first features is smaller. Since the fidelity of any one of the N first features is greater than the second feature corresponding to any one of the first features, it is beneficial to reduce the probability of distortion of the second image.

[0008] In one possible implementation, before fusing the N first features and the N second features, the method also includes: inputting the N second features into a semantic prediction model to obtain the semantics of each of the N regions and a rejection rate of each region when predicting the semantics, the rejection rate being positively correlated with the distortion of the image obtained by decoding each second feature, and the rejection rate being positively correlated with the fusion weight of the first feature corresponding to each second feature.

[0009] In this way, by identifying semantics and rejection rate through the semantic prediction model, the implicitly expressed second feature can be visualized through semantics, and the fusion weight of the first feature can be affected by the rejection rate. The generation of the second image can be indirectly guided by the rejection rate, thus realizing semantic-guided implicitization.

[0010] In a possible implementation, the semantic prediction model is trained in the following manner: M third features are obtained, and the M third features are used to characterize M regions in an image sample, where the image sample includes multiple semantics, and M is a positive integer greater than 1; the M third features are input into a source model to obtain the semantics of each region in the M regions and the rejection rate of each region when predicting the semantics, wherein the source model is used to: calculate the prediction probability that the semantics of the i-th region in the M regions is each of the multiple semantics, obtain multiple prediction probabilities, and obtain a target probability that meets preset conditions from the multiple prediction probabilities, and based on the anti-correlation relationship between the target probability, the prediction probability and the rejection rate, calculate the rejection rate of the i-th region in the M regions when predicting the semantics, and output the semantics corresponding to the target probability and the rejection rate of the i-th region in the M regions when predicting the semantics; wherein i is an integer traversing from 1 to M; when the loss function meets the preset model convergence condition, a semantic prediction model is obtained, and the loss function is related to the difference between the predicted probability and the true probability.

[0011] This helps to enable the semantic prediction model to predict semantics and output the rejection rate when predicting semantics.

[0012] In one possible implementation, when the rejection rate of any one of the N regions when predicting semantics is less than or equal to the first value and greater than or equal to the second value, the fusion weight of any one of the N first features is the first weight; when the rejection rate of any one of the N regions when predicting semantics is greater than or equal to the third value and less than the second value, the fusion weight of any one of the N first features is the second weight; wherein the second value is less than the first value, the third value is less than the second value, and the second weight is less than the first weight.

[0013] When the rejection rate is within a certain range, the weight is the same; when the rejection rate is within different ranges, the weight is different. This is conducive to flexibly adjusting the weight according to the rejection rate.

[0014] In a possible implementation, the electronic device includes an encoding module and a decoding module; N first features are extracted from a first image by the encoding module, the encoding module includes L layers of downsampling, the first layer of downsampling in the L layers of downsampling is used to extract N fourth features for representing N regions from the first image, the resolution of the image obtained by decoding the N fourth features is greater than the resolution of the image obtained by decoding the N first features, and L is a positive integer greater than or equal to 1; the N first features and the N second features are fused to generate a second image, including: after the decoding module decodes the N second features, the N fourth features are fused to generate the second image. In this way, fusing the outputs of layers with the same resolution is conducive to generating the second image.

[0015] In a possible implementation, N fourth features are represented by Fe, the output after decoding the N second features is represented by Fd, and the second image is represented by Fm, then the second image satisfies the following formula:

[0016] Fe*W+Fd=Fm

[0017] Among them, W is used to represent the matrix composed of the fusion weights of any fourth feature among the N fourth features, and the matrix composed of the fusion weights of any first feature among the N first features is obtained by transforming W.

[0018] In this way, when the distortion rate is larger, the value of the weight is larger, the elements in W are larger, the role of Fe is larger, and correspondingly, the role of Fd is smaller, which is beneficial to reducing the distortion rate of the image.

[0019] In a possible implementation, after the decoding module decodes the N second features and before fusing the N fourth features, the method also includes: performing feature parallel connection on the output after decoding the N second features and the N fourth features; inputting the output result after the feature parallel connection into the residual dense block RRDB to obtain the output of RRDB; convolving the output of RRDB to obtain the output result; after the decoding module decodes the N second features, the N fourth features are fused, including: after the decoding module decodes the N second features, the output result is fused.

[0020] In this way, the output of the encoder can successfully act on the input of the decoder, reducing the probability of distortion.

[0021] In a possible implementation, obtaining N second features corresponding one-to-one to the N first features from the preset features includes: finding a feature having the smallest Euclidean distance with each of the N first features from the preset features to obtain the N second features.

[0022] This is helpful to find features that can decode a higher-definition image and improve the clarity of the second image.

[0023] In one possible implementation, in response to a photographing operation, a first image is acquired, including: in response to the photographing operation, a plurality of frames of original images are acquired; based on the plurality of frames of original images, a registration matrix and / or AE parameters are calculated, wherein the registration matrix is ​​a matrix for representing a geometric transformation relationship between the plurality of frames of original images, and the AE parameters include at least one of exposure time, brightness or sensitivity of the plurality of frames of original images; and the plurality of frames of original images are fused based on the registration matrix and / or AE parameters to obtain the first image.

[0024] In this way, fusing multiple frames of images is beneficial to reducing noise in the first image, reducing the phenomenon of motion blur, and further beneficial to improving the quality of the second image.

[0025] In one possible implementation, after fusing N first features and N second features to generate a second image, the method further includes: processing the brightness of the second image by gamma correction, global tone mapping GTM, sharpening, or at least one of the first curves, wherein the first curve is used to represent the correspondence between brightness and contrast.

[0026] This helps to adjust the brightness of the second image to better meet the needs.

[0027] In a second aspect, an embodiment of the present application provides an image processing device, which may be an electronic device, or a chip or chip system in an electronic device. The image processing device may include an acquisition unit and a processing unit. When the image processing device is an electronic device, the processing unit may be a processor. The image processing device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit in the chip (for example, a register, a cache, etc.), or a storage unit in the electronic device located outside the chip (for example, a read-only memory, a random access memory, etc.).

[0028] Exemplarily, an acquisition unit is used to acquire a first image in response to a photographing operation, where the first image includes N areas, where N is a positive integer greater than or equal to 1; a processing unit is used to obtain N second features corresponding to the N first features in preset features, where the N first features are used to characterize the N areas, and each of the N second features is similar to the first feature corresponding to each second feature, wherein the clarity of an image obtained by decoding the N second features is greater than the clarity of an image obtained by decoding the N first features, and the distortion of an image obtained by decoding the N second features is greater than or equal to the distortion of an image obtained by decoding the N first features; the N first features and the N second features are fused to generate a second image, and the fusion weight of any one of the N first features is positively correlated with the distortion of an image obtained by decoding the second feature corresponding to any one of the first features; and the fusion weight of any one of the N second features is negatively correlated with the distortion of an image obtained by decoding any one of the second features.

[0029] In one possible implementation, the processing unit is also used to: input N second features into a semantic prediction model to obtain the semantics of each of the N regions and a rejection rate of each region when predicting semantics, the rejection rate is positively correlated with the distortion of the image obtained by decoding each second feature, and the rejection rate is positively correlated with the fusion weight of the first feature corresponding to each second feature.

[0030] In a possible implementation, the acquisition unit is also used to: acquire M third features, the M third features are used to characterize M regions in an image sample, the image sample includes multiple semantics, and M is a positive integer greater than 1; the processing unit is also used to: input the M third features into a source model to obtain the semantics of each region in the M regions and the rejection rate of each region when predicting the semantics, wherein the source model is used to: calculate the prediction probability that the semantics of the i-th region in the M regions is each of the multiple semantics, obtain multiple prediction probabilities, and obtain a target probability that meets preset conditions from the multiple prediction probabilities, and based on the anti-correlation relationship between the target probability, the prediction probability and the rejection rate, calculate the rejection rate of the i-th region in the M regions when predicting the semantics, and output the semantics corresponding to the target probability and the rejection rate of the i-th region in the M regions when predicting the semantics; wherein i is an integer traversing from 1 to M; when the loss function meets the preset model convergence condition, a semantic prediction model is obtained, and the loss function is related to the difference between the predicted probability and the true probability.

[0031] In one possible implementation, when the rejection rate of any one of the N regions when predicting semantics is less than or equal to the first value and greater than or equal to the second value, the fusion weight of any one of the N first features is the first weight; when the rejection rate of any one of the N regions when predicting semantics is greater than or equal to the third value and less than the second value, the fusion weight of any one of the N first features is the second weight; wherein the second value is less than the first value, the third value is less than the second value, and the second weight is less than the first weight.

[0032] In a possible implementation, the image processing device also includes an encoding module and a decoding module; the N first features are extracted from the first image by the encoding module, the encoding module includes L layers of downsampling, the first layer of downsampling in the L layers of downsampling is used to extract N fourth features for representing N regions from the first image, the resolution of the image obtained by decoding the N fourth features is greater than the resolution of the image obtained by decoding the N first features, and L is a positive integer greater than or equal to 1; the decoding module is used to: after decoding the N second features, fuse the N fourth features to generate a second image.

[0033] In a possible implementation, N fourth features are represented by Fe, the output after decoding the N second features is represented by Fd, and the second image is represented by Fm, then the second image satisfies the following formula:

[0034] Fe*W+Fd=Fm

[0035] Among them, W is used to represent the matrix composed of the fusion weights of any fourth feature among the N fourth features, and the matrix composed of the fusion weights of any first feature among the N first features is obtained by transforming W.

[0036] In a possible implementation, the processing unit is also used to: perform feature parallel connection on the output after decoding the N second features and the N fourth features; input the output result after the feature parallel connection into the residual dense block RRDB to obtain the output of RRDB; convolve the output of RRDB to obtain the output result; the decoding module is also used to: after decoding the N second features, fuse the output results.

[0037] In a possible implementation, the processing module is further used to: find a feature with the smallest Euclidean distance to each of the N first features among the preset features, to obtain N second features.

[0038] In a possible implementation, the acquisition unit is further used to: acquire multiple frames of original images in response to a photo-taking operation; the processing unit is further used to: calculate a registration matrix and / or AE parameters based on the multiple frames of original images, the registration matrix is ​​a matrix used to represent the geometric transformation relationship between the multiple frames of original images, and the AE parameters include at least one of the exposure time, brightness or sensitivity of the multiple frames of original images; and fuse the multiple frames of original images based on the registration matrix and / or AE parameters to obtain a first image.

[0039] In a possible implementation, the processing unit is further used to: process the brightness of the second image through gamma correction, global tone mapping GTM, sharpening, or at least one of the first curves, wherein the first curve is used to represent the corresponding relationship between brightness and contrast.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method described in the first aspect or any possible implementation of the first aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are executed on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation of the first aspect.

[0043] In a sixth aspect, the present application provides a chip or a chip system, which is applied to an electronic device, and the chip or chip system includes at least one or more processors, and the one or more processors are used to call computer instructions to execute the method described in the first aspect or any possible implementation of the first aspect.

[0044] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0045] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a semantic guidance provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a flow chart of a VQVAE algorithm provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a technical concept provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of a generative network provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of a network structure in a decoder provided in an embodiment of the present application;

[0054] Fig. 9 A schematic diagram of weight calculation provided in an embodiment of the present application;

[0055] Fig.10 A schematic diagram of a network structure in a decoder provided in an embodiment of the present application;

[0056] Fig.11 A schematic flow chart of another image processing method provided in an embodiment of the present application;

[0057] Fig.12 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0059] 1. Semantic Guidance

[0060] Semantic guidance can guide the algorithm to generate a specific object by providing semantic information. In some examples, the semantic information can be a semantic segmentation map, the algorithm can be a neural network algorithm, and the specific object can be a scene.

[0061] For example, Figure 1 A schematic diagram of semantic guidance is shown. Figure 1 As shown in FIG, the semantic segmentation map can be used as a guiding framework for generating images, by guiding the generation model to generate multiple images with similar semantic structures through the pixels in the semantic segmentation image and the semantics corresponding to the pixels.

[0062] Semantic guidance can be implemented in two ways: implicit semantic guidance and explicit semantic guidance.

[0063] Semantic guidance implicit means that the generative model does not need to know the semantics of each pixel explicitly, but uses the semantics of each pixel as an implicit guiding signal to indirectly control the semantic structure and details of the generated image. This implicit approach allows the generative model to learn the semantic structure of the image without explicit semantic labels, thereby generating multiple images with similar semantic information.

[0064] Semantic guidance explicitness means that the generative model needs to know the semantics of each pixel explicitly, and the semantic structure and details of the generated image are controlled by the semantics of each pixel. This explicit approach allows the generative model to learn the semantic structure of the image in the presence of explicit semantic labels, thereby generating multiple images with similar semantic information.

[0065] 2. Vector quantized variational auto encoder (VQVAE) algorithm

[0066] The VQVAE algorithm is a method that combines vector quantization (VQ) and variational auto encoder (VAE).

[0067] Among them, VAE is a generative model that can include: an encoder, a latent space, and a decoder. The encoder can encode the input data of the generative model into the latent space, the data in the latent space satisfies the prior distribution, and the decoder can reconstruct the data in the latent space.

[0068] VQ is used to discretize the data in the latent space of VAE.

[0069] For example, Figure 2 A schematic diagram of the flow of a VQVAE algorithm is shown. Figure 2 As shown in Figure 1, VQ discretizes the data in the latent space of VAE through the embedding space. The embedding space includes an embedding table, which can include k vectors, each of which has a dimension of D. The k vectors can be represented as e1, e2, ..., e k .

[0070] The processing of the VQVAE algorithm can include:

[0071] 1) After passing an image through the encoder, we can get a H×W×D feature map, which can be represented by z e (x) indicates.

[0072] 2) These H×W D-dimensional vectors can be found in the embedding table respectively. i , and use e i The index (index) of q(z|x) can be expressed as q(z|x). The numbers in q(z|x) are all discrete, and each number can be regarded as a probability distribution. There are K dimensions in total, and each dimension represents the corresponding embedding table e i The probability of (i=1, 2, ..., k).

[0073] 3) Feature map z e (x) can be obtained by embedding the closest e in the table i After substitution, we get z q (x). Among them, from z e (x) to z q The change of (x) can be regarded as a cluster, that is, the feature obtained by the encoder is represented by a vector closest to it in the embedding table.

[0074] Among them, 2) and 3) can be called discrete feature matching.

[0075] 4) z q (x) is the input of the decoder, z q (x) After the decoder, we can get the processed image p(x|z q ).

[0076] It can be understood that the VQVAE algorithm includes three processing steps: encoding, discrete feature matching, and decoding.

[0077] 3. Other terms

[0078] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first image and the second image are only used to distinguish different images, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0079] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0080] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0081] 4. Electronic equipment

[0082] The electronic device of the embodiment of the present application may include a handheld device, a vehicle-mounted device, etc. with processing functions. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 5G networks or future evolved public land mobile communication networks (public landmobile networks). The embodiments of the present application do not limit this.

[0083] In addition, in the embodiments of the present application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0084] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0085] In a photo-taking scenario, an electronic device may collect images through one or more configured cameras. However, in the above photo-taking scenario, the images captured by the electronic device are prone to image distortion.

[0086] In some implementations, the electronic device may be configured with a complementary metal oxide semiconductor (CMOS) camera. In a photo-taking scenario, the electronic device may capture a raw image (Raw Image) through a CMOS camera in response to a photo-taking operation, and the data format of the raw image may adopt a Bayer arrangement. The electronic device may be deployed with an image processing algorithm (ISP), which may be used to denoise, enhance, and optimize the raw image captured by the CMOS camera to provide a clear and accurate image to the user.

[0087] However, the ISP algorithm is limited by the technical bottlenecks of various processing algorithms, especially for solving irreversible problems such as super-resolution. It is difficult to restore high-definition images at high magnifications through traditional upsampling algorithms or deterministic learning methods. This will result in poor clarity of images taken by electronic devices.

[0088] In some examples, in order to improve the clarity of images taken by the electronic device, the electronic device may be deployed with a generative model. In a photo-taking scenario, the original image captured by the CMOS camera may be processed by the generative model to obtain a high-definition image. For example, the generative model may be VQVAE.

[0089] However, the generative model is limited by the technical bottleneck of its own algorithm, which can cause image distortion.

[0090] For example, in the generative model VQVAE, the feature map z e (x) can be obtained by embedding the closest e in the table i After substitution, we get z q (x), for z q (x) is decoded, so that the features used for decoding are not the features output by the encoder, but the features embedded in the table i , e embedded in the table i The clarity of the image obtained after decoding is greater than the clarity of the image obtained after decoding the features output by the encoder, but the e embedded in the table iThe fidelity of the decoded image is less than the fidelity of the image obtained after decoding the features output by the encoder. Therefore, although the image processed by the generative model has improved clarity, image distortion will occur. It can be understood that fidelity and distortion are inversely correlated. The smaller the fidelity, the greater the distortion. Conversely, the greater the fidelity, the smaller the distortion.

[0091] In view of this, an embodiment of the present application provides an image processing method and a related device. When the fidelity of the vectors in the embedding table is poor or the distortion is high, in the process of obtaining the desired image by using the features output by the encoder and the features fused with the vectors in the embedding table, the fusion weight of the features output by the encoder is larger. In this way, compared with generating images using the vectors in the embedding table, it is helpful to reduce the probability of image distortion.

[0092] In order to better understand the method provided in the embodiments of the present application, Figure 3 Introduce the technical concept of the embodiments of the present application.

[0093] Figure 3 A schematic diagram of a technical concept is shown. Figure 3 As shown in Figure 2, the purpose of semantic guidance is to guide generation. The generation model includes the process of vector discretization, in which the vector discretization is implicitly expressed, that is, the embedding table e is unknown. i (i=1, 2, ..., k) represents the meaning.

[0094] The embodiment of the present application combines semantic-guided implicitization with implicit expression concretization, realizes implicit expression concretization through the meaning of vectors in the semantic expression embedding table, and then realizes discretization representation of spatial regions, and calculates the rejection rate when predicting semantics. The rejection rate is positively correlated with the distortion. The distortion of the vectors in the embedding table is reflected by the rejection rate. The fusion weight of the features output by the encoder is calculated through the rejection rate and the correlation between the rejection rate and the weight. The generation of the image is guided by the weight to realize semantic-guided implicitization (i.e., indirectly guiding the generation of the image through semantics).

[0095] When the rejection rate is large and the distortion is large, the fusion weight of the features output by the encoder is large. When the rejection rate is small and the distortion is small, the fusion weight of the features output by the encoder is small. This is helpful to reduce the probability of image distortion.

[0096] The method provided in the embodiment of the present application can be executed by an electronic device. To facilitate understanding, the hardware structure of the electronic device provided in the embodiment of the present application is first introduced.

[0097] For example, Figure 4 FIG. 1 shows a hardware structure diagram of an electronic device provided in an embodiment of the present application. Figure 4As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, and a display screen 194, etc.

[0098] Optionally, the above-mentioned sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0099] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0100] In the embodiment of the present application, the processor 110 may be deployed with the image processing method provided in the embodiment of the present application. In the photo shooting scene, the processor 110 may obtain the image captured by the camera 193, and process the captured image by the method provided in the embodiment of the present application and then output it. The image processed by the processor 110 may be displayed on the display screen 194.

[0101] The camera 193 may include one or more cameras, which is not limited in the present application. In some examples, the camera 193 may include a CMOS camera and / or a charge coupled device (CCD) camera.

[0102] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The layered architecture can adopt an Android system, an Apple (IOS) system, or other operating systems, which are not limited in the embodiments of the present application. The following takes the Android system of the layered architecture as an example to exemplify the software architecture of the electronic device provided in the embodiments of the present application.

[0103] Figure 5A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application is shown. Figure 5 As shown, the layered architecture divides the software architecture of the electronic device into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom, namely, application layer (applications), application framework layer (application framework), hardware abstraction layer (hardware abstraction layer, HAL), kernel layer (kernel) and hardware layer.

[0104] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the application framework layer. Figure 5 As shown, the application package may include applications such as camera and gallery.

[0105] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 5 As shown, the application framework layer may include a camera access interface and a view system. The camera access interface may be used to provide an application programming interface and a programming framework for camera applications.

[0106] like Figure 5 As shown, the HAL layer may include a camera hardware abstraction layer and a camera algorithm library. Among them, the camera hardware abstraction layer may provide virtual hardware for the camera device. The camera algorithm library may include operating code and data for implementing the image processing method provided in the embodiment of the present application. In other implementations, the camera algorithm library may be deployed in the camera hardware abstraction layer, which is not limited in the embodiment of the present application.

[0107] The kernel layer is the layer between hardware and software. Figure 5 As shown, the kernel layer may include a camera device driver.

[0108] The hardware layer may include hardware such as cameras and image signal processors.

[0109] It should be understood that in some embodiments, the layers implementing the same function may be called by other names, or the layers that can implement the functions of multiple layers may be regarded as one layer, or the layers that can implement the functions of multiple layers may be divided into multiple layers. The embodiments of the present application are not limited to this.

[0110] The following is combined with the above Figure 5The software structure shown in the figure specifically describes the image processing method in the embodiment of the present application:

[0111] In response to the user's operation of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera by calling the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the kernel layer. The camera device driver can start the corresponding camera to collect images.

[0112] The camera can transmit the collected image light signal to the image signal processor for preprocessing to obtain the image electrical signal, that is, the original image, and transmit the original image to the camera hardware abstraction layer through the camera device driver.

[0113] The camera hardware abstraction layer can send the original image to the camera algorithm library. The camera algorithm library stores the degree code for implementing the image processing method provided by the embodiment of the present application. Executing the code can reduce the probability of image distortion while improving image clarity.

[0114] The camera algorithm library can send the processed image to the camera hardware abstraction layer, and the camera hardware abstraction layer will display the processed image.

[0115] Combined with the above Figure 4 and Figure 5 , introduces the software and hardware structure of the electronic device in the embodiment of the present application, and then introduces the method applied to the electronic device.

[0116] In order to better understand the method provided in the embodiment of the present application, the deployment location of the core algorithm provided in the embodiment of the present application is first introduced.

[0117] For example, Figure 6 FIG. 1 is a schematic diagram of an image processing method provided by an embodiment of the present application. Figure 6 As shown, the electronic device enters the photo mode in response to the photo operation, and can obtain multiple frames of original images captured by the camera, and calculate the registration matrix and automatic exposure (AE) parameters based on the multiple frames of original images. Among them, the registration matrix can also be called a transformation matrix, which is a matrix used to describe the geometric transformation relationship between multiple frames of original images. The AE parameters may include parameters such as brightness, aperture size, exposure time, and sensitivity of each frame of the original image in the multiple frames.

[0118] The electronic device can perform denoising on each of the multiple frames of original images, and can perform format conversion on each of the frames. For example, the electronic device can convert the format of each frame of the image into RGB format through a demosaic algorithm. The electronic device can also perform high dynamic range fusion (HDR fusion) on the format-converted images according to the registration matrix and AE parameters to obtain a fused image.

[0119] The electronic device can use the generative network provided in the embodiment of the present application to process the fused image to obtain a processed image. The clarity of the processed image is greater than the clarity of the image before processing. The generative network provided in the embodiment of the present application can combine the semantic guidance implicitization with the implicit expression concretization, embed the meaning of the vector in the table through the semantic expression, and calculate the rejection rate when predicting the semantics. In the case of a large rejection rate, in the process of obtaining the desired image using the features output by the encoder and the features fused by the vectors in the embedding table, the fusion weight of the features output by the encoder is large. In the case of a small rejection rate, in the process of obtaining the desired image using the features output by the encoder and the features fused by the vectors in the embedding table, the fusion weight of the features output by the encoder is small.

[0120] Electronic devices can perform gamma correction and global tone mapping (GTM) on processed images. Both GTM and gamma correction can be used to adjust the brightness and contrast of images. GTM focuses more on global color and brightness optimization, while gamma correction focuses more on correcting the nonlinear response of the device.

[0121] The electronic device can also sharpen the processed image and adjust the brightness of the image through the brightness curve. Sharpening is used to adjust the clarity and details of the image. The brightness curve is used to represent the corresponding relationship between brightness and contrast to adjust the brightness of the image.

[0122] The electronic device may also perform a color look-up table (LUT) on the processed image, wherein the color LUT is used to adjust the color and brightness of the image.

[0123] The electronic device can compress the image after the color LUT processing. Figure 6 As shown, the electronic device can perform JPEG compression on the image after applying the color LUT processing.

[0124] The electronic device may output the compressed image, which may be displayed in a gallery application or in a preview of a camera application.

[0125] The structure and function of the generative network are introduced below.

[0126] For example, Figure 7 Figure 2 shows a schematic diagram of the structure of a generative network. Figure 7 As shown, the generative network includes an encoder, discrete feature matching, a decoder, and a discrete vector semantic prediction network. The encoder is used to extract features of the input image. The embedding space includes multiple sets of discrete implicit feature bases (or multiple vectors), and the discrete feature matching is used to find the discrete implicit feature base with the smallest Euclidean distance in the embedding table for each feature extracted by the encoder, and the index corresponding to the discrete implicit feature base is used to correspond to each feature, and the discrete implicit feature base is used to replace the features extracted by the encoder.

[0127] The specific implementation can refer to the following formula:

[0128]

[0129] z q (x) = e i , k = argmin i ‖z e (x)-e i ‖2

[0130] Among them, z e (x) is the feature extracted by the encoder, e i is a discrete implicit feature basis, and the argmin function is used to calculate the minimum value. There are k sets of discrete implicit feature bases in the embedding table. The k sets of discrete implicit feature bases are traversed. If the discrete implicit feature base e in the k sets of discrete implicit feature bases is i With z e (x) is closest, then the probability q(z=k|x) of the discrete implicit feature basis is 1. If the discrete implicit feature basis e in the k sets of discrete implicit feature bases is i Not with z e (x) is closest, then the probability q(z=k|x) of the discrete implicit feature basis is 0. Using the closest discrete implicit feature basis to replace the features extracted by the encoder, we get z q (x).

[0131] like Figure 7 As shown, 1, 2, 3, 4, ..., P are P groups of discrete implicit feature bases. After discrete feature matching, the P groups of discrete implicit feature bases are input to the decoder.

[0132] In the embodiment of the present application, the encoder and the discrete feature matching are the same as above. Figure 2 The process shown is the same.

[0133] The discrete vector semantic prediction network can obtain a set of dual-channel two-dimensional spaces based on a set of discrete implicit feature bases. If the above P sets of discrete implicit feature bases are input into the discrete vector semantic prediction network, P sets of dual-channel two-dimensional spaces can be obtained. This application takes a set of dual-channel two-dimensional spaces as an example for explanation.

[0134] The two-dimensional space of one channel in the dual-channel two-dimensional space represents the pixel-wise semantics in the semantic segmentation task, and the other channel represents the rejection rate. The rejection rate reflects the confidence of the semantic prediction. When the rejection rate is higher, the confidence is lower, indicating that the probability that the fidelity of the image obtained by decoding the discrete implicit feature base meets the requirements is smaller. When the rejection rate is lower, the confidence is higher, indicating that the probability that the fidelity of the image obtained by decoding the discrete implicit feature base meets the requirements is greater.

[0135] Among them, pixel-level semantics is used to indicate that each pixel in the image has corresponding semantics. The rejection rate corresponds to the pixel, and each pixel has a corresponding rejection rate.

[0136] The embodiment of the present application presets a corresponding relationship between the rejection rate and the weight. The greater the rejection rate, the greater the weight value. Conversely, the smaller the rejection rate, the smaller the weight value.

[0137] In some examples, the value of the rejection rate may be between 0 and 1, and the value of the weight may be between 0 and 1. The corresponding relationship between the rejection rate and the weight may include:

[0138] If the rejection rate is greater than or equal to T1 and less than or equal to 1, the weight is equal to a1. If the rejection rate is greater than or equal to T2 and less than T1, the weight is equal to a2. If the rejection rate is greater than or equal to 0 and less than T2, the weight is equal to a3. Among them, T1 is greater than T2, a1 is greater than a2, and a2 is greater than a3.

[0139] The input of the discrete vector semantic prediction network is P groups of discrete implicit feature bases, each group of discrete implicit feature bases corresponds to pixel-level semantics and rejection rates, and the output of the discrete vector semantic prediction network is P groups of pixel-level semantics and P groups of rejection rates. A group of discrete implicit feature bases can correspond to one or more pixels. If a group of discrete implicit feature bases includes a pixel, the pixel-level semantics obtained based on the group of discrete feature bases includes the semantics of the pixel and a rejection rate. If a group of discrete implicit feature bases includes multiple pixels, the pixel-level semantics obtained based on the group of discrete feature bases includes the semantics of multiple pixels and multiple rejection rates.

[0140] Based on the P group rejection rate and the corresponding relationship between the rejection rate and the weight, the P group weight can be obtained. The decoder can fuse the output of the encoder based on the P group weight in the process of decoding the output of discrete feature matching. The P group weight can be represented in the form of a matrix or array, which is not limited to the embodiments of the present application.

[0141] The number of weights is the same as the number of pixels in the decoder's input image. In other words, multiple weights can be represented by a weight matrix whose dimension is the same as the resolution of the decoder's input image.

[0142] First, the network structure of the decoder in the embodiment of the present application is introduced. Figure 8 FIG. 1 shows a schematic diagram of a network structure in a decoder. Figure 8 As shown, the decoder network structure can include multiple upsampling processes. Figure 8 In the example shown, the decoder includes four upsampling processes. The decoder processing may include:

[0143] 1) Upsample the input information four times in sequence to obtain Fd.

[0144] 2) Obtain the output of the first layer network in the encoder. In the embodiment of the present application, Fe is used to represent the output of the first layer network in the encoder. The first layer network of the encoder may be down-sampled. The number of layers included in the encoder may be the same as the number of up-sampled layers included in the decoder.

[0145] 3) After fusion processing of Fe and Fd, Fa is obtained, Fa*W+Fd=Fm, W is a matrix, Fm is a processed image. The elements in W are used to represent the fusion weights of each pixel in Fe.

[0146] The fusion processing of Fe and Fd may include: performing feature concatenation on Fd and Fe, and inputting the processing result after the feature concatenation into the fusion network to obtain Fa. The fusion network may include multiple possible implementation methods.

[0147] In a possible implementation, the fusion network may include two RRDB (residual in residual dense block) networks and a convolutional network.

[0148] In this case, the fusion processing of Fe and Fd may include: performing feature parallel connection on Fd and Fe, inputting the processing results after the feature parallel connection into two RRDB (residual in residual dense block) networks, and convolving the outputs of the two RRDB networks to obtain the output result Fa.

[0149] In another possible implementation, the fusion network may include one RRDB network and a convolutional network.

[0150] In this case, the fusion processing of Fe and Fd may include: performing feature parallel connection on Fd and Fe, inputting the processing result after the feature parallel connection into the RRDB network, and convolving the output of the RRDB network to obtain the output result Fa.

[0151] In this way, the fusion processing of Fe and Fd can make the output of the encoder successfully act on the input of the decoder, reducing the probability of distortion. In addition, when the rejection rate is larger, the value of the weight is larger, the value of the element in W is larger, the role of Fe is larger, and correspondingly, the role of Fd is smaller, which is conducive to reducing the distortion rate of the image.

[0152] The dimension of W is the same as the resolution of the image obtained by the last upsampling of the decoder. If the resolution of the image obtained by the last upsampling of the decoder is the same as the resolution of the input image of the decoder, then the elements in W are the weights obtained by the rejection rate. If the resolution of the image obtained by the last upsampling of the decoder is less than the resolution of the input image of the decoder, then the elements in W can be obtained by transforming the weights obtained by the rejection rate.

[0153] For example, the dimension of the weight matrix formed by the weights obtained by the rejection rate is H*W, and the resolution of the image obtained by the last upsampling of the decoder is H / 2*W / 2, then the dimension of W should be H / 2*W / 2. The dimension of the weight matrix is ​​4 times the dimension of W, so each element in W can be obtained by averaging every 4 elements in the weight matrix.

[0154] For example, Fig. 9 A schematic diagram of weight calculation is shown. Fig. 9 As shown, the dimension of W is H / 2*W / 2, and the dimension of the weight matrix is ​​H*W. Then the average value of each of the four upper, lower, left and right elements in these H*W elements is an element in W, which can be expressed by the formula:

[0155] (a m,n +a m,n+1 +a m+1,n +a m+1,n+1 ) / 4=A m,n

[0156] Among them, A m,n is the element in the mth row and nth column of W, a m,n is the element in the mth row and nth column of the weight matrix, a m,n+1 is the element in the mth row and (n+1)th column of the weight matrix, a m+1,n is the element in the (m+1)th row and nth column of the weight matrix, am+1,n+1 is the element in the (m+1)th row and (n+1)th column in the weight matrix.

[0157] exist Fig. 9 In the example shown, the average values ​​of the four upper, lower, left and right elements in the weight matrix may include 0.9, 0.5 and 0.1, and the elements in W may include 0.9, 0.5 and 0.1, etc.

[0158] Above Figure 8 In the processing of the decoder shown, after the last upsampling, the output of the first layer network of the encoder is fused. In the embodiment of the present application, the output of the network corresponding to the encoder can also be fused after each layer of upsampling, and the output of the network corresponding to the encoder can also be fused after partial upsampling, and the output of the network corresponding to the encoder is not fused after partial upsampling. The embodiment of the present application is not limited to this. Among them, the last upsampling of the decoder corresponds to the first layer network of the encoder, and the penultimate upsampling of the decoder corresponds to the second layer network of the encoder. Others are similar, and the embodiment of the present application will not be listed one by one. In the processing of the decoder, the more levels of the output of the fusion encoder, the more conducive it is to reduce the probability of image distortion, and the fewer levels of the output of the fusion encoder, the less computing power is required.

[0159] For example, Fig.10 FIG. 1 shows a schematic diagram of a network structure in a decoder. Fig.10 As shown, the decoder network structure can include multiple upsampling processes. Fig.10 In the example shown, the decoder includes four upsampling processes. The processing of the decoder may include:

[0160] 1) Upsample the input information once to obtain Fd1.

[0161] 2) Obtain the output of the last network layer in the encoder. In the embodiment of the present application, Fe1 is used to represent the output of the last network layer in the encoder.

[0162] 3) After fusing Fd1 and Fe1, Fa1 is obtained, Fa1*W1+Fd1=Fm1. The fusing process can be as described above. Figure 8 As shown, no further description is given here.

[0163] 4) Upsample Fm1 three times in sequence to obtain Fd2.

[0164] 5) Obtain the output of the first layer of the network in the encoder. In the embodiment of the present application, Fe2 is used to represent the output of the last layer of the network in the encoder.

[0165] 6) After fusion processing of Fd2 and Fe2, Fa2 is obtained, Fa2*W2+Fd2=Fm2. Fm2 is the processed image. The fusion processing method can be as described above. Figure 8 As shown, no further description is given here.

[0166] W1 and W2 are both matrices, and the dimensions of W1 and W2 are different. The dimension of W1 is the same as the resolution of the image obtained by the first upsampling of the decoder, and the dimension of W2 is the same as the resolution of the image obtained by the last upsampling of the decoder.

[0167] If the resolution of the image obtained by the last upsampling of the decoder is the same as the resolution of the decoder's input image, then W2 is the above weight matrix. If the resolution of the image obtained by the last upsampling of the decoder is smaller than the resolution of the decoder's input image, then W2 can be obtained by transforming the above weight matrix. For details, please refer to the above Fig. 9 , which will not be described in detail here. W1 can be obtained by W2 or by the above weight matrix transformation, which will not be described in detail here.

[0168] Through the above Figures 7 to 10 From the description, it can be known that the image processing method provided by the embodiment of the present application reflects the distortion of the image obtained by decoding the discrete implicit feature base through the rejection rate during semantic prediction. The higher the rejection rate, the higher the distortion, the higher the fusion weight of the feature output by the encoder, and correspondingly, the smaller the fusion weight of the discrete implicit feature base. In this way, it is helpful to reduce the probability of image distortion.

[0169] The embodiment of the present application performs semantic prediction through a discrete vector semantic prediction network, and the training process of the discrete vector semantic prediction network may include the following steps:

[0170] 1) Obtain an image sample, which may include multiple pixels. The image sample may correspond to multiple semantics. In some examples, the semantics in the image sample may include human hair, human eyes, human nose, human eyebrows, human mouth, human arms, human legs, cat eyes, cat nose, cat mouth, dog eyes, dog nose, dog mouth, etc.

[0171] 2) Input the image sample into the encoder to obtain a feature map, which includes multiple features 1, and find the feature closest to each feature in the multiple features 1 in the embedding table to obtain multiple features 2.

[0172] 3) Inputting the plurality of features 2 into the source model to obtain the output of the source model. The output of the source model may include the semantics of each pixel in the plurality of pixels and the rejection rate when predicting the semantics. The source model may be a neural network model, for example, the source model may be a Unet network.

[0173] The semantics of each pixel and the rejection rate when predicting semantics are obtained by the source model in the following way:

[0174] The source model calculates the semantics of each pixel as the prediction probability of each semantics, obtains multiple prediction probabilities, and obtains the prediction probability with the largest probability value from the multiple prediction probabilities. 1 minus the prediction probability with the largest probability value obtains the rejection rate when predicting the semantics. The semantics corresponding to the prediction probability with the largest probability value is the semantics of the pixel.

[0175] For example, the image sample includes 10,000 pixels, corresponding to 10 semantics. Taking a pixel as an example, the source model can calculate the prediction probability of the pixel for these 10 semantics. If the prediction probabilities of these 10 semantics are 0.8, 0.1, 0.05, 0.01, 0.01, 0.01, 0.01, 0.01, 0, and 0 respectively, and the prediction probability with the largest probability value is 0.8, then the semantics corresponding to the prediction probability is the semantics of the pixel. 1-0.8=0.2, then the rejection rate of the pixel when predicting the semantics is 0.2.

[0176] 4) Calculate the loss function of the source model, and obtain the discrete vector semantic prediction network when the loss function meets the preset model convergence condition. The loss function is calculated by the predicted probability and the true probability.

[0177] For example, in the above example, taking a pixel as an example, the predicted probabilities of the 10 semantics are 0.8, 0.1, 0.05, 0.01, 0.01, 0.01, 0.01, 0.01, 0 and 0 respectively. If the true probabilities of the 10 semantics are 1, 0, 0, 0, 0, 0, 0, 0, 0, 0 and 0 respectively. The loss function is related to the 20 probability values. The loss function can be determined by the mean square error, the average mean square error or the cross entropy loss. The 20 probability values ​​are input into the formula for calculating the loss function to obtain the loss function.

[0178] The above combination Figures 7 to 10 The method provided by the embodiment of the present application is introduced from the perspective of a network, and the method provided by the embodiment of the present application is introduced from the perspective of an electronic device.

[0179] For example, Fig.11 FIG. 1 is a schematic flow chart of an image processing method provided by an embodiment of the present application. Fig.11 As shown, the method comprises the following steps:

[0180] S1101. In response to a photographing operation, a first image is acquired, where the first image includes N regions, where N is a positive integer greater than or equal to 1.

[0181] Each of the N regions may include one pixel or multiple pixels, which is not limited in this embodiment of the present application.

[0182] S1102. Obtain N second features corresponding to the N first features one by one from the preset features, the N first features are used to characterize the N regions, each of the N second features is similar to the first feature corresponding to each second feature, wherein the clarity of an image obtained by decoding the N second features is greater than the clarity of an image obtained by decoding the N first features, and the distortion of an image obtained by decoding the N second features is greater than or equal to the distortion of an image obtained by decoding the N first features.

[0183] The preset feature is used to represent the vector in the above embedding space. The vector in the above embedding space is obtained during the model training process and preset in the model. The preset feature is only an example of a name, and the embodiment of the present application does not limit this.

[0184] The N first features are used to characterize the N regions, and it can be understood that the N first features are obtained after extracting features from the N regions. In an example, the N first features can be extracted from the first image by an encoder.

[0185] Each of the N second features is similar to the first feature corresponding to it. It can be understood that each second feature and the first feature corresponding to it belong to the same category, or in other words, each second feature and the first feature corresponding to it can be clustered into one category.

[0186] The distortion of the image obtained by decoding N second features is greater than or equal to the distortion of the image obtained by decoding N first features. It can also be described as: the fidelity of the image obtained by decoding N second features is less than or equal to the fidelity of the image obtained by decoding N first features. The embodiment of the present application does not limit this.

[0187] In the above example, N second features corresponding to the N first features are obtained from the preset features, and the above discrete feature matching can be referred to. e (x), N second features can refer to the above z q (x).

[0188] S1103. Fusing N first features and N second features to generate a second image, wherein the fusion weight of any first feature among the N first features is positively correlated with the distortion of an image obtained by decoding a second feature corresponding to any first feature; and the fusion weight of any second feature among the N second features is negatively correlated with the distortion of an image obtained by decoding any second feature.

[0189] When the distortion of the image obtained by decoding the second feature corresponding to any one of the N first features is large, the fusion weight of any one of the first features is large, and accordingly, the fusion weight of the second feature corresponding to any one of the first features is smaller. Since the fidelity of any one of the N first features is greater than the second feature corresponding to any one of the first features, it is beneficial to reduce the probability of distortion of the second image.

[0190] Optionally, before fusing the N first features and the N second features, the method further includes: inputting the N second features into a semantic prediction model to obtain the semantics of each of the N regions and a rejection rate of each region when predicting the semantics, the rejection rate being positively correlated with the distortion of the image obtained by decoding each second feature, and the rejection rate being positively correlated with the fusion weight of the first feature corresponding to each second feature.

[0191] The semantic prediction model can refer to the above-mentioned discrete vector semantic prediction model. The model name is just an example and is not limited to this in the embodiments of the present application.

[0192] If each of the N regions includes a pixel, the semantics of each pixel and the rejection rate of each pixel when predicting the semantics can be obtained through the semantic prediction model.

[0193] The rejection rate is positively correlated with the distortion of the image obtained by decoding each second feature. The higher the rejection rate, the greater the distortion.

[0194] In this way, by identifying semantics and rejection rate through the semantic prediction model, the implicitly expressed second feature can be visualized through semantics, and the fusion weight of the first feature can be affected by the rejection rate. The generation of the second image can be indirectly guided by the rejection rate, thus realizing semantic-guided implicitization.

[0195] Optionally, the semantic prediction model is trained in the following manner: obtaining M third features, the M third features are used to characterize M regions in an image sample, the image sample includes multiple semantics, and M is a positive integer greater than 1; the M third features are input into a source model to obtain the semantics of each region in the M regions and the rejection rate of each region when predicting the semantics, wherein the source model is used to: calculate the prediction probability that the semantics of the i-th region in the M regions is each of the multiple semantics, obtain multiple prediction probabilities, and obtain a target probability that meets preset conditions from the multiple prediction probabilities, and based on the anti-correlation relationship between the target probability, the prediction probability and the rejection rate, calculate the rejection rate of the i-th region in the M regions when predicting the semantics, output the semantics corresponding to the target probability and the rejection rate of the i-th region in the M regions when predicting the semantics; wherein i is an integer traversing from 1 to M; when the loss function satisfies the preset model convergence condition, a semantic prediction model is obtained, and the loss function is related to the difference between the predicted probability and the true probability.

[0196] Each of the M regions may include one pixel or multiple pixels. The M third features may refer to the above-mentioned multiple features 2. The preset condition may be the maximum prediction probability, and the target probability is the maximum prediction probability among the multiple prediction probabilities.

[0197] The anti-correlation relationship between the prediction probability and the rejection probability can be: the sum of the prediction probability and the rejection probability is 1. In this way, the rejection rate of the i-th region in the M regions when predicting semantics and the prediction probability of the i-th region in the M regions when predicting semantics are summed to 1.

[0198] The training process of the semantic prediction model can refer to the training process of the discrete vector semantic prediction network mentioned above, which will not be repeated here.

[0199] This helps to enable the semantic prediction model to predict semantics and output the rejection rate when predicting semantics.

[0200] Optionally, when the rejection rate of any one of the N regions in predicting semantics is less than or equal to the first value and greater than or equal to the second value, the fusion weight of any one of the N first features is the first weight; when the rejection rate of any one of the N regions in predicting semantics is greater than or equal to the third value and less than the second value, the fusion weight of any one of the N first features is the second weight; wherein the second value is less than the first value, the third value is less than the second value, and the second weight is less than the first weight. In one example, the first value is 1, the second value can refer to the above T1, and the first weight can refer to the above a1. The third value can refer to the above T2, and the second weight can refer to the above a2.

[0201] In another example, the first value may refer to the above T1, the second value may refer to the above T2, the first weight may refer to the above a2, the third value may be 0, and the second weight may refer to the above a3.

[0202] When the rejection rate is within a certain range, the weight is the same; when the rejection rate is within different ranges, the weight is different. This is conducive to flexibly adjusting the weight according to the rejection rate.

[0203] Optionally, the electronic device includes an encoding module and a decoding module; N first features are extracted from a first image by the encoding module, the encoding module includes L layers of downsampling, the first layer of downsampling in the L layers of downsampling is used to extract N fourth features for representing N regions from the first image, the resolution of the image obtained by decoding the N fourth features is greater than the resolution of the image obtained by decoding the N first features, and L is a positive integer greater than or equal to 1; the N first features and the N second features are fused to generate a second image, including: after the decoding module decodes the N second features, the N fourth features are fused to generate the second image.

[0204] If the encoding module includes 1 layer of downsampling, the N fourth features are the same as the N first features. The resolution of the image obtained by decoding the N fourth features is the same as the resolution of the image obtained by decoding the N first features.

[0205] If the encoding module includes multi-layer downsampling, the N fourth features are different from the N first features. The resolution of an image obtained by decoding the N fourth features is greater than the resolution of an image obtained by decoding the N first features.

[0206] The resolution of the image obtained by decoding the N second features is the same as the resolution of the image obtained by decoding the N fourth features. Therefore, after the decoding module decodes the N features, the N fourth features are fused to generate the second image.

[0207] In this way, the outputs of layers with the same resolution are fused to facilitate the generation of the second image.

[0208] Optionally, the N fourth features are represented by Fe, the output after decoding the N second features is represented by Fd, and the second image is represented by Fm, then the second image satisfies the following formula:

[0209] Fe*W+Fd=Fm

[0210] Among them, W is used to represent the matrix composed of the fusion weights of any fourth feature among the N fourth features, and the matrix composed of the fusion weights of any first feature among the N first features is obtained by transforming W.

[0211] If the N fourth features are the same as the N first features, the dimension of W is the same as the dimension of the matrix formed by the fusion weights of any one of the N first features, and W is the matrix formed by the fusion weights of any one of the N first features.

[0212] If the N fourth features are different from the N first features, the dimension of W is greater than the dimension of the matrix formed by the fusion weights of any of the N first features, and the matrix formed by the fusion weights of any of the N first features can be transformed by W. The changes between matrices can refer to the above Fig. 9 , I will not go into details here.

[0213] In this way, when the distortion rate is larger, the value of the weight is larger, the elements in W are larger, the role of Fe is larger, and correspondingly, the role of Fd is smaller, which is beneficial to reducing the distortion rate of the image.

[0214] Optionally, after the decoding module decodes the N second features and before fusing the N fourth features, the method also includes: performing feature parallel connection on the output after decoding the N second features and the N fourth features; inputting the output result after the feature parallel connection into the residual dense block RRDB to obtain the output of RRDB; convolving the output of RRDB to obtain the output result; after the decoding module decodes the N second features, the N fourth features are fused, including: after the decoding module decodes the N second features, the output result is fused.

[0215] This method can refer to the above Figure 8 In this way, the output of the encoder can be successfully applied to the input of the decoder, reducing the probability of distortion.

[0216] Optionally, obtaining N second features corresponding one-to-one to the N first features in the preset features includes: finding a feature having the smallest Euclidean distance with each of the N first features in the preset features to obtain the N second features.

[0217] The specific implementation of this method can refer to the above formula, which will not be described here. In this way, it is helpful to find features that can decode a high-definition image and improve the clarity of the second image.

[0218] Optionally, in response to a photographing operation, obtaining a first image includes: in response to the photographing operation, obtaining multiple frames of original images; calculating a registration matrix and / or AE parameters based on the multiple frames of original images, the registration matrix being a matrix used to represent a geometric transformation relationship between the multiple frames of original images, the AE parameters including at least one of exposure time, brightness or sensitivity of the multiple frames of original images; and fusing the multiple frames of original images based on the registration matrix and / or AE parameters to obtain a first image.

[0219] This method can refer to the above Figure 6 In this way, fusing multiple frames of images is beneficial to reducing the noise of the first image, reducing the phenomenon of motion blur, and thus improving the quality of the second image.

[0220] Optionally, after fusing the N first features and the N second features to generate a second image, the method further includes: processing the brightness of the second image by gamma correction, global tone mapping GTM, sharpening, or at least one of the first curves, wherein the first curve is used to represent the correspondence between brightness and contrast.

[0221] This method can refer to the above Figure 6 This is helpful to adjust the brightness of the second image to better meet the needs.

[0222] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.

[0223] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0224] The image processing method of the embodiment of the present application has been described above, and the device for executing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above image processing method.

[0225] Fig.12 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Fig.12 As shown, the chip 120 includes one or more (including two) processors 1201 , a communication line 1202 , a communication interface 1203 , and a memory 1204 .

[0226] In some implementations, the memory 1204 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0227] The image processing method described in the above embodiment of the present application can be applied to the processor 1201, or implemented by the processor 1201. The processor 1201 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above image processing method can be completed by an integrated logic circuit of hardware or software instructions in the processor 1201. The above processor 1201 can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1201 can implement or execute the methods, steps and logic block diagrams related to each processing disclosed in the embodiment of the present application.

[0228] The steps of the image processing method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 1204, and the processor 1201 reads the information in the memory 1204 and completes the steps of the above method in combination with its hardware.

[0229] The processor 1201 , the memory 1204 , and the communication interface 1203 may communicate with each other via the communication line 1202 .

[0230] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0231] The image processing method provided in the embodiment of the present application can be applied to an electronic device with a camera function. The electronic device includes a terminal device, and the specific device form of the terminal device can refer to the above related description, which will not be repeated here.

[0232] An embodiment of the present application provides a terminal device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the terminal device to execute the above method.

[0233] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program or instruction. The above method is implemented when the computer program or instruction is executed by an electronic device. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0234] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0235] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program code runs on an electronic device, the electronic device executes the above method.

[0236] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0237] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: Used in electronic equipment, including: In response to a photographing operation, acquiring a first image, wherein the first image includes N regions, where N is a positive integer greater than or equal to 1; Obtaining N second features corresponding to the N first features one by one from the preset features, the N first features are used to characterize the N regions, each of the N second features is similar to the first feature corresponding to each second feature, wherein the clarity of an image obtained by decoding the N second features is greater than the clarity of an image obtained by decoding the N first features, and the distortion of an image obtained by decoding the N second features is greater than or equal to the distortion of an image obtained by decoding the N first features; The N first features and the N second features are fused to generate a second image, wherein the fusion weight of any one of the N first features is positively correlated with the distortion of the image obtained by decoding the second feature corresponding to the any one of the first features; the fusion weight of any one of the N second features is negatively correlated with the distortion of the image obtained by decoding the any one of the second features.

2. The method according to claim 1, characterized in that Before fusing the N first features and the N second features, the method further includes: The N second features are input into the semantic prediction model to obtain the semantics of each of the N regions and the rejection rate of each region when predicting the semantics, the rejection rate is positively correlated with the distortion of the image obtained by decoding each second feature, and the rejection rate is positively correlated with the fusion weight of the first feature corresponding to each second feature.

3. The method according to claim 2, characterized in that The semantic prediction model is trained in the following way: Obtaining M third features, where the M third features are used to characterize M regions in an image sample, where the image sample includes multiple semantics, and M is a positive integer greater than 1; The M third features are input into the source model to obtain the semantics of each region in the M regions and the rejection rate of each region when predicting the semantics, wherein the source model is used to: calculate the prediction probability that the semantics of the i-th region in the M regions is each of the multiple semantics to obtain multiple prediction probabilities, and obtain a target probability that meets a preset condition from the multiple prediction probabilities, and calculate the rejection rate of the i-th region in the M regions when predicting the semantics based on the anti-correlation relationship among the target probability, the prediction probability and the rejection rate, and output the semantics corresponding to the target probability and the rejection rate of the i-th region in the M regions when predicting the semantics; wherein i is an integer traversing from 1 to M; When the loss function satisfies a preset model convergence condition, the semantic prediction model is obtained, and the loss function is related to the difference between the predicted probability and the true probability.

4. The method according to claim 2 or 3, characterized in that: The rejection rate of any one of the N regions when predicting semantics is less than or equal to the first value and greater than or equal to the second value, and the fusion weight of any one of the N first features is the first weight; When the rejection rate of any one of the N regions in predicting semantics is greater than or equal to the third value and less than the second value, the fusion weight of any one of the N first features is the second weight; Among them, the second value is smaller than the first value, the third value is smaller than the second value, and the second weight is smaller than the first weight.

5. The method according to any one of claims 1 to 4, characterized in that The electronic device includes an encoding module and a decoding module; the N first features are extracted from the first image by the encoding module, the encoding module includes L layers of downsampling, a first layer of downsampling in the L layers of downsampling is used to extract N fourth features for representing the N regions from the first image, a resolution of an image obtained by decoding the N fourth features is greater than a resolution of an image obtained by decoding the N first features, and L is a positive integer greater than or equal to 1; The fusing the N first features and the N second features to generate a second image includes: After decoding the N second features, the decoding module fuses the N fourth features to generate the second image.

6. The method according to claim 5, characterized in that The N fourth features are represented by Fe, the output after decoding the N second features is represented by Fd, and the second image is represented by Fm. Then the second image satisfies the following formula: Fe*W+Fd=Fm Among them, W is used to represent the matrix composed of the fusion weights of any fourth feature among the N fourth features, and the matrix composed of the fusion weights of any first feature among the N first features is obtained by transforming W.

7. The method according to claim 5 or 6, characterized in that: After the decoding module decodes the N second features and before fusing the N fourth features, the method further includes: Performing feature parallel connection on outputs after decoding the N second features and the N fourth features; Input the output result after the feature parallel connection into the residual dense block RRDB to obtain the output of the RRDB; Convolving the output of the RRDB to obtain an output result; After decoding the N second features, the decoding module fuses the N fourth features, including: After decoding the N second features, the decoding module fuses the output results.

8. The method according to any one of claims 1 to 7, characterized in that The obtaining N second features corresponding to the N first features in a one-to-one manner from the preset features includes: A feature having the smallest Euclidean distance to each of the N first features is found among the preset features to obtain the N second features.

9. The method according to any one of claims 1 to 8, characterized in that The step of acquiring a first image in response to a photographing operation includes: In response to a photographing operation, acquiring multiple frames of original images; Calculate a registration matrix and / or AE parameters according to the multiple frames of original images, where the registration matrix is ​​a matrix used to represent a geometric transformation relationship between the multiple frames of original images, and the AE parameters include at least one of exposure time, brightness, or sensitivity of the multiple frames of original images; The multiple frames of original images are fused based on the registration matrix and / or the AE parameter to obtain the first image.

10. The method according to any one of claims 1 to 9, characterized in that After fusing the N first features and the N second features to generate a second image, the method further includes: The brightness of the second image is processed by at least one of gamma correction, global tone mapping (GTM), sharpening, or a first curve, wherein the first curve is used to represent a corresponding relationship between brightness and contrast.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1 to 10.

12. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 11.

Images