Image processing method and device, equipment, medium, product and vehicle

Abstract

The invention discloses an image processing method and device, equipment, a medium, a product and a vehicle, and relates to the technical field of image processing. The image processing method comprises the following steps: acquiring a first image acquired by an infrared camera; performing super-division processing on the first image by using an image super-division model to obtain a second image, the image super-division model being obtained by training a plurality of high-definition images and a plurality of fuzzy degraded images corresponding to each high-definition image, the plurality of fuzzy degradation images corresponding to each high-definition image are generated through a plurality of image fuzzy degradation models, and the plurality of image fuzzy degradation models have different influence factor sequences and / or different influence modes; and performing gray processing on the second image to obtain a target image. Through the scheme disclosed by the invention, the image super-resolution effect can be improved, and the definition effect of the image shot by the infrared camera is further improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an image processing method, apparatus, device, medium, product and vehicle. Background Art

[0002] Security technology is a perfect combination of electronic technology, communication technology, control technology and computer technology. The security system constructed by security technology can provide a safe, convenient and comfortable protected space for building a safe and harmonious society and protecting the safety of people and property.

[0003] Infrared (IR) cameras are often used for nighttime security. However, when an object is far away, the clarity of the image captured by the IR camera is poor.

[0004] In the related art, in order to improve the clarity of images captured by an IR camera, the images captured by the infrared camera are usually input into an image super-resolution model, and the image super-resolution model outputs images with higher clarity. However, the image super-resolution model is trained using multiple high-definition images and a blurred degraded image corresponding to each high-definition image. The multiple blurred degraded images are all generated by the same image blurred degradation model. This causes the image super-resolution model to only be able to super-resolve the images captured by the infrared camera from a certain specific dimension, resulting in poor image super-resolution effect, and thus poor effect in improving the clarity of the images captured by the IR camera. Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device, medium, product and vehicle, which can solve the problem of poor image super-resolution effect and poor clarity improvement of images captured by IR cameras.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0007] Acquire a first image captured by an infrared camera;

[0008] Super-resolution processing is performed on the first image using an image super-resolution model to obtain a second image, wherein the image super-resolution model is trained using multiple high-definition images and multiple blurred and degraded images corresponding to each high-definition image, and the multiple blurred and degraded images corresponding to each high-definition image are generated using multiple image blur degradation models, and the multiple image blur degradation models have different influencing factor orders and / or different influencing modes;

[0009] Grayscale processing is performed on the second image to obtain a target image.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, comprising:

[0011] An acquisition module, configured to acquire a first image captured by an infrared camera;

[0012] a super-resolution module, configured to super-resolve the first image using an image super-resolution model to obtain a second image, wherein the image super-resolution model is trained using multiple high-definition images and multiple blurred and degraded images corresponding to each high-definition image, the multiple blurred and degraded images corresponding to each high-definition image are generated using multiple image blur degradation models, and the multiple image blur degradation models have different influencing factor orders and / or different influencing modes;

[0013] The grayscale processing module is used to perform grayscale processing on the second image to obtain a target image.

[0014] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor and a memory storing computer program instructions; when the processor reads and executes the computer program instructions, the steps of the image processing method provided in the first aspect of the embodiment of the present application are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the steps of the image processing method provided in the first aspect of the embodiment of the present application are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the steps of the image processing method provided in the first aspect of the embodiment of the present application are implemented.

[0017] In a sixth aspect, an embodiment of the present application provides a vehicle, comprising at least one of the following items:

[0018] The image processing device provided by the second aspect of the embodiment of the present application;

[0019] The electronic device provided by the third aspect of the embodiment of the present application;

[0020] The fourth aspect of the embodiment of the present application provides a computer-readable storage medium.

[0021] In an embodiment of the present application, a first image captured by an infrared camera is super-resolved using an image super-resolution model trained using multiple high-definition images and multiple blurred degraded images corresponding to each high-definition image to obtain a second image; and the second image is grayscale processed to obtain a target image. Since the image super-resolution model is trained using multiple high-definition images and multiple blurred degraded images corresponding to each high-definition image, the multiple blurred degraded images corresponding to each high-definition image are generated using multiple image blur degradation models, and the multiple image blur degradation models have different influencing factor orders and / or different influencing modes, the trained image super-resolution model can super-resolve images from multiple dimensions corresponding to different influencing factor orders and / or different influencing modes, thereby improving the image super-resolution effect and, in turn, improving the clarity of images captured by the infrared camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 Schematic diagram of the image processing method provided in the embodiment of the present application;

[0024] Figure 2 This is a schematic diagram of the structure of the RRDB module provided in an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a first image provided in an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a target image provided in an embodiment of the present application;

[0027] Figure 5 This is a schematic diagram of the structure of the image processing provided by the embodiment of the present application;

[0028] Figure 6 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application can also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present application, not all of the embodiments.

[0031] The image processing method, apparatus, device, medium, product, and vehicle provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0032] In some possible implementations of the embodiments of the present application, the image processing method provided by the embodiments of the present application can be applied to electronic devices.

[0033] Figure 1 It is a flowchart of the image processing method provided in an embodiment of the present application.

[0034] In some possible implementations of the embodiments of the present application, the image processing method provided in the embodiments of the present application can be applied to an electronic device. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiments of the present application are not specifically limited.

[0035] like Figure 1 As shown, the image processing method may include:

[0036] Step 101: Acquire a first image captured by an infrared camera;

[0037] Step 102: Super-resolution the first image using an image super-resolution model to obtain a second image, wherein the image super-resolution model is trained using multiple high-definition images and multiple blurred and degraded images corresponding to each high-definition image, and the multiple blurred and degraded images corresponding to each high-definition image are generated using multiple image blur degradation models, and the multiple image blur degradation models have different influencing factor orders and / or different influencing modes;

[0038] In some possible implementations of the embodiments of the present application, the high-definition image in the embodiments of the present application refers to an image with a resolution higher than a resolution threshold, which is high in clarity and good in quality as perceived by humans; a blurred and degraded image refers to an image whose quality has deteriorated due to certain factors or processing relative to the original high-definition image, for example, reduced resolution, increased noise, etc.; super-resolution processing is the process of upgrading a low-resolution image to a high-resolution image.

[0039] In some possible implementations of the embodiments of the present application, in step 102, the first image may be input into an image super-resolution model, and the image super-resolution model outputs a second image obtained by super-resolving the first image.

[0040] In some possible implementations of the embodiments of the present application, the image super-resolution model provided by the embodiments of the present application may be an image super-resolution model based on a generative adversarial network (GAN). GAN consists of a generator and a discriminator. The generator randomly samples from the latent space as input, and its output needs to imitate the real samples in the training set as much as possible. The input of the discriminator is a real sample or the output of the generator, and the purpose is to distinguish the output of the generator from the real sample as much as possible. The generator must deceive the discriminator as much as possible. The two models compete with each other and continuously adjust parameters. The ultimate goal is to make it impossible for the discriminator to determine whether the output of the generator is real.

[0041] In some possible implementations of the embodiments of the present application, when training an image super-resolution model, each high-definition image among a plurality of high-definition images and a plurality of blurred degraded images corresponding to the high-definition image can be formed into an image pair to obtain a plurality of image pairs corresponding to the high-definition image, and then a predicted image corresponding to the first blurred degraded image is generated by a GAN generator, wherein the resolution of the predicted image is greater than the resolution of the first blurred degraded image, and the first blurred degraded image is an image among the plurality of blurred degraded images corresponding to the plurality of high-definition images; a discrimination result of whether the predicted image and the high-definition image corresponding to the first blurred degraded image are real images is generated by a GAN discriminator; the predicted image, the high-definition image corresponding to the first blurred degraded image, and the discrimination result are input into the loss function of the generator to obtain the loss value of the generator; the discrimination result is input into the loss function of the discriminator to obtain the loss value of the discriminator; the parameters of the generative adversarial network are updated according to the loss value of the generator and the loss value of the discriminator to obtain the image super-resolution model.

[0042] In some possible implementations of the embodiments of the present application, the predicted image in the embodiments of the present application is a high-resolution image obtained from a low-resolution image through super-resolution processing, also called a reconstructed image.

[0043] In some possible implementations of the embodiments of the present application, the first blurred and degraded image may be input into a generator of a GAN, and the generator of the GAN generates a predicted image corresponding to the first blurred and degraded image.

[0044] In some possible implementations of the embodiments of the present application, a single discriminator can output a probability value of whether the predicted image and the high-definition image corresponding to the first blurred and degraded image are real images, and the probability value corresponds to the discrimination result. Alternatively, a dual discriminator structure (including a local discriminator and a global discriminator) can be used to capture the continuity of the image texture and the global characteristics of the image, and output a matrix in which each element corresponds to a receptive field of the image. Finally, the average value of each element is taken to represent the probability that the final predicted image is the high-definition image corresponding to the first blurred and degraded image.

[0045] In some possible implementations of the embodiments of the present application, the loss function of the generator and the loss function of the discriminator may adopt a perceptual loss function, an adversarial loss function, and the like.

[0046] In some possible implementations of the embodiments of the present application, the gradient of the loss function of the generator can be calculated through backpropagation and optimization algorithms, and the parameters of the generator are updated according to the gradient direction to minimize the generator loss value. Similarly, the parameters of the discriminator are updated to minimize the discriminator loss value. During the training process, the parameters of the generator and the discriminator are cross-updated, so that the generator can eventually generate realistic predicted images, and the discriminator can more accurately distinguish between the predicted images and the high-definition images corresponding to the blurred and degraded images.

[0047] In some possible implementations of the embodiments of the present application, the image super-resolution model provided by the embodiments of the present application may be an image super-resolution model based on enhanced super-resolution generative adversarial networks (ESRGAN).

[0048] ESRGAN uses the Residual-in-Residual Dense Block (RRDB) module without the Batch Normalizing layer. Figure 2 This is a schematic diagram of the structure of the RRDB module provided in the embodiment of the present application. Figure 2 In

[15] , the RRDB module includes three dense blocks, each of which is densely connected by five convolutional layers (conv), each of which includes a hidden layer neuron output, and β is a parameter. In some possible implementations of the embodiments of the present application, the activation function of the hidden layer neuron output can be an LReLU function, a PReLU function, or an RReLU function.

[0049] The discriminator of ESRGAN is shown in the following formula (1):

[0050]

[0051] In formula (1), x r is a real picture, x f To predict the image, the function C(·) represents the original output of the discriminator, and the function E x (·) represents the expected distribution of the predicted image. Function σ(·) represents the sigmoid function. Formula (1) essentially calculates the probability that one image is true relative to another image.

[0052] Based on the discriminator shown in formula (1) above, the discriminator loss of ESRGAN is shown in formula (2) below:

[0053]

[0054] The generator loss of ESRGAN is shown in the following formula (3):

[0055]

[0056] In formulas (2) and 3, For real picture x r The expected value of To predict the image x fThe expected value of D Ra (x r ,x f ) is the output of the discriminator, representing the real picture x r and predicted image x f The discriminator loss and the generator loss both consist of two parts, one of which is D Ra (x r ,x f ), output 0 as much as possible during the training process, so that the discriminator cannot distinguish the real picture x r and predicted image x f , the other part is D Ra (x f ,x r ), output 1 as much as possible during the training process so that the discriminator can distinguish the predicted image x f With real picture x r Therefore, the generator needs to generate predicted images that are as close to the real images as possible to deceive the discriminator and minimize the loss value.

[0057] In some possible implementations of the embodiments of the present application, when generating multiple blurred degraded images corresponding to each high-definition image, the first high-definition image can be input into multiple image blur degradation models respectively, and the multiple image blur degradation models generate multiple blurred degraded images corresponding to the first high-definition image, wherein the first high-definition image is any one of the multiple high-definition images, and the image blur degradation model includes three influencing factors: blur kernel, downsampling and noise, and the same influencing factor has different influencing modes.

[0058] It can be understood that each image blur degradation model generates a blur degraded image corresponding to the first high-definition image.

[0059] In some possible implementations of the embodiment of the present application, the image blur degradation model in the embodiment of the present application is shown in the following formula (4):

[0060]

[0061] In formula (4), Y is the blurred degraded image, X is the original image, Represents convolution, A, B and C represent three influencing factors: blur kernel K, downsampling S and noise N.

[0062] When A in an image blur degradation model is the blur kernel K, B is the downsampling S, and C is the noise N (i.e., KSN order), the image blur degradation model is: When A is downsampling S, B is blur kernel K, and C is noise N (i.e., SKN order) in an image blur degradation model, the image blur degradation model is When A is the blur kernel K, B is the noise N, and C is the downsampling S (i.e., KNS order) in an image blur degradation model, the image blur degradation model is The embodiment of the present application does not list the image blur degradation models of blur kernel K, downsampling S, and noise N in different orders one by one. Other forms of image blur degradation models can refer to the image blur degradation model described above.

[0063] In some possible implementations of the embodiments of the present application, the same influencing factor may have different influencing modes. For example, the blur kernel K may include Gaussian blur kernel, box blur kernel, dual blur kernel, bokeh blur kernel, tilt-shift blur kernel, and aperture blur kernel. Exemplarily, the order of the three influencing factors in the two image blur degradation models is the same, with the blur kernel in one image blur degradation model being a Gaussian blur kernel and the blur kernel in the other image blur degradation model being a box blur kernel. For downsampling S, the order of the three influencing factors in the two image blur degradation models is the same, with the downsampling in one image blur degradation model being bicubic downsampling and the downsampling in the other image blur degradation model being nearest neighbor downsampling. For noise N, it may be random noise. Exemplarily, the order of the three influencing factors in the two image blur degradation models is the same, with the noise in one image blur degradation model being N1 and the noise in the other image blur degradation model being N2.

[0064] The following uses the Gaussian blur kernel as an example to illustrate the blur kernel. A square (for example, 5*5) pixel array can be defined as the Gaussian blur kernel. Each pixel in the image is multiplied by the Gaussian blur kernel to obtain the blur value of each pixel. All the blur values ​​are added together to obtain the blur value of the image.

[0065] The following uses nearest neighbor downsampling as an example to illustrate downsampling. First, the image pixels are extracted in rows and columns. For a specified width and height, the pixels are extracted with a certain compensation. The extracted pixels are retained, and pixels with similar colors of adjacent pixels are merged to obtain new pixels. The embodiments of the present application do not limit the method used to determine whether the colors are similar. Any available method can be applied to the embodiments of the present application, for example, taking the difference of pixel values ​​and comparing the difference with a threshold value, or calculating the similarity of pixel values.

[0066] When a certain image is input into multiple image blur degradation models with different influencing factor orders and / or different influencing modes, the multiple image blur degradation models each output a blur degradation picture corresponding to the image, thereby obtaining multiple blur degradation pictures corresponding to the image. For example, image A is input into N image blur degradation models respectively, and the multiple image blur degradation models each output blur degradation pictures A1 to A2 corresponding to image A. N .

[0067] In one possible implementation of an embodiment of the present application, when training an image super-resolution model, an image can be composed of image pairs with multiple blurred degraded pictures corresponding to the image to obtain multiple image pairs, and the image super-resolution model can be trained using the multiple image pairs corresponding to the multiple images.

[0068] In the embodiment of the present application, a large amount of data for training the image super-resolution model can be obtained, which can improve the accuracy of the image super-resolution model.

[0069] Step 103: grayscale processing is performed on the second image to obtain a target image.

[0070] In some possible implementations of the embodiments of the present application, step 103 may include: calculating the cumulative distribution corresponding to each grayscale value of multiple grayscale values ​​corresponding to the second image; determining a first grayscale threshold and a second grayscale threshold based on the cumulative distribution, wherein the second grayscale threshold is greater than the first grayscale threshold; keeping the grayscale values ​​of pixels in the second image whose grayscale values ​​are greater than or equal to the first grayscale threshold and less than or equal to the second grayscale threshold unchanged; adjusting the grayscale values ​​of pixels in the second image whose grayscale values ​​are less than the first grayscale threshold to the first grayscale value; adjusting the grayscale values ​​of pixels in the second image whose grayscale values ​​are greater than the second grayscale threshold to the second grayscale value, to obtain a target image, wherein the second grayscale value is greater than the first grayscale value.

[0071] In some possible implementations of the embodiments of the present application, the cumulative distribution corresponding to a grayscale value may be the sum of the probabilities of occurrence of grayscale values ​​less than and equal to the grayscale value. For example, if the probability of occurrence of grayscale value 0 is 2%, then the cumulative distribution corresponding to grayscale value 0 is 2%. If the probability of occurrence of grayscale value 1 is 0.5%, then the cumulative distribution corresponding to grayscale value 1 is the sum of the probabilities of occurrence of grayscale value 0 and the probability of occurrence of grayscale value 1, that is, the cumulative distribution corresponding to grayscale value 1 = 2% + 0.5% = 2.5%.

[0072] The embodiments of the present application do not limit the method used to calculate the cumulative distribution corresponding to each of the multiple grayscale values ​​corresponding to the second image. For example, a grayscale histogram corresponding to the second image may be calculated, and the cumulative distribution corresponding to each grayscale value may be calculated based on the grayscale histogram. The grayscale histogram represents the number of pixels of each grayscale value in the image.

[0073] In some possible implementations of the embodiments of the present application, when determining the first grayscale threshold based on the cumulative distribution, the first target grayscale value used to determine the first grayscale threshold can be assigned to zero; determine whether the first cumulative distribution corresponding to the first target grayscale value is greater than or equal to the first cumulative distribution threshold; when the first cumulative distribution is less than the first cumulative distribution threshold, the first target grayscale value is assigned to the sum of the first target grayscale value and 1, and the step of determining whether the first cumulative distribution corresponding to the first target grayscale value is greater than or equal to the first cumulative distribution threshold is continued; when the first cumulative distribution is greater than or equal to the first cumulative distribution threshold, the first target grayscale value is used as the first grayscale threshold.

[0074] In some possible implementations of the embodiments of the present application, the first cumulative distribution threshold can be set according to actual needs.

[0075] For example, first determine whether the cumulative distribution corresponding to the grayscale value 0 is greater than or equal to the first cumulative distribution threshold. When the cumulative distribution corresponding to the grayscale value 0 is less than the first cumulative distribution threshold, determine whether the cumulative distribution corresponding to the grayscale value 1 is greater than or equal to the first cumulative distribution threshold. When the cumulative distribution corresponding to the grayscale value 1 is less than the first cumulative distribution threshold, determine whether the cumulative distribution corresponding to the grayscale value 2 is greater than or equal to the first cumulative distribution threshold. This cycle repeats. Assuming that the cumulative distribution corresponding to the grayscale value 25 is greater than or equal to the first cumulative distribution threshold, the grayscale value 25 is used as the first grayscale threshold. The first grayscale threshold can be called the grayscale lower boundary.

[0076] In some possible implementations of the embodiments of the present application, when determining the second grayscale threshold based on the cumulative distribution, the second target grayscale value used to determine the second grayscale threshold can be determined based on the number of grayscale values ​​in the second image; it is determined whether the second cumulative distribution corresponding to the second target grayscale value is greater than or equal to the second cumulative distribution threshold; when the second cumulative distribution is greater than or equal to the second cumulative distribution threshold, the second target grayscale value is assigned the difference between the second target grayscale value and 1, and the step of determining whether the second cumulative distribution corresponding to the second target grayscale value is greater than or equal to the second cumulative distribution threshold is continued; when the second cumulative distribution is less than the second cumulative distribution threshold, the second target grayscale value is used as the second grayscale threshold.

[0077] In some possible implementations of the embodiments of the present application, the second cumulative distribution threshold can be set according to actual needs.

[0078] Exemplarily, the grayscale values ​​corresponding to the second image include 0-155, 158-234, and 236-255, that is, the number of grayscale values ​​in the second image is 253. First, it is determined whether the cumulative distribution corresponding to the grayscale value 253 is greater than or equal to the second cumulative distribution threshold. When the cumulative distribution corresponding to the grayscale value 253 is greater than or equal to the second cumulative distribution threshold, it is determined whether the cumulative distribution corresponding to the grayscale value 252 is greater than or equal to the second cumulative distribution threshold. When the cumulative distribution corresponding to the grayscale value 252 is greater than or equal to the second cumulative distribution threshold, it is determined whether the cumulative distribution corresponding to the grayscale value 251 is greater than or equal to the second cumulative distribution threshold. This cycle is repeated. Assuming that the cumulative distribution corresponding to the grayscale value 245 is determined to be less than the second cumulative distribution threshold, the grayscale value 245 is used as the second grayscale threshold. The second grayscale threshold can be referred to as the grayscale lower boundary.

[0079] In the embodiment of the present application, by adjusting the grayscale values ​​of the pixels in the second image, it is possible to facilitate the distinction of objects in the image.

[0080] In some possible implementations of the embodiments of the present application, the above-mentioned first grayscale value and second grayscale value can be set according to actual needs, for example, the first grayscale value is set to 0 and the second grayscale value is set to 255, where the grayscale value 0 is black and the grayscale value 255 is white.

[0081] For example, taking the first grayscale threshold as 25 and the second grayscale threshold as 245 as an example, the grayscale values ​​of pixels in the second image with grayscale values ​​less than 25 are adjusted to 0, and the grayscale values ​​of pixels in the second image with grayscale values ​​greater than 245 are adjusted to 255.

[0082] In an embodiment of the present application, by adjusting the grayscale values ​​of pixels in the second image whose grayscale values ​​are less than a first grayscale threshold to the first grayscale value, and adjusting the grayscale values ​​of pixels in the second image whose grayscale values ​​are greater than the second grayscale threshold to the second grayscale value, objects in the image can be easily distinguished. When the grayscale values ​​of pixels in the second image whose grayscale values ​​are less than the first grayscale threshold are adjusted to 0, and the grayscale values ​​of pixels in the second image whose grayscale values ​​are greater than the second grayscale threshold are adjusted to 255, the pixels at the edges of objects in the image can be adjusted to "either black or white or both black or white," making it easier to distinguish objects in the image.

[0083] For example, Figure 3 and Figure 4 As shown, Figure 3 is a schematic diagram of a first image provided in an embodiment of the present application, Figure 4 Schematic diagram of the target image provided in the embodiment of the present application. Figure 3 shows the original image captured by the infrared camera, Figure 4The target image is obtained by super-resolution and grayscale processing of the original image captured by the infrared camera. Figure 3 and Figure 4 It can be seen that Figure 4 The target image shown is Figure 3 The original image shown is clear.

[0084] In an embodiment of the present application, a first image captured by an infrared camera is super-resolved using an image super-resolution model trained using multiple high-definition images and multiple blurred degraded images corresponding to each high-definition image to obtain a second image; and the second image is grayscale processed to obtain a target image. Since the image super-resolution model is trained using multiple high-definition images and multiple blurred degraded images corresponding to each high-definition image, the multiple blurred degraded images corresponding to each high-definition image are generated using multiple image blur degradation models, and the multiple image blur degradation models have different influencing factor orders and / or different influencing modes, the trained image super-resolution model can super-resolve images from multiple dimensions corresponding to different influencing factor orders and / or different influencing modes, thereby improving the image super-resolution effect and, in turn, improving the clarity of images captured by the infrared camera.

[0085] Corresponding to the above method embodiment, the embodiment of the present application also provides an image processing device. Figure 5 As shown, Figure 5 : is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device 500 may include:

[0086] An acquisition module 501 is configured to acquire a first image captured by an infrared camera;

[0087] a super-resolution module 502 configured to super-resolve the first image using an image super-resolution model to obtain a second image, wherein the image super-resolution model is trained using multiple high-definition images and multiple blurred and degraded images corresponding to each high-definition image, and the multiple blurred and degraded images corresponding to each high-definition image are generated using multiple image blur degradation models, wherein the multiple image blur degradation models have different influencing factor orders and / or different influencing modes;

[0088] The grayscale processing module 503 is used to perform grayscale processing on the second image to obtain a target image.

[0089] In an embodiment of the present application, a first image captured by an infrared camera is super-resolved using an image super-resolution model trained using multiple high-definition images and multiple blurred degraded images corresponding to each high-definition image to obtain a second image; and the second image is grayscale processed to obtain a target image. Since the image super-resolution model is trained using multiple high-definition images and multiple blurred degraded images corresponding to each high-definition image, the multiple blurred degraded images corresponding to each high-definition image are generated using multiple image blur degradation models, and the multiple image blur degradation models have different influencing factor orders and / or different influencing modes, the trained image super-resolution model can super-resolve images from multiple dimensions corresponding to different influencing factor orders and / or different influencing modes, thereby improving the image super-resolution effect and, in turn, improving the clarity of images captured by the infrared camera.

[0090] In some possible implementations of the embodiments of the present application, the image processing apparatus 500 provided in the embodiments of the present application may further include:

[0091] A training module is used to generate a predicted image corresponding to a first blurred degraded image through a generator of a generative adversarial network, wherein the resolution of the predicted image is greater than the resolution of the first blurred degraded image, and the first blurred degraded image is an image among multiple blurred degraded images corresponding to multiple high-definition images; generate a discrimination result of whether the predicted image and the high-definition image corresponding to the first blurred degraded image are real images through a discriminator of the generative adversarial network; input the predicted image, the high-definition image corresponding to the first blurred degraded image, and the discrimination result into the loss function of the generator to obtain the loss value of the generator; input the discrimination result into the loss function of the discriminator to obtain the loss value of the discriminator; update the parameters of the generative adversarial network according to the loss value of the generator and the loss value of the discriminator to obtain an image super-resolution model, wherein the image super-resolution model is an image super-resolution model based on the generative adversarial network.

[0092] In some possible implementations of the embodiments of the present application, the image processing apparatus 500 provided in the embodiments of the present application may further include:

[0093] The blur degradation module is used to input the first high-definition image into multiple image blur degradation models respectively to obtain multiple blur degraded images corresponding to the first high-definition image, wherein the first high-definition image is any one of the multiple high-definition images, and the image blur degradation model includes three influencing factors: blur kernel, downsampling and noise, and the same influencing factor has different influencing modes.

[0094] In some possible implementations of the embodiments of the present application, the grayscale processing module may include:

[0095] a calculation submodule, configured to calculate a cumulative distribution corresponding to each of a plurality of grayscale values ​​corresponding to the second image;

[0096] a determination submodule, configured to determine a first grayscale threshold and a second grayscale threshold according to the cumulative distribution, wherein the second grayscale threshold is greater than the first grayscale threshold;

[0097] The adjustment submodule is used to keep the grayscale values ​​of pixels in the second image whose grayscale values ​​are greater than or equal to the first grayscale threshold and less than or equal to the second grayscale threshold unchanged; adjust the grayscale values ​​of pixels in the second image whose grayscale values ​​are less than the first grayscale threshold to the first grayscale value; adjust the grayscale values ​​of pixels in the second image whose grayscale values ​​are greater than the second grayscale threshold to the second grayscale value, and obtain a target image, wherein the second grayscale value is greater than the first grayscale value.

[0098] In some possible implementations of the embodiments of the present application, the determining submodule may include:

[0099] a first value assigning unit, configured to assign a first target grayscale value used to determine a first grayscale threshold value to zero;

[0100] a first judging unit, configured to judge whether a first cumulative distribution corresponding to a first target grayscale value is greater than or equal to a first cumulative distribution threshold;

[0101] a second assignment unit, configured to assign the first target grayscale value to the sum of the first target grayscale value and 1 when the first cumulative distribution is less than a first cumulative distribution threshold, thereby triggering the first judgment unit;

[0102] The first determining unit is configured to use the first target grayscale value as a first grayscale threshold when the first cumulative distribution is greater than or equal to a first cumulative distribution threshold.

[0103] In some possible implementations of the embodiments of the present application, the determining submodule may include:

[0104] a second determining unit, configured to determine a second target grayscale value for determining a second grayscale threshold according to the number of grayscale values ​​in the second image;

[0105] a second judging unit, configured to judge whether a second cumulative distribution corresponding to a second target grayscale value is greater than or equal to a second cumulative distribution threshold;

[0106] a third assignment unit, configured to assign the second target grayscale value to the difference between the second target grayscale value and 1 when the second cumulative distribution is greater than or equal to the second cumulative distribution threshold, thereby triggering the second judgment unit;

[0107] The third determining unit is configured to use the second target grayscale value as the second grayscale threshold when the second cumulative distribution is less than the second cumulative distribution threshold.

[0108] Figure 6It is a structural diagram of an electronic device provided in an embodiment of the present application.

[0109] The electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0110] Specifically, the processor 601 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0111] Memory 602 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be inside or outside the electronic device. In some specific embodiments, memory 602 is a non-volatile solid-state memory.

[0112] In some specific embodiments, the memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the image processing method according to the present application.

[0113] The processor 601 implements the image processing method provided in the embodiment of the present application by reading and executing computer program instructions stored in the memory 602.

[0114] In some examples, the electronic device may further include a communication interface 603 and a bus 610. Figure 6 As shown, the processor 601, the memory 602, and the communication interface 603 are connected via a bus 610 and communicate with each other.

[0115] The communication interface 603 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0116] The bus 610 includes hardware, software, or both that couples components of the electronic device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 610 may include one or more buses. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0117] The electronic device can execute the image processing method provided in the embodiment of the present application, thereby achieving the corresponding technical effects of the image processing method provided in the embodiment of the present application.

[0118] In addition, in conjunction with the image processing method in the above embodiment, embodiments of the present application also provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when executed by a processor, the computer program instructions implement the steps of the image processing method provided in the embodiments of the present application. Examples of computer-readable storage media include non-transitory computer-readable media, such as ROM, RAM, magnetic disks, or optical disks.

[0119] An embodiment of the present application also provides a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the steps of the image processing method provided in the embodiment of the present application are implemented and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0120] The present application also provides a vehicle, comprising at least one of the following items:

[0121] The image processing device provided in the embodiment of the present application;

[0122] The electronic device provided in the embodiments of the present application;

[0123] The computer-readable storage medium provided in the embodiments of the present application.

[0124] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image captured by an infrared camera; Super-resolution processing is performed on the first image using an image super-resolution model to obtain a second image, wherein the image super-resolution model is trained using multiple high-definition images and multiple blurred and degraded images corresponding to each high-definition image, the multiple blurred and degraded images corresponding to each high-definition image are generated using multiple image blur degradation models, and the multiple image blur degradation models have different influencing factor orders and / or different influencing modes; Grayscale processing is performed on the second image to obtain a target image.

2. The method according to claim 1, wherein Training the image super-resolution model includes: generating, by a generator of a generative adversarial network, a predicted image corresponding to a first blurred and degraded image, wherein a resolution of the predicted image is greater than a resolution of the first blurred and degraded image, and the first blurred and degraded image is an image among a plurality of blurred and degraded images corresponding to the plurality of high-definition images; generating, by the discriminator of the generative adversarial network, a discrimination result of whether the high-definition image corresponding to the predicted image and the first blurred and degraded image is a real image; Inputting the predicted image, the high-definition image corresponding to the first blurred and degraded image, and the discrimination result into the loss function of the generator to obtain a loss value of the generator; Inputting the discrimination result into the loss function of the discriminator to obtain the loss value of the discriminator; According to the loss value of the generator and the loss value of the discriminator, the parameters of the generative adversarial network are updated to obtain the image super-resolution model, wherein the image super-resolution model is an image super-resolution model based on the generative adversarial network.

3. The method according to claim 1, wherein Generating a plurality of blurred degraded images corresponding to each high-definition image includes: The first high-definition image is input into the multiple image blur degradation models respectively, and the multiple image blur degradation models generate multiple blurred degraded images corresponding to the first high-definition image, wherein the first high-definition image is any one of the multiple high-definition images, and the image blur degradation model includes three influencing factors: blur kernel, downsampling and noise, and the same influencing factor has different influencing modes.

4. The method according to claim 1, wherein The grayscale processing of the second image to obtain a target image includes: Calculating a cumulative distribution corresponding to each of a plurality of grayscale values ​​corresponding to the second image; Determining a first grayscale threshold and a second grayscale threshold according to the cumulative distribution, wherein the second grayscale threshold is greater than the first grayscale threshold; Maintaining unchanged the grayscale values ​​of pixels in the second image whose grayscale values ​​are greater than or equal to the first grayscale threshold and less than or equal to the second grayscale threshold; Adjusting the grayscale values ​​of pixels in the second image whose grayscale values ​​are less than the first grayscale threshold to the first grayscale value; The grayscale values ​​of pixels in the second image whose grayscale values ​​are greater than the second grayscale threshold are adjusted to a second grayscale value to obtain the target image, wherein the second grayscale value is greater than the first grayscale value.

5. The method according to claim 4, wherein Determining the first grayscale threshold according to the cumulative distribution includes: Assigning a first target grayscale value used to determine the first grayscale threshold to zero; Determining whether a first cumulative distribution corresponding to the first target grayscale value is greater than or equal to a first cumulative distribution threshold; If the first cumulative distribution is less than the first cumulative distribution threshold, assigning the first target grayscale value to the sum of the first target grayscale value and 1, and continuing to perform the step of determining whether the first cumulative distribution corresponding to the first target grayscale value is greater than or equal to the first cumulative distribution threshold; When the first cumulative distribution is greater than or equal to the first cumulative distribution threshold, the first target grayscale value is used as the first grayscale threshold.

6. The method according to claim 4, wherein Determining the second grayscale threshold according to the cumulative distribution includes: determining a second target grayscale value for determining the second grayscale threshold according to the number of grayscale values ​​in the second image; Determining whether a second cumulative distribution corresponding to the second target grayscale value is greater than or equal to a second cumulative distribution threshold; If the second cumulative distribution is greater than or equal to the second cumulative distribution threshold, assigning the second target grayscale value to the difference between the second target grayscale value and 1, and continuing to perform the step of determining whether the second cumulative distribution corresponding to the second target grayscale value is greater than or equal to the second cumulative distribution threshold; When the second cumulative distribution is less than the second cumulative distribution threshold, the second target grayscale value is used as the second grayscale threshold.

7. An image processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a first image captured by an infrared camera; a super-resolution module, configured to super-resolve the first image using an image super-resolution model to obtain a second image, wherein the image super-resolution model is trained using multiple high-definition images and multiple blurred and degraded images corresponding to each high-definition image, the multiple blurred and degraded images corresponding to each high-definition image are generated using multiple image blur degradation models, and the multiple image blur degradation models have different influencing factor orders and / or different influencing modes; The grayscale processing module is used to perform grayscale processing on the second image to obtain a target image.

8. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the steps of the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the steps of the image processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises computer program instructions, which, when executed by a processor, implement the steps of the image processing method according to any one of claims 1 to 6.

11. A vehicle, characterized in that: The vehicle includes at least one of the following: The image processing device according to claim 7; The electronic device according to claim 8; The computer-readable storage medium of claim 9.

Images