Image processing method, device and equipment

Abstract

The invention discloses an image processing method, device and equipment. The method comprises the following steps: the electronic equipment acquires a first image of a first object under illumination; the electronic device acquires a second image of the first object without illumination. The electronic equipment inputs the first image and the second image into an enhancement model and outputs a third image, the third image is an enhancement result image of the second image, and the image quality evaluation of the third image is higher than that of the second image. In the application, the enhancement model is a deep learning model, and is obtained by training according to the reference number of sample images, the to-be-enhanced image obtained by the sample images and the enhancement result image. Therefore, by inputting the second image to be enhanced and the first image for assistance into the enhancement model, a high-quality enhanced result image can be output. Compared with an image processing mode of adjacent frame superposition in the related technology, the image quality of the image obtained by the method can better meet user requirements.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to image processing methods, devices and equipment. Background Art

[0002] Images captured by cameras at night often have poor image quality in dark areas. To address this technical issue, related technologies typically use strong fill light to capture images, resulting in ideal image quality. Furthermore, with growing environmental awareness, there's a trend toward using weaker fill light instead of stronger fill light to still achieve ideal image quality. However, this weak fill light approach reduces the signal-to-noise ratio (SNR) of the image, resulting in poor image quality in dark areas and image quality that fails to meet user expectations. Summary of the Invention

[0003] This application provides an image processing method, apparatus, and device to solve the problems provided by related technologies. The technical solutions are as follows:

[0004] In a first aspect, an image processing method is provided, the method comprising: acquiring a first image, the first image being an image of a first object under light; acquiring a second image, the second image being an image of the first object under no light; inputting the first image and the second image into an enhancement model, and outputting a third image, the third image being an enhanced result image of the second image, the image quality of the third image being higher than the image quality of the second image; wherein the enhancement model is trained based on a reference number of sample images, images to be enhanced, and enhanced result images, the sample images providing feature detail support for the enhancement processing operation of the images to be enhanced, and the enhanced result image being the image after the enhancement processing of the images to be enhanced; the sample images, the images to be enhanced, and the enhanced result images contain the same content.

[0005] In this application, since the enhancement model is a deep learning model, it is trained based on a reference number of sample images, and the image to be enhanced and the enhanced result image obtained from the sample images. Therefore, by inputting the second image to be enhanced and the first image used for auxiliary purposes into the enhancement model, a high-quality enhanced result image (i.e., the third image) can be output. Compared to the image processing method of the related art by superimposing adjacent frames, the image quality of the image obtained in this application will be better and can meet user needs.

[0006] It should be understood that in the present application, based on the enhancement model, the difference area image between the first image and the second image can be calculated, and the difference area image can be processed in different forms to achieve different processing effects of the second image on the difference area image. It can be manifested as the area different from the first image is processed using the information of the second image, and the other areas will use part of the content of the first image to enhance the details of the second image. Such processing will not produce negative signals such as afterimages, so that the image quality of the enhanced result image (i.e., the third image) output by the enhancement model is better.

[0007] In one possible implementation, before inputting the first image and the second image into the enhancement model and outputting the third image, the method further includes: obtaining a reference number of sample images, where the sample images are images of the second object under illumination; obtaining the image to be enhanced and the enhanced result image based on the sample images; and training the enhancement model using the sample images and the image to be enhanced as input data and the enhanced result image as output data.

[0008] In a possible implementation, the sample image and the image to be enhanced are used as input data, and the enhanced result image is used as output data to train the enhancement model, including: obtaining an output result image based on the sample image and the image to be enhanced; comparing the sample image and the enhanced result image in a first dimension to obtain a difference area image; comparing the output result image, the enhanced result image and the difference area image in a second dimension to obtain the similarity between the output result image and the enhanced result image, wherein the second dimension at least includes the first dimension; and obtaining the enhancement model when the similarity satisfies a threshold interval. In the present application, by calculating the difference area image between the sample image and the image to be enhanced, the difference area image is processed in different forms to achieve different effects of the image to be enhanced on the difference area image, which can be manifested as the area different from the sample image is processed using the information of the image to be enhanced, and the other areas will use part of the content of the sample image to enhance the details of the image to be enhanced.

[0009] In one possible implementation, acquiring the first image includes: acquiring a fourth image of the first object during a first time period, where the first time period is a period of illumination at the location of the first object; and determining that the fourth image is the first image when an average pixel value of the fourth image satisfies a first threshold. In this application, the electronic device can acquire auxiliary images within a specified time period, eliminating the need to constantly acquire auxiliary images, effectively saving power.

[0010] In a possible implementation, the method further includes: if the average pixel value of the fourth image does not meet the first threshold, acquiring a fifth image of the first object in a second time period; the second time period is another time period in the illumination time period of the location of the first object; if the average pixel value of the fifth image meets the first threshold, determining that the fifth image is the first image. In the present application, the electronic device can acquire auxiliary images within a specified time period. If the image does not meet the requirements, the electronic device continues to acquire auxiliary images within the next specified time period until an image that meets the requirements is acquired. This allows the electronic device to purposefully acquire a first image that meets the requirements, ensure the standardization of the first image, and further ensure the accuracy of subsequent image processing.

[0011] In one possible implementation, acquiring the first image data includes: acquiring the weather at the location of the first object during a third time period; the third time period being a time period of illumination at the location of the first object; and acquiring the first image during the third time period when the weather at the location of the first object is a specified weather condition. In this application, the electronic device can acquire an auxiliary first image in conjunction with weather information, enabling accurate acquisition of a first image that meets the requirements, effectively preventing power consumption caused by multiple image acquisitions, and improving efficiency.

[0012] In a second aspect, an image processing device is provided, comprising: an acquisition module and a processing module; wherein the acquisition module is used to acquire a first image, which is an image of a first object under light; the acquisition module is also used to acquire a second image, which is an image of the first object under no light; the processing module is used to input the first image and the second image into an enhancement model, and output a third image, which is an enhanced result image of the second image, and the image quality of the third image is higher than that of the second image; wherein the enhancement model is trained based on a reference number of sample images, images to be enhanced, and enhancement result images, the sample images provide feature detail support for the enhancement processing operation of the images to be enhanced, and the enhancement result image is the image after the enhancement processing of the images to be enhanced; the sample images, the images to be enhanced, and the enhancement result images contain the same content.

[0013] In a possible implementation, the acquisition module is further used to: acquire a fourth image of the first object in a first time period, where the first time period is a period of illumination time period at the location of the first object; and determine that the fourth image is the first image when the average pixel value of the fourth image meets a first threshold.

[0014] In one possible implementation, the apparatus further includes a determination module. The acquisition module is further configured to, if the average pixel value of the fourth image does not meet the first threshold, acquire a fifth image of the first object during a second time period; the second time period being another period of illumination during which the location of the first object is located. The determination module is configured to, if the average pixel value of the fifth image meets the first threshold, determine that the fifth image is the first image.

[0015] In one possible implementation, the acquisition module is also used to: acquire the weather at the location of the first object in a third time period; the third time period is the lighting time period at the location of the first object; and acquire the first image in the third time period when the weather at the location of the first object is the specified weather.

[0016] According to a third aspect, an electronic device is provided, comprising a memory and a processor; at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor so that the electronic device implements the methods in various aspects.

[0017] In a fourth aspect, a computer program (product) is provided, which includes: computer program code, which, when executed by a computer, enables the computer to execute the methods in the above aspects.

[0018] In a fifth aspect, a computer-readable storage medium is provided, which stores a program or instruction. When the program or instruction runs on a computer, the methods in the above aspects are executed.

[0019] In a sixth aspect, a chip is provided, comprising a processor for calling and executing instructions stored in a memory from the memory, so that a communication device equipped with the chip executes the methods in the above aspects.

[0020] In the seventh aspect, another chip is provided, comprising: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the methods in the above aspects.

[0021] It should be understood that the beneficial effects achieved by the technical solutions of the second to seventh aspects of this application and the corresponding possible implementation methods can be referred to the technical effects of the first aspect and its corresponding possible implementation methods mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of a reference frame (or auxiliary) image provided for related technology;

[0023] Figure 2 A schematic diagram of an image to be enhanced provided for related technologies;

[0024] Figure 3 A schematic diagram of an enhanced result image provided for related technology;

[0025] Figure 4 A schematic diagram of an enhanced result image provided in an embodiment of the present application;

[0026] Figure 5 A flowchart of a training method for an enhanced model provided in an embodiment of the present application;

[0027] Figure 6 A flowchart of an image processing method provided in an embodiment of the present application;

[0028] Figure 7 A schematic structural diagram of a camera module provided in an embodiment of the present application;

[0029] Figure 8 A schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0030] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0032] In order to avoid the poor image quality of dark areas in images taken by cameras at night, in a related technology, weak fill light is used instead of strong fill light to still obtain an ideal image. However, the weak fill light solution reduces the signal-to-noise ratio of the image, resulting in poor image quality in dark areas of the image, and the image quality cannot meet user requirements. In order to solve the problem of low signal-to-noise ratio of images caused by the weak fill light method, in another related technology, a method is used to obtain a signal-to-noise ratio from the surveillance video. Figure 1 The reference frame shown and Figure 2 The low-light image to be enhanced is shown in FIG. 1 . The background area of the adjacent frame image of the low-light image is calculated using the reference frame image during the day, and then the background area is superimposed on the low-light image to obtain the following image: Figure 3 However, its essence is the principle of superimposing adjacent frames, which has limited improvement in signal-to-noise ratio and still leaves the problem that image quality cannot meet user requirements.

[0033] In order to solve the above technical problems, an embodiment of the present application provides an image processing method, the execution subject of the method can be an electronic device, and the method includes: the electronic device obtains a first image of the first object under light. The electronic device obtains a second image of the first object under no light. The electronic device inputs the first image and the second image into an enhancement model, and outputs a third image, the third image is the enhanced result image of the second image, and the image quality evaluation of the third image (which can be referred to as image quality) is higher than the image quality evaluation of the second image. In the present application, since the enhancement model is a deep learning model, it is trained based on a reference number of sample images, and the image to be enhanced and the enhanced result image obtained by the sample images. Therefore, by inputting the second image to be enhanced and the first image for assistance into the enhancement model, a high-quality enhanced result image can be output. Compared with the image processing method of superimposing adjacent frames in the related art, the image quality of the image obtained in the present application will be better and can meet user needs.

[0034] It should be understood that in the present application, based on the enhancement model, the difference area image between the first image and the second image can be calculated, and the difference area image can be processed in different forms to achieve different processing effects of the second image on the difference area image. It can be manifested as the area different from the first image is processed using the information of the second image, and the other areas will use part of the content of the first image to enhance the details of the second image. Such processing will not produce negative signals such as afterimages, so that the image quality of the enhanced result image (i.e., the third image) output by the enhancement model is better.

[0035] In some embodiments, image quality assessment (IQA) may include at least one of the following: the average pixel value of the image, the clarity of the image, the structural similarity (or similarity) with the first image, and the signal-to-noise ratio of the image. Exemplarily, the image quality assessment includes the average pixel value of the image, and the average pixel value of the third image is greater than the average pixel value of the second image. The image quality assessment includes the clarity of the image, and the clarity of the third image is greater than the clarity of the second image. The image quality assessment includes the signal-to-noise ratio of the image, and the signal-to-noise ratio of the third image is greater than the signal-to-noise ratio of the second image. Of course, the image quality assessment may include information of multiple dimensions. For example, the image quality assessment may include the average pixel value and the clarity of the image, and the average pixel value of the third image is greater than the average pixel value of the second image, and the clarity of the third image is greater than the clarity of the second image. These are not listed one by one in the embodiments of the present application.

[0036] For example, following the above example, assuming that the first image is Figure 1 The image shown, the second image is Figure 2 The electronic device will Figure 1 The images shown and Figure 2The image input enhancement model shown in the figure outputs Figure 4 The third image shown is the enhanced result image of the second image. Since the enhancement model is trained based on a certain amount of sample images, the image to be enhanced corresponding to the sample images, and the enhanced result image. Therefore, the enhancement model can be based on the auxiliary Figure 1 The image shown and the one to be enhanced Figure 2 The image shown in Figure 1 derives Figure 4 The enhanced result image is shown. It can be seen that Figure 4 The enhanced result image shown is compared with Figure 3 The enhanced result image shown has higher average pixel value and clarity, that is, the image quality evaluation is higher and the image quality is better.

[0037] The above-mentioned electronic device can be a terminal, a computer or a cloud platform, etc. The specific form of the electronic device is not particularly limited in the embodiments of the present application.

[0038] The image processing method provided in the embodiment of the present application can be introduced in detail below in different stages.

[0039] The first stage is the training stage of the enhanced model.

[0040] Figure 5 This is a flow chart of a training method for an enhanced model provided in an embodiment of the present application. Figure 5 As shown, the execution subject of the training method of the enhanced model may be an electronic device, and the method may include: S501-S502 (some steps of which are optional).

[0041] S501: The electronic device obtains a reference number of sample images, where the sample images are images of a second object under illumination.

[0042] The sample image can be understood as an auxiliary image, and the detailed features of the objects in the image can be clearly displayed. For example, the average pixel value of the sample image can be between 60-255.

[0043] The second thing can refer to any thing, especially a thing with a large static area. For example, the second thing can include indoors, residential courtyards, campuses, parks, communities, lakes, woods, roads, etc. Of course, the embodiments of the present application are not limited to the above examples.

[0044] Illumination can be understood as illumination from a light source. A light source can include the sun or a lamp. Exemplarily, illumination from a light source can include sunlight or light. In practical applications, illumination is understood as sunlight, which is applicable to objects with large static areas, such as residential areas and parks. Illumination is understood as light illumination, which is applicable to objects with small static areas, such as indoors and residential courtyards. In the embodiments of the present application, illumination is generally understood as sunlight.

[0045] In one possible implementation, S501 may be implemented as follows: the electronic device may download the sample image from the Internet; or the electronic device may have a camera and may capture the sample image through the camera. In the embodiment of the present application, the source of the sample image is not specifically limited.

[0046] S502: The electronic device uses the sample image and the image to be enhanced as input data and the enhanced result image as output data to train the enhancement model. The image to be enhanced and the enhanced result image are determined based on the sample image, and the enhanced result image is the image after the image to be enhanced is enhanced.

[0047] The enhancement model may include, but is not limited to, a fully convolutional neural network (U-net) network model. The U-net network model may include a contracting path and an expanding path. The contracting path is used to obtain context information. The expanding path is used for precise localization. The contracting path and the expanding path are symmetrical.

[0048] In a possible implementation, S502 may include S5021-S5024.

[0049] S5021. The electronic device obtains an output result image according to the sample image and the image to be enhanced.

[0050] S5022: The electronic device compares the sample image and the enhanced result image in the first dimension to obtain a difference region image. The difference region image can be understood as an image composed of different pixels of the sample image and its corresponding enhanced result image.

[0051] S5023. The electronic device compares the output result image, the enhanced result image, and the difference area image in a second dimension to obtain similarity between the output result image and the enhanced result image, where the second dimension at least includes the first dimension.

[0052] S5024: When the similarity satisfies the threshold range, the electronic device completes the training of the enhancement model.

[0053] In an embodiment of the present application, by calculating the difference area image between the sample image and the image to be enhanced, the difference area image is processed in different forms to achieve different processing effects of the image to be enhanced on the difference area image. It can be manifested as the area different from the sample image is processed using the information of the image to be enhanced, and other areas will use part of the content of the sample image to enhance the details of the image to be enhanced.

[0054] The second stage is the stage of enhancing the use of the model.

[0055] Figure 6 This is a flowchart of an image processing method provided in an embodiment of the present application, or a flowchart of a method for using an enhanced model. Figure 6 As shown, the execution subject of the training method of the enhanced model may be an electronic device, and the method may include: S601-S603 (some steps of which are optional).

[0056] S601: The electronic device obtains a first image, where the first image is an image of a first object under light.

[0057] The first thing may be the same as the second thing mentioned above. For details, please refer to the relevant description of the second thing mentioned above, which will not be repeated here.

[0058] The first image can be understood as an auxiliary image. The image data of the first image does not contain noise. For example, the average pixel value of the first image is between 60 and 255. Typically, the first image can be an image of the first object during the day.

[0059] The first image may be obtained indirectly by the electronic device. For example, the first image is an image downloaded by the electronic device over a network.

[0060] Of course, the first image can also be obtained directly by an electronic device. For example, the electronic device may include a camera module. The camera module may include a motor, a drive module, a processor, a filter module, a lens module, and a sensor module. Among them, the camera module in some related technologies may include a polarizer 2 and an infrared filter. Polarizer 2 is generally used to filter strong reflections from glass (such as car window glass). Of course, the camera module in other related technologies may not include polarizer 2, which is not limited here. In the present application, the filter module may also include polarizer 1. Polarizer 1 is used to filter most of the high-light signals in the picture, not just to filter the reflection signals from the car window glass.

[0061] like Figure 7 As shown, the filter module is set between the lens module and the sensor module. The filter module is electrically connected to the motor, which is connected to the driving end of the drive module. The control end of the drive module is electrically connected to the processor. The processor is used to control the driving end of the drive module to rotate, so as to drive the motor to work and drive the polarizer 1, polarizer 2 or infrared filter in the filter module to move. The motor can push the polarizer 1 to Figure 7 Alternatively, the motor can push the polarizer 2 to Figure 7 Alternatively, the motor can push the infrared filter to position 2 as shown in Figure 7Position 3 is shown. With polarizer 1 in position 1, light reflected from the target passes through the lens module, then through polarizer 1, filtering out most of the high-light signals. The resulting light signal is received by the sensor module and transmitted to the processor, which generates an image based on this light signal. This significantly reduces overexposed areas in the image, ensuring that the image is not overly bright and has a large amount of overexposed areas, allowing image details to be seen. Therefore, when the motor pushes polarizer 1 to position 1 between the lens module and the sensor module, the camera can capture the first image.

[0062] In the embodiment of the present application, the first image is acquired through a polarizing plate, so that the details of the static area in the first image are easily visible, while the loss of details due to overexposure is suppressed.

[0063] However, the image acquired by the electronic device is not necessarily the first image that meets the requirements. Therefore, in order to enable the electronic device to acquire the first image that meets the requirements, the following acquisition method can be used:

[0064] Acquisition method one, S601 may include S6011 and S6012. S6011, the electronic device acquires a fourth image of the first object in a first time period, where the first time period is a period of time within the illumination time period of the location of the first object. For example, the location of the first object is City A, and the illumination time period of City A is 07:32:02-16:52:24. Assume that the first time period is 10:00:00-10:30:00. Then, the electronic device acquires the fourth image of the first object within 10:00:00-10:30:00. After the electronic device acquires the fourth image, the electronic device determines the average pixel value of the fourth image based on the pixel value of each pixel in the fourth image. S6012, when the average pixel value of the fourth image meets the first threshold, the electronic device determines that the fourth image is the first image. The range of the first threshold can be 60-255. If the average pixel value of the fourth image meets 60-255, the electronic device determines that the fourth image is the first image.

[0065] In practical applications, the first acquisition method can be applied to any scenario and is not specifically limited in the embodiments of the present application.

[0066] In an embodiment of the present application, the electronic device can capture auxiliary images within a specified time period without having to capture auxiliary images all the time, thereby effectively saving power consumption.

[0067] Acquisition method 2 differs from method 1 in that, if the average pixel value of the fourth image does not meet the first threshold, the electronic device continues to acquire images until the average pixel value of the acquired images meets the first threshold. For example, S601 may also include S6013 and S6014. S6013: If the average pixel value of the fourth image does not meet the first threshold, acquire a fifth image of the first object during a second time period; the second time period is another time period within the illumination time period at the location of the first object. Continuing with the above example, the second time period is 10:31:00-11:00:00. Then, the electronic device acquires a fifth image of the first object during 10:31:00-11:00:00. Similarly, after acquiring the fifth image, the electronic device determines the average pixel value of the fifth image based on the pixel values of each pixel in the fifth image. S6014: If the average pixel value of the fifth image meets the first threshold, determine that the fifth image is the first image. If the average pixel value of the fifth image falls within the range of 60-255, the electronic device determines that the fifth image is the first image. Of course, if the average pixel value of the fifth image does not meet the range of 60-255, the electronic device continues to capture images of the first object in the next time period until an image with an average pixel value that meets the first threshold is acquired. In some embodiments, if the first image is not acquired during the day's light-emitting period, the electronic device may use the historical image as an auxiliary image to enhance the second image captured that night.

[0068] In practical applications, the second acquisition method can also be applied to any scenario and is not specifically limited in the embodiments of this application.

[0069] In an embodiment of the present application, the electronic device can capture an auxiliary image within a specified time period. If the image does not meet the requirements, the electronic device continues to capture auxiliary images within the next specified time period until an image that meets the requirements is captured. In this way, the first image that meets the requirements can be obtained purposefully, ensuring the standardization of the first image, and further ensuring the accuracy of subsequent image processing.

[0070] Acquisition method three, compared with the above-mentioned method one and method two, is that the electronic device acquires the first image in combination with the weather at the location of the first thing. Exemplarily, S601 may include S6015 and S6016. S6015. In the third time period, the electronic device acquires the weather at the location of the first thing; the third time period is the lighting time period at the location of the first thing. Continuing with the above example, assume that from 11:31:00 to 12:00:00 on November 1, 2023, the electronic device acquires that the weather in City A is sunny. S6016. In the third time period, and when the weather at the location of the first thing is the specified weather, the electronic device acquires the first image. In one example, the specified weather may include: sunny or cloudy. Continuing with the above example, from 11:31:00 to 12:00:00 on November 1, 2023, the weather in City A is sunny. Then, the electronic device defaults to the situation where the image captured meets the first threshold value between 11:31:00 and 12:00:00 on November 1, 2023, when it is sunny. At this time, the electronic device directly captures the first image. In some embodiments, during the third time period, if the weather at the location of the first object is not the specified weather, the electronic device does not capture the first image. For example, assume that on November 2, 2023, the electronic device captures that the weather in City A is foggy and hazy. Then, the electronic device does not capture the first image on November 2, 2023.

[0071] In practical applications, the third acquisition method can be applied to a small area scenario, for example, the first thing is a residential courtyard, a farmyard, etc. In this scenario, the electronic device can connect to the Internet and obtain weather information at the location of the electronic device through the Internet.

[0072] In an embodiment of the present application, the electronic device can obtain an auxiliary first image in combination with weather information, so that a first image that meets the requirements can be accurately obtained, effectively preventing power consumption caused by multiple image acquisitions and improving efficiency.

[0073] S602: The electronic device acquires a second image, where the second image is an image of the first object in the absence of light.

[0074] The second image can be understood as the image to be enhanced, and may have a large number of dark areas. The image data of the second image may be raw data, which is a data format directly read from a camera without any processing. For example, the average pixel value of the second image may be between 0 and 20. Typically, the second image may be an image of the first object in a low-light environment. For example, an image of the first object in the absence of light. Another example is an image of the first object at night.

[0075] In one possible implementation, using the above example, S602 may be implemented as follows: when the driving module drives the polarizer to move away from the lens module and the sensor module, and the camera can acquire the second image during a fourth time period, the fourth time period being a time when the location of the first object is not illuminated.

[0076] In an embodiment of the present application, a polarizing film is used on a camera. A polarizing film is used to obtain a high-definition first image during a period of illumination, which serves as an auxiliary image for a second image without illumination. After the first image is acquired, and during a period of no illumination, the camera does not use a polarizing film to acquire a second image. At this time, the image data obtained by the camera not only has an ultra-high signal-to-noise ratio and removes some signal interference, but also the overall data distribution of the two images is close, which is more conducive to algorithm processing. In addition, the details of the static area in the first image obtained through the polarizing film are easy to see, and no additional non-existent content will be generated due to overexposure, so the authenticity is guaranteed, and no afterimages will be generated.

[0077] S603: The electronic device inputs the first image and the second image into an enhancement model and outputs a third image, where the third image is an enhanced result image of the second image, and an image quality evaluation of the third image is higher than that of the second image.

[0078] The image quality evaluation may include at least one of the following: the average pixel value of the image, the clarity of the image, the structural similarity (or similarity) with the first image, and the signal-to-noise ratio of the image. Exemplarily, the image quality evaluation includes the average pixel value of the image, and the average pixel value of the third image is greater than the average pixel value of the second image. The image quality evaluation includes the clarity of the image, and the clarity of the third image is greater than the clarity of the second image. The image quality evaluation includes the signal-to-noise ratio of the image, and the signal-to-noise ratio of the third image is greater than the signal-to-noise ratio of the second image. Of course, the image quality evaluation may include information of multiple dimensions. For example, the image quality evaluation may include the average pixel value and the clarity of the image, the average pixel value of the third image is greater than the average pixel value of the second image, and the clarity of the third image is greater than the clarity of the second image. These are not listed one by one in the embodiments of the present application.

[0079] In actual application, when there is sufficient sunlight during the day, the electronic device collects the data of a certain area of cell A. Figure 1 In the absence of light at night, the electronic device collects the first image of the area. Figure 2 The electronic device inputs the first image and the second image into the enhancement model to obtain Figure 4 The third image shown is Figure 2 The enhanced result image of the second image is shown. Figure 3The enhanced result image obtained after the related technology processing shown in the figure has a higher image quality evaluation and better image quality than the enhanced result image obtained by the enhancement model in the embodiment of the present application.

[0080] In an embodiment of the present application, based on the enhancement model, the difference area image between the first image and the second image can be calculated, and the difference area image can be processed in different forms to achieve different processing effects of the second image on the difference area image. It can be manifested as the area different from the first image is processed using the information of the second image, and the other areas will use part of the content of the first image to enhance the details of the second image. Such processing will not produce negative signals such as afterimages, so that the image quality of the enhanced result image (i.e., the third image) output by the enhancement model is better.

[0081] In some embodiments, before the electronic device executes S603, the electronic device may pre-process the acquired image. For example, the image processing method provided in the embodiment of the present application may further include: the electronic device converting the format of the acquired first image and the second image to obtain a data format that conforms to the enhancement model.

[0082] In other embodiments, after the electronic device executes S603, the electronic device may further process the obtained image. For example, the image processing method provided in the embodiment of the present application may further include: the electronic device performing processing operations such as demosaicing, contrast adjustment, and brightness stretching on the third image, and displaying the image after the processing is completed.

[0083] like Figure 8 As shown, an embodiment of the present application also provides an image processing device, which includes: an acquisition module 801 and a processing module 802; wherein the acquisition module 801 is used to acquire a first image, which is an image of the first object under light; the acquisition module 801 is also used to acquire a second image, which is an image of the first object under no light; the processing module 802 is used to input the first image and the second image into an enhancement model, and output a third image, which is an enhanced result image of the second image, and the image quality of the third image is higher than the image quality of the second image; wherein the enhancement model is trained based on a reference number of sample images, images to be enhanced and enhanced result images, the sample images provide feature detail support for the enhancement processing operation of the images to be enhanced, and the enhanced result image is the image after the image to be enhanced is enhanced; the sample images, the images to be enhanced and the enhanced result images contain the same content.

[0084] In this application, the enhancement model is a deep learning model, trained based on a reference number of sample images, the image to be enhanced, and the enhanced result image obtained from the sample images. Therefore, by inputting the second image to be enhanced and the auxiliary first image into the enhancement model, a high-quality enhanced result image can be output. Compared to the image processing method of the related art that superimposes adjacent frames, the image quality obtained in this application is better.

[0085] It should be understood that in the present application, based on the enhancement model, the difference area image between the first image and the second image can be calculated, and the difference area image can be processed in different forms to achieve different processing effects of the second image on the difference area image. It can be manifested as the area different from the first image is processed using the information of the second image, and the other areas will use part of the content of the first image to enhance the details of the second image. Such processing will not produce negative signals such as afterimages, so that the image quality of the enhanced result image (i.e., the third image) output by the enhancement model is better.

[0086] In one possible implementation, the enhancement model includes a fully convolutional neural network U-net model.

[0087] In a possible implementation, the image quality includes at least one of the following: an average pixel value of the image, image clarity, structural similarity with the first image, and a signal-to-noise ratio of the image.

[0088] In one possible implementation, acquisition module 801 is further configured to: acquire a fourth image of the first object during a first time period, where the first time period is a period of illumination at the location of the first object; and determine that the fourth image is the first image when an average pixel value of the fourth image satisfies a first threshold. In this application, the electronic device can capture auxiliary images within a specified time period, eliminating the need to constantly capture auxiliary images, thereby effectively saving power consumption.

[0089] In a possible implementation, the device 800 also includes: a determination module 803. The acquisition module 801 is also used to acquire a fifth image of the first object in a second time period when the average pixel value of the fourth image does not meet the first threshold; the second time period is another time period in the illumination time period of the location of the first object. The determination module 803 is used to determine that the fifth image is the first image when the average pixel value of the fifth image meets the first threshold. In the present application, the electronic device can collect auxiliary images within a specified time period. If the image does not meet the requirements, the electronic device continues to collect auxiliary images within the next specified time period until an image that meets the requirements is collected. It can purposefully obtain a first image that meets the requirements, ensure the standardization of the first image, and further ensure the accuracy of subsequent image processing.

[0090] In one possible implementation, acquisition module 801 is further configured to: acquire the weather at the location of the first object during a third time period; the third time period being a period of illumination at the location of the first object; and acquire the first image during the third time period when the weather at the location of the first object is a specified weather condition. In this application, the electronic device can combine weather information to acquire an auxiliary first image, enabling accurate acquisition of a first image that meets the requirements, effectively preventing power consumption caused by multiple image acquisitions, and improving efficiency.

[0091] It should be understood that the above Figure 8 The provided device is illustrated only by the division of the above-mentioned functional modules when implementing its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0092] See also Figure 9 , Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 9 The electronic device 100 shown is used to perform the above Figure 5 The training method of the enhanced model shown, and / or, execution Figure 6 The operations involved in the image processing method shown are as follows: The electronic device 100 is, for example, a terminal, a computer, etc. The electronic device 100 can be implemented by a general bus architecture.

[0093] like Figure 9 As shown, the electronic device 100 includes at least one processor 101 , a memory 103 and at least one communication interface 104 .

[0094] The processor 101 is, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the solution of the present application. For example, the processor 101 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can implement or execute the various logic blocks, modules, and circuits described in conjunction with the disclosure of the embodiments of the present application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0095] Optionally, the electronic device 100 further includes a bus. The bus is used to transmit information between the components of the electronic device 100. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0096] The memory 103 is, for example, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 103 is, for example, independent and connected to the processor 101 via a bus. The memory 103 can also be integrated with the processor 101.

[0097] The communication interface 104 uses any transceiver-like device for communicating with other devices or communication networks. The communication network can be Ethernet, a radio access network (RAN), or a wireless local area network (WLAN). The communication interface 104 can include a wired communication interface and a wireless communication interface. Specifically, the communication interface 104 can be an Ethernet interface, a fast Ethernet (FE) interface, a gigabit Ethernet (GE) interface, an asynchronous transfer mode (ATM) interface, a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In an embodiment of the present application, the communication interface 104 can be used for the electronic device 100 to communicate with other devices.

[0098] In a specific implementation, as an embodiment, the processor 101 may include one or more CPUs, such as Figure 9 0 and CPU1 are shown in FIG. Each of these processors can be a single-CPU processor or a multi-CPU processor. A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0099] In a specific implementation, as an embodiment, the electronic device 100 may include multiple processors, such as Figure 9 1 and 105. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0100] In a specific implementation, as an embodiment, the electronic device 100 may further include an output device and an input device. The output device communicates with the processor 101 and can display information in a variety of ways. For example, the output device can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor 101 and can receive user input in a variety of ways. For example, the input device can be a mouse, a keyboard, a touch screen device, or a sensor device.

[0101] In some embodiments, the memory 103 is used to store program code 110 for executing the solution of the present application, and the processor 101 can execute the program code 110 stored in the memory 103. That is, the electronic device 100 can implement the image processing method provided by the method embodiment through the processor 101 and the program code 110 in the memory 103. The program code 110 may include one or more software modules. Optionally, the processor 101 itself may also store program code or instructions for executing the solution of the present application.

[0102] In a specific embodiment, the electronic device 100 of the embodiment of the present application may correspond to the computing device in the above-mentioned various method embodiments.

[0103] in, Figure 9 Each step of the image processing method shown is completed by an integrated logic circuit of hardware or software instructions in the processor of the electronic device 100. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0104] An embodiment of the present application further provides an electronic device, comprising a processor configured to load and execute at least one instruction to enable the electronic device to implement any of the above methods. Optionally, the device further comprises a memory coupled to the processor and configured to store the at least one instruction.

[0105] An embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0106] The embodiments of the present application further provide a computer program (product), which, when executed by a computer, can enable a processor or computer to execute the corresponding steps and / or processes in the above method embodiments.

[0107] An embodiment of the present application further provides a chip, which includes a processor for calling and executing instructions stored in a memory from the memory, so that a communication device equipped with the chip executes any of the above methods.

[0108] An embodiment of the present application also provides another chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute any of the methods described above.

[0109] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0110] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the setting results involved in this application are all obtained with full authorization.

[0111] Those skilled in the art will appreciate that the various method steps and modules described in conjunction with the embodiments disclosed herein can be implemented in software, hardware, firmware, or any combination thereof. In order to clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0112] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0113] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer program instructions. As an example, the method of the embodiment of the present application can be described in the context of a machine executable instruction, and the machine executable instruction is such as included in the program module executed in the device on the real or virtual processor of the target. Generally speaking, a program module includes a routine, a program, a library, an object, a class, a component, a data structure, etc., which performs a specific task or realizes a specific abstract data structure. In various embodiments, the function of the program module can be merged or split between the described program modules. The machine executable instruction for the program module can be executed in a local or distributed device. In a distributed device, the program module can be located in both a local and a remote storage medium.

[0114] The computer program code for realizing the method for the embodiment of the present application can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable image processing device so that the program code, when being executed by the computer or other programmable image processing device, causes the function / operation specified in the flow chart and / or block diagram to be implemented. The program code can be executed completely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer or completely on a remote computer or server.

[0115] In the context of the embodiments of the present application, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like.

[0116] Examples of signals may include electrical, optical, radio, acoustic or other forms of propagated signals, such as carrier waves, infrared signals, etc.

[0117] A machine-readable medium may be any tangible medium that contains or stores a program for or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More detailed examples of machine-readable storage media include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity of description, the specific working processes of the above-described systems, devices, and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or can be electrical, mechanical or other forms of connection.

[0120] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0121] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0122] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0123] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items that have substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity or order of execution. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, a first image may be referred to as a second image, and similarly, a second image may be referred to as a first image. Both the first image and the second image may be images, and in some cases, may be separate and different images.

[0124] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0125] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, "plurality of second images" means two or more second images. The terms "system" and "network" are often used interchangeably herein.

[0126] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0127] It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the listed items. The term "and / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this application generally indicates that the associated objects are in an "or" relationship.

[0128] It will also be understood that the term “comprise” (also known as “includes,” “including,” “comprises,” and / or “comprising”) when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0129] It should also be understood that the terms “if” and “if” may be interpreted to mean “when” or “upon” or “in response to determining” or “in response to detecting.” Similarly, the phrases “if it is determined that ” or “if [stated condition or event] is detected” may be interpreted to mean “upon determining ” or “in response to determining ” or “upon detecting [stated condition or event]” or “in response to detecting [stated condition or event],” depending on the context.

[0130] It should be understood that determining B based on A does not mean determining B based solely on A. B can also be determined based on A and / or other information.

[0131] It should also be understood that references throughout this specification to "one embodiment," "an embodiment," or "one possible implementation" mean that specific features, structures, or characteristics associated with that embodiment or implementation are included in at least one embodiment of the present application. Therefore, the appearance of "in one embodiment," "in an embodiment," or "one possible implementation" throughout this specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image, where the first image is an image of a first object under illumination; Acquire a second image, where the second image is an image of the first object in the absence of light; Inputting the first image and the second image into an enhancement model, and outputting a third image, wherein the third image is an enhanced result image of the second image, and the image quality of the third image is higher than the image quality of the second image; In which, the enhancement model is trained based on a reference number of sample images, images to be enhanced and enhanced result images, the sample images provide feature detail support for the enhancement processing operation of the images to be enhanced, and the enhanced result images are the images after the enhancement processing of the images to be enhanced; the sample images, the images to be enhanced and the enhanced result images contain the same content.

2. The method according to claim 1, wherein The acquiring of the first image comprises: acquiring a fourth image of the first object in a first time period, where the first time period is a period of illumination time at a location where the first object is located; When the average pixel value of the fourth image meets a first threshold, the fourth image is determined to be the first image.

3. The method according to claim 2, characterized in that Also includes: When the average pixel value of the fourth image does not meet the first threshold, acquiring a fifth image of the first object in a second time period; The second time period is another time period in the illumination time period of the location where the first object is located; When the average pixel value of the fifth image meets the first threshold, the fifth image is determined to be the first image.

4. The method according to claim 1, wherein The acquiring of the first image data includes: Obtaining the weather at the location of the first object in a third time period; the third time period is a lighting time period at the location of the first object; In the third time period, when the weather at the location of the first object is specified weather, the first image is acquired.

5. The method according to any one of claims 1 to 4, characterized in that Before inputting the first image and the second image into the enhancement model and outputting the third image, the method further includes: Obtaining an output result image according to the sample image and the image to be enhanced; Comparing the sample image and the enhanced result image in a first dimension to obtain a difference area image; Comparing the output result image, the enhanced result image, and the difference region image in a second dimension to obtain similarity between the output result image and the enhanced result image, wherein the second dimension at least includes the first dimension; When the similarity satisfies a threshold range, the enhanced model is obtained.

6. An image processing device, characterized in that The device includes: an acquisition module and a processing module; wherein, The acquisition module is used to acquire a first image, where the first image is an image of a first object under illumination; The acquisition module is further configured to acquire a second image, where the second image is an image of the first object in the absence of light; The processing module is used to input the first image and the second image into the enhancement model, and output a third image, where the third image is an enhanced result image of the second image, and the image quality of the third image is higher than the image quality of the second image; In which, the enhancement model is trained based on a reference number of sample images, images to be enhanced and enhanced result images, the sample images provide feature detail support for the enhancement processing operation of the images to be enhanced, and the enhanced result images are the images after the enhancement processing of the images to be enhanced; the sample images, the images to be enhanced and the enhanced result images contain the same content.

7. An electronic device, characterized in that: The device includes a memory and a processor; the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor, so that the electronic device implements any one of the methods described in claims 1-5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to enable a computer to implement the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and the computer program / instruction is executed by a processor to enable a computer to implement the method according to any one of claims 1 to 5.

10. A chip, characterized in that: The chip includes a processor configured to call and execute instructions stored in a memory, so that an electronic device equipped with the chip executes the method according to any one of claims 1 to 5.

Images