Image processing method and device, computer readable storage medium and electronic device

Abstract

The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, and relates to the field of image processing technology. The image processing method comprises: obtaining first image data sensed by a first type of image sensing area of ​​an image sensor, and obtaining second image data sensed by a second type of image sensing area of ​​the image sensor; wherein the exposure time of the first type of image sensing area is longer than the exposure time of the second type of image sensing area; performing noise reduction processing on the second image data using the first image data to obtain third image data; and generating an output image based on the first image data and the third image data. The present disclosure can improve the image noise reduction effect.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing method, an image processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the popularity of electronic devices with shooting functions, more and more users use electronic devices to obtain images. However, due to the shooting method or shooting environment of the camera module, the obtained image will have noise. Although some image noise reduction solutions have emerged, there is still a problem of poor noise reduction effect. Summary of the invention

[0003] The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, thereby overcoming the problem of poor image noise reduction effect at least to a certain extent.

[0004] According to a first aspect of the present disclosure, there is provided an image processing method, comprising: acquiring first image data sensed by a first type of image sensing area of ​​an image sensor, and acquiring second image data sensed by a second type of image sensing area of ​​the image sensor; wherein the exposure time of the first type of image sensing area is longer than the exposure time of the second type of image sensing area; performing noise reduction processing on the second image data using the first image data to obtain third image data; and generating an output image based on the first image data and the third image data.

[0005] According to a second aspect of the present disclosure, there is provided an image processing device, comprising: an image data acquisition module, used to acquire first image data sensed by a first type of image sensing area of ​​an image sensor, and to acquire second image data sensed by a second type of image sensing area of ​​the image sensor; wherein the exposure time of the first type of image sensing area is longer than the exposure time of the second type of image sensing area; a noise reduction module, used to perform noise reduction processing on the second image data using the first image data to obtain third image data; and an image generation module, used to generate an output image based on the fusion of the first image data and the third image data.

[0006] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned image processing method is implemented.

[0007] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising a processor; and a memory for storing one or more programs, wherein when the one or more programs are executed by the processor, the processor implements the above-mentioned image processing method.

[0008] In the technical solutions provided by some embodiments of the present disclosure, for the output of the same image sensor, the first image data obtained by a long exposure time can be used to reduce the noise of the second image data obtained by a short exposure time. On the one hand, the solution of the present disclosure utilizes the characteristics of low noise and clear image texture of long exposure data, and reduces the noise of data generated by short exposure based on these data, so as to achieve a fine noise reduction effect. On the other hand, the solution of the present disclosure utilizes the data output by the same image sensor in the noise reduction process, which is easy to implement and low in cost.

[0009] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0011] Figure 1 A schematic diagram showing an exemplary system architecture of an image processing solution according to an embodiment of the present disclosure;

[0012] Figure 2 A schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present disclosure is shown;

[0013] Figure 3 A flowchart schematically shows an image processing method according to an exemplary embodiment of the present disclosure;

[0014] Figure 4 A schematic diagram showing an image sensing area according to an embodiment of the present disclosure is shown;

[0015] Figure 5 A schematic diagram showing an image sensing area according to another embodiment of the present disclosure;

[0016] Figure 6 A flowchart schematically shows the entire process of generating an output image according to an embodiment of the present disclosure;

[0017] Figure 7 A block diagram schematically shows an image processing apparatus according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0019] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0020] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation. In addition, all the terms "first", "second", "third", and "fourth" below are only for the purpose of distinction and should not be used as limitations of the present disclosure.

[0021] Usually, the images captured by the terminal device will have noise, including the noise generated in the photoelectric conversion process of the image sensor. The darker the environment, the greater the noise of the image. This is because the darker the environment, the less light enters, resulting in a smaller signal-to-noise ratio of the image. For ordinary shooting equipment, the noise in the images taken at night is greater than that in the images taken during the day. In night scene shooting, extending the exposure time can reduce some noise, but the exposure time has certain limitations. Too long will reduce the frame rate and may also cause overexposure of the highlight area in the image. Therefore, some soft and hard means are needed to suppress noise together.

[0022] With the development of technology, the exposure method of image sensors has also undergone new changes, and some technologies have emerged that can output long and short exposure images with the same image sensor, such as DolSensor. The exposure time of the odd-numbered rows of these image sensors can be different from the exposure time of the even-numbered rows.

[0023] For images output by such image sensors, it is currently difficult to avoid sacrificing clarity to achieve noise reduction during the noise reduction process, the algorithm complexity is high and the noise reduction effect is poor.

[0024] In view of this, the embodiments of the present disclosure provide a new image processing solution.

[0025] The image sensor targeted by the implementation method of the present disclosure can use different exposure times for alternate rows. For example, the exposure time of the even rows is d, and the exposure time of the odd rows is 2*d. Thus, the even rows of the image generate a short-exposure image, and the odd rows generate a long-exposure image. The height of these two images is half the height of the image produced by normal exposure of the image sensor. The disclosed solution can utilize the characteristics of long-exposure images with low noise and clear texture to provide a reference for finer textures for short-exposure images, reduce noise on the short-exposure images, and then obtain the final image to be output by means of image fusion.

[0026] Figure 1 A schematic diagram showing an exemplary system architecture of an image processing solution according to an embodiment of the present disclosure.

[0027] like Figure 1 As shown, the system architecture may include a terminal device 1100 and a server 1200. The terminal device 1100 and the server 1200 may be connected via a network, and the connection type of the network may include, for example, a wired or wireless communication link or an optical fiber cable.

[0028] It should be understood that the number of terminal devices 1100 and servers 1200 is only illustrative. According to implementation requirements, there may be any number of terminal devices and servers. For example, the server 1200 may be a server cluster composed of multiple servers. The server 1200 may also be referred to as a cloud or cloud server.

[0029] The terminal device 1100 can interact with the server 1200 through the network to receive or send messages, etc. Figure 1 In the figure, a smart phone is used as an example, however, the terminal device 1100 also includes a tablet computer, a smart wearable device, a personal computer and other devices with a shooting function. Among them, the terminal device 1100 can also be called a terminal, a mobile terminal, a mobile terminal, a smart terminal, etc.

[0030] When the image processing process of an exemplary embodiment of the present disclosure is performed by the terminal device 1100, first, the terminal device 1100 can obtain first image data sensed by a first type of image sensing area of ​​the image sensor, and obtain second image data sensed by a second type of image sensing area of ​​the image sensor, wherein the exposure time of the first type of image sensing area is longer than the exposure time of the second type of image sensing area.

[0031] Among them, the first type of image sensing area of ​​the image sensor is the odd-numbered rows of the image sensor, and the second type of image sensing area of ​​the image sensor is the even-numbered rows of the image sensor; or the first type of image sensing area of ​​the image sensor is the even-numbered rows of the image sensor, and the second type of image sensing area of ​​the image sensor is the odd-numbered rows of the image sensor.

[0032] Next, the terminal device 1100 may perform noise reduction processing on the second image data using the first image data to obtain third image data, and generate an output image based on the first image data and the third image data.

[0033] In addition, in the image processing process disclosed herein, brightness correction, image alignment, reduction of the impact of motion on the image, detail enhancement and other means may be combined to further enhance the noise reduction effect and improve the image quality of the output image.

[0034] In the case where the image processing process of an exemplary embodiment of the present disclosure is performed by the server 1200, first, the server 1200 can obtain first image data sensed by a first type of image sensing area of ​​the image sensor and second image data sensed by a second type of image sensing area of ​​the image sensor from the terminal device 1100 or other devices, wherein the exposure time of the first type of image sensing area is longer than the exposure time of the second type of image sensing area.

[0035] Next, the server 1200 can use the first image data to perform noise reduction processing on the second image data to obtain third image data, and generate an output image based on the first image data and the third image data. In addition, the noise reduction effect can be further improved by combining brightness correction, image alignment, reducing the impact of motion on the image, detail enhancement, and other means to improve the image quality of the output image.

[0036] Subsequently, the server 1200 may also feed back the generated output image to the terminal device 1100 for display and / or storage.

[0037] It should be noted that any of the above steps of processing the image can be performed by the terminal device 1100 or the server 1200, and the present disclosure does not limit this. For example, the terminal device 1100 can send the first image data and the second image data to the server 1200, and the server 1200 generates the third image data and sends it to the terminal device 1100, and the terminal device 1100 generates the output image based on the first image data and the third image data.

[0038] Figure 2 Schematic diagram of an electronic device suitable for implementing the exemplary embodiment of the present disclosure is shown. The terminal device of the exemplary embodiment of the present disclosure may be configured as follows Figure 2 It should be noted that Figure 2 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0039] The electronic device of the present disclosure includes at least a processor and a memory, wherein the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the image processing method of the exemplary embodiment of the present disclosure.

[0040] Specifically, Figure 2 As shown, the electronic device 200 may include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a Subscriber Identification Module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0041] It is to be understood that the structure illustrated in the embodiment of the present disclosure does not constitute a specific limitation on the electronic device 200. In other embodiments of the present disclosure, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0042] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. In addition, a memory may be provided in the processor 210 for storing instructions and data.

[0043] The electronic device 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor. In some embodiments, the electronic device 200 may include 1 or N camera modules 291, N is a positive integer greater than 1, and if the electronic device 200 includes N cameras, one of the N cameras is a main camera. In the disclosed solution, at least one of the N cameras is a camera that can realize interlaced unequal exposure, that is, a camera with different interlaced exposure times.

[0044] The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 can include a program storage area and a data storage area. The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.

[0045] The present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device.

[0046] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0047] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0048] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0049] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0050] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0051] The following describes the steps of the image processing method of the present disclosure executed by a terminal device. In this case, the image processing apparatus of the embodiment of the present disclosure can be configured in the terminal device.

[0052] Figure 3 The flowchart of the image processing method according to the exemplary embodiment of the present disclosure is schematically shown. Figure 3 , the image processing method may include the following steps:

[0053] S32. Acquire first image data sensed by a first type of image sensing area of ​​the image sensor, and acquire second image data sensed by a second type of image sensing area of ​​the image sensor; wherein the exposure time of the first type of image sensing area is longer than the exposure time of the second type of image sensing area.

[0054] In the exemplary embodiment of the present disclosure, the image sensing area is an area where the image sensor senses image information, and the purpose of distinguishing between “first category” and “second category” is only to indicate that the two are different in exposure time.

[0055] Specifically, the exposure time of the first type of image sensing area is longer than the exposure time of the second type of image sensing area. For example, the exposure time of the first type of image sensing area is 2*d, and the exposure time of the second type of image sensing area is d. It should be noted that the present disclosure does not limit the specific value of the exposure time.

[0056] According to some embodiments of the present disclosure, the first type of image sensing area of ​​the image sensor is the odd-numbered rows of the image sensor, and the second type of image sensing area of ​​the image sensor is the even-numbered rows of the image sensor. That is, the exposure time of the odd-numbered rows of the image sensor is longer than the exposure time of the even-numbered rows. Figure 4 The corresponding manners of the image sensing areas under these embodiments are schematically shown.

[0057] According to some other embodiments of the present disclosure, the first type of image sensing area of ​​the image sensor is the even-numbered rows of the image sensor, and the second type of image sensing area of ​​the image sensor is the odd-numbered rows of the image sensor. That is, the exposure time of the even-numbered rows of the image sensor is longer than the exposure time of the odd-numbered rows. Figure 5 The corresponding manners of the image sensing areas under these embodiments are schematically shown.

[0058] S34. Use the first image data to perform noise reduction processing on the second image data to obtain third image data.

[0059] In an exemplary embodiment of the present disclosure, a guided filter may be used to implement noise reduction processing on the second image data. Guided filtering refers to filtering another image data through guided image data so that the obtained filtering result is consistent with the guided image data in texture to achieve a noise reduction effect.

[0060] Specifically, the guided image data may be determined based on the first image data, and the guided image data may be used to perform guided filtering on the second image data to obtain the third image data.

[0061] The process of determining the guide image data will be exemplarily described below.

[0062] According to some embodiments of the present disclosure, the first image data may be directly used as the guide image data.

[0063] According to other embodiments of the present disclosure, since there may be a large difference between the image brightness corresponding to the first image data and the image brightness corresponding to the second image data, the terminal device may first perform brightness correction on the first image data to obtain first intermediate image data, and then determine the guide image data based on the first intermediate image data. The image brightness corresponding to the first intermediate image data is consistent with the image brightness corresponding to the second image data.

[0064] First, the brightness correction ratio can be determined based on the exposure time of the first type of image sensing area and the exposure time of the second type of image sensing area. The brightness correction ratio can be expressed as d0*δ / d1, where d0 represents the exposure time of the first type of image sensing area, d1 represents the exposure time of the second type of image sensing area, and δ is a correction factor used to eliminate the nonlinear relationship between image brightness and exposure time.

[0065] Next, the brightness correction ratio may be used to perform brightness correction on the first image data to obtain first intermediate image data.

[0066] If the pixel value of the pixel point (coordinate i, j) represented by the first image data is recorded as Img0, and the pixel value of the pixel point (coordinate i, j) represented by the first intermediate image data is recorded as Img1, then Img1 = Img0*d0*δ / d1. By traversing all the pixel points, the first intermediate image data can be obtained.

[0067] The above process can be applied to scenarios where the first image data and the second image data have not undergone nonlinear transformation (such as gamma correction, etc.). For scenes that have undergone nonlinear processing, the present disclosure can also use brightness alignment algorithms such as Histogram Matching to perform brightness correction, and the present disclosure does not impose any restrictions on this.

[0068] It should be noted that, the determining of the guide image data based on the first intermediate image data described in the present disclosure may include: directly using the first intermediate image data as the guide image data.

[0069] In addition, the method may further process the first intermediate image data and then determine the guide image data.

[0070] According to some other embodiments of the present disclosure, there are texture differences between the first intermediate image data and the second image data, and these differences mainly come from two aspects: on the one hand, there is a certain angle difference between the first type image sensing area and the second type image sensing area of ​​the same image sensor; on the other hand, when there is a moving object in the scene, the different exposure times will cause the position and posture of the moving object in the image to be different. Therefore, the second image data can be used to perform an image alignment operation on the first intermediate image data to obtain the second intermediate image data, and then the guide image data can be determined based on the second intermediate image data.

[0071] Specifically, image alignment may be achieved by, for example, extracting features (such as SIFT features) and matching features, which is not limited in the present disclosure.

[0072] In one embodiment, the second intermediate image data may be used as the guide image data. In another embodiment, the second intermediate image data may be further optimized and then the guide image data may be determined, which is not limited in the present disclosure.

[0073] According to some further embodiments of the present disclosure, the second image data may be directly used to perform an image alignment operation on the first image data to obtain third intermediate image data, and the guide image data may be determined based on the third intermediate image data.

[0074] Specifically, the terminal device may directly determine the third intermediate image data as the guide image data. Alternatively, the terminal device performs brightness correction on the third intermediate image data to obtain fourth intermediate image data, and determines the fourth intermediate image data as the guide image data. Alternatively, the fourth intermediate image data is further optimized and then determined as the guide image data, which is not limited in the present disclosure.

[0075] The processes of image alignment and brightness correction in these embodiments are similar to those in the above embodiments and will not be described in detail.

[0076] Therefore, the terminal device can use the guided image data obtained in any of the above embodiments to perform guided filtering on the second image data to obtain the third image data.

[0077] S36. Generate an output image based on the first image data and the third image data.

[0078] According to some embodiments of the present disclosure, the terminal device may fuse the first image data with the third image data to obtain an output image.

[0079] When the image sensing area corresponds to the rows of the image sensor, the fusion method is interlaced fusion. For example, the first image data is the data collected by the odd rows of the image sensor, and the third image data is the data collected by the even rows of the image sensor and processed. In this case, the first row of data of the output image is the first row of data of the first image data, the second row of data is the first row of data of the third image data, the third row of data is the second row of data of the first image data, the fourth row of data is the second row of data of the third image data, and so on.

[0080] According to some other embodiments of the present disclosure, given that the brightness of the first intermediate image data and the third image data is consistent, the terminal device may fuse the first intermediate image data obtained by brightness correction of the first image data with the third image data to generate an output image. The fusion method still adopts the above-mentioned interlaced fusion method.

[0081] In addition, the second intermediate image data may be fused with the third image data, or the third intermediate image data may be fused with the third image data, or the fourth intermediate image data may be fused with the third image data, to generate an output image.

[0082] In view of the fact that there may be moving objects in the scenes of some embodiments of the present disclosure, when the images are fused, there may be an issue of unnaturalness between image lines. The present disclosure also provides a solution to this problem.

[0083] Firstly, the terminal device may obtain a fused image based on the first image data and the third image data, wherein the fusion method is as described in the above embodiment and will not be described in detail.

[0084] Next, the terminal device can estimate the motion area in the fused image. For example, the difference between the pixel value corresponding to the first intermediate image data and the pixel value corresponding to the second intermediate image data can be calculated and used to obtain a differential image to determine the motion area. The present disclosure does not limit the means of motion estimation.

[0085] Then, the terminal device can calculate the gradient of the pixels of the first type of image sensing area in the motion area as the first gradient, and calculate the gradient of the pixels of the second type of image sensing area adjacent to the first type of image sensing area as the second gradient. Specifically, the gradient can be calculated using operators such as the Canny operator, the Sobel operator, and the Laplace operator, and the present disclosure does not limit this.

[0086] For example, when the first type of image sensing area is an odd-numbered row of the image sensor, the gradient of the pixels in the odd-numbered row can be calculated as the first gradient, and the gradient of the pixels in the adjacent even-numbered row can be calculated as the second gradient.

[0087] After the first gradient and the second gradient are obtained, the first gradient may be compared with the second gradient, and the image data of the motion region may be adjusted based on the comparison result to obtain an adjusted motion region.

[0088] Specifically, a gradient threshold can be configured. If the similarity between the first gradient and the second gradient is less than the gradient threshold, that is, the first gradient and the second gradient are relatively close, it means that the pixel transition here is relatively natural, and no processing is required, or a simple interpolation process is performed. If the similarity between the first gradient and the second gradient is greater than or equal to the gradient threshold, it means that the gradient of the current pixel is greatly different from the surrounding gradients, and the transition is unnatural. The pixel values ​​of these pixels can be adjusted by bilinear interpolation.

[0089] After obtaining the adjusted motion region, the terminal device may fuse the adjusted motion region with the non-motion region of the fused image to generate an output image.

[0090] In addition, after the output image is generated, operations such as detail restoration and artifact elimination may be performed on the output image, which is not limited in the present disclosure.

[0091] The following will refer to Figure 6 The image processing process of one embodiment of the present disclosure is described.

[0092] In step S602, the image sensor with interlaced unequal exposure can output image data. Since the interlaced exposure time of this type of image sensor is different, the first image data sensed by the long exposure area and the second image data sensed by the short exposure area can be obtained.

[0093] In step S604, the terminal device may perform brightness correction on the first image data to obtain brightness-corrected image data.

[0094] In step S606, the terminal device may perform image alignment on the image data after brightness correction to obtain the guide image data. Specifically, the image data after brightness correction may be aligned using the second image data.

[0095] In step S608, the terminal device may use the guided image data to perform guided filtering on the second image data to obtain a third image.

[0096] In step S610, the terminal device may merge the above-mentioned image data after brightness correction with the third image data to generate an output image.

[0097] In addition, the above image processing process uses guided filtering as an example to illustrate the noise reduction process. However, the disclosed solution is not limited thereto, and an image processing method of weighted fusion or superposition of the second intermediate image data and the second image data may also be used to achieve a certain degree of noise reduction effect.

[0098] The image processing scheme of the disclosed embodiment can reduce the noise of short-exposure data by utilizing the characteristics of long-exposure data with low noise and clear image texture, and utilizes data output by the same image sensor, which is easy to implement and low-cost.

[0099] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0100] Furthermore, this exemplary embodiment also provides an image processing device.

[0101] Figure 7 FIG. 1 schematically shows a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure. Figure 7 The image processing device 7 according to an exemplary embodiment of the present disclosure may include an image data acquisition module 71 , a noise reduction module 73 , and an image generation module 75 .

[0102] Specifically, the image data acquisition module 71 can be used to acquire first image data sensed by a first type of image sensing area of ​​the image sensor, and to acquire second image data sensed by a second type of image sensing area of ​​the image sensor; wherein the exposure time of the first type of image sensing area is longer than the exposure time of the second type of image sensing area; the noise reduction module 73 can be used to perform noise reduction processing on the second image data using the first image data to obtain third image data; the image generation module 75 can be used to generate an output image based on the fusion of the first image data and the third image data.

[0103] According to an exemplary embodiment of the present disclosure, the first type of image sensing areas of the image sensor are the odd-numbered rows of the image sensor, and the second type of image sensing areas of the image sensor are the even-numbered rows of the image sensor; or the first type of image sensing areas of the image sensor are the even-numbered rows of the image sensor, and the second type of image sensing areas of the image sensor are the odd-numbered rows of the image sensor.

[0104] According to an exemplary embodiment of the present disclosure, the noise reduction module 73 may be configured to execute: determining guided image data based on the first image data; and performing guided filtering on the second image data using the guided image data to obtain third image data.

[0105] According to an exemplary embodiment of the present disclosure, the process of determining the guide image data by the noise reduction module 73 can be configured to execute: performing brightness correction on the first image data to obtain first intermediate image data; wherein the image brightness corresponding to the first intermediate image data is consistent with the image brightness corresponding to the second image data; and determining the guide image data based on the first intermediate image data.

[0106] According to an exemplary embodiment of the present disclosure, the process of the noise reduction module 73 performing brightness correction on the first image data can be configured to execute: determining a brightness correction ratio based on the exposure time of the first type of image sensing area and the exposure time of the second type of image sensing area; and performing brightness correction on the first image data using the brightness correction ratio.

[0107] According to an exemplary embodiment of the present disclosure, the process of determining the guide image data by the noise reduction module 73 can also be configured to execute: performing an image alignment operation on the first intermediate image data using the second image data to obtain the second intermediate image data; and determining the guide image data based on the second intermediate image data.

[0108] According to an exemplary embodiment of the present disclosure, the process of determining the guide image data by the noise reduction module 73 can also be configured to execute: performing an image alignment operation on the first image data using the second image data to obtain third intermediate image data; and determining the guide image data based on the third intermediate image data.

[0109] According to an exemplary embodiment of the present disclosure, the image generation module 75 may be configured to perform: interlaced merging of first intermediate image data obtained by brightness correction of the first image data with third image data to generate an output image.

[0110] According to an exemplary embodiment of the present disclosure, the image generation module 75 can also be configured to perform: obtaining a fused image based on the first image data and the third image data; estimating the motion area in the fused image; calculating the gradient of the pixels of the first type of image sensing area in the motion area as the first gradient, and calculating the gradient of the pixels of the second type of image sensing area adjacent to the first type of image sensing area as the second gradient; comparing the first gradient with the second gradient, and adjusting the image data of the motion area based on the comparison result to obtain an adjusted motion area; fusing the adjusted motion area with the non-motion area of ​​the fused image to generate an output image.

[0111] Since the functional modules of the image processing device in the embodiment of the present disclosure are the same as those in the above-mentioned method embodiment, they will not be described in detail here.

[0112] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0113] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0114] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0115] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0116] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire first image data sensed by a first type of image sensing area of ​​an image sensor, and acquire second image data sensed by a second type of image sensing area of ​​the image sensor; wherein an exposure time of the first type of image sensing area is longer than an exposure time of the second type of image sensing area; Performing noise reduction processing on the second image data using the first image data to obtain third image data; generating an output image based on the first image data and the third image data; The first type of image sensing area of ​​the image sensor is the odd-numbered rows of the image sensor, and the second type of image sensing area of ​​the image sensor is the even-numbered rows of the image sensor; or the first type of image sensing area of ​​the image sensor is the even-numbered rows of the image sensor, and the second type of image sensing area of ​​the image sensor is the odd-numbered rows of the image sensor; Generating an output image based on the first image data and the third image data includes: Obtaining a fused image based on the first image data and the third image data; estimating a motion region in the fused image; Calculating the gradient of the pixels of the first type of image sensing area in the motion area as a first gradient, and calculating the gradient of the pixels of the second type of image sensing area adjacent to the first type of image sensing area as a second gradient; comparing the first gradient with the second gradient, and adjusting the image data of the motion region based on the comparison result to obtain an adjusted motion region; The adjusted motion region is fused with the non-motion region of the fused image to generate the output image.

2. The image processing method according to claim 1, characterized in that: Performing noise reduction processing on the second image data by using the first image data to obtain third image data, comprising: determining guide image data based on the first image data; The second image data is subjected to guided filtering using the guided image data to obtain the third image data.

3. The image processing method according to claim 2, characterized in that: Determining the guide image data based on the first image data includes: Performing brightness correction on the first image data to obtain first intermediate image data; wherein the image brightness corresponding to the first intermediate image data is consistent with the image brightness corresponding to the second image data; The guide image data is determined based on the first intermediate image data.

4. The image processing method according to claim 3, characterized in that: Performing brightness correction on the first image data includes: determining a brightness correction ratio based on the exposure time of the first type of image sensing area and the exposure time of the second type of image sensing area; The brightness correction is performed on the first image data using the brightness correction ratio.

5. The image processing method according to claim 3, characterized in that: Determining the guide image data based on the first intermediate image data includes: performing an image alignment operation on the first intermediate image data using the second image data to obtain second intermediate image data; The guide image data is determined based on the second intermediate image data.

6. The image processing method according to claim 2, characterized in that: Determining the guide image data based on the first image data includes: performing an image alignment operation on the first image data using the second image data to obtain third intermediate image data; The guide image data is determined based on the third intermediate image data.

7. The image processing method according to claim 3, characterized in that: Generating an output image based on the first image data and the third image data includes: The first intermediate image data obtained by brightness correction of the first image data is merged with the third image data in an interlaced manner to generate an output image.

8. An image processing device, characterized in that: include: an image data acquisition module, configured to acquire first image data sensed by a first type of image sensing area of ​​an image sensor, and acquire second image data sensed by a second type of image sensing area of ​​the image sensor; wherein an exposure time of the first type of image sensing area is longer than an exposure time of the second type of image sensing area; a noise reduction module, configured to perform noise reduction processing on the second image data using the first image data to obtain third image data; An image generation module, configured to generate an output image based on the first image data and the third image data; The first type of image sensing area of ​​the image sensor is the odd-numbered rows of the image sensor, and the second type of image sensing area of ​​the image sensor is the even-numbered rows of the image sensor; or the first type of image sensing area of ​​the image sensor is the even-numbered rows of the image sensor, and the second type of image sensing area of ​​the image sensor is the odd-numbered rows of the image sensor; The image generation module is also configured to obtain a fused image based on the first image data and the third image data; estimate the motion area in the fused image; calculate the gradient of the pixels of the first type of image sensing area in the motion area as the first gradient, and calculate the gradient of the pixels of the second type of image sensing area adjacent to the first type of image sensing area as the second gradient; compare the first gradient with the second gradient, and adjust the image data of the motion area based on the comparison result to obtain an adjusted motion area; fuse the adjusted motion area with the non-motion area of ​​the fused image to generate the output image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; A memory for storing one or more programs, which, when executed by the processor, enables the processor to implement the image processing method according to any one of claims 1 to 7.

Images