Image detection method and electronic device
By using a channel-based computation method combining Bayer array maps and noise maps in high dynamic range image synthesis, ghosting regions can be directly detected, solving the problem of ghosting in dynamic scenes and achieving higher detection accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN GOODIX TECH CO LTD
- Filing Date
- 2022-08-04
- Publication Date
- 2026-07-03
Smart Images
Figure CN115170554B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image detection technology, and more particularly to image detection methods and electronic devices. Background Technology
[0002] High Dynamic Range (HDR) images, compared to Low Dynamic Range (LDR) images, can represent more detail in both bright and dark areas of a scene, playing a crucial role in film, television, medicine, and many other fields. One method for synthesizing HDR images involves capturing multiple LDR images and fusing them to obtain the HDR image. However, this method can produce ghosting artifacts, especially in dynamic scenes, significantly impacting the image quality. Therefore, removing ghosting artifacts from HDR images, particularly in dynamic scenes, is a key consideration in HDR image synthesis. The core of ghosting removal in HDR images lies in ghost region detection—that is, identifying ghosting regions in the LDR images used for HDR image synthesis. Summary of the Invention
[0003] This application provides an image detection method and apparatus that can more accurately detect ghosting regions in LDR images.
[0004] In a first aspect, embodiments of this application provide an image detection method, comprising: acquiring a first Bayer array map and a second Bayer array map; the first Bayer array map is a Bayer array map of a target frame image, and the second Bayer array map is a Bayer array map of a reference frame image; calculating the ghost region of the target frame image in n channels based on the first Bayer array map and the second Bayer array map; n is greater than or equal to 1; and determining the ghost region of the target frame image based on the ghost region of the target frame image in n channels. In this method, ghost region detection is performed directly based on the Bayer array maps of the target frame image and the reference frame image, and ghost regions are detected separately for each channel. This channel-based ghost region calculation improves the robustness of ghost region detection, thereby enabling more accurate detection of ghost regions in LDR images.
[0005] In one possible implementation, calculating the ghost region of the target frame image in n channels based on the first Bayer array map and the second Bayer array map includes: calculating a first noise map of the first Bayer array map; calculating a second noise map of the second Bayer array map; and calculating the ghost region of the target frame image in n channels based on the first Bayer array map, the second Bayer array map, the first noise map, and the second noise map. This method combines the influence of noise with the Bayer array maps of the target frame image and the reference frame image, thereby improving the detection accuracy of the ghost region and achieving a better ghost region detection effect.
[0006] In one possible implementation, calculating the ghost region of the target frame image in n channels based on the first Bayer array map, the second Bayer array map, the first noise map, and the second noise map includes:
[0007] Calculate a first array map of the target frame image in the first channel based on the first Bayer array map; calculate a third noise map of the target frame image in the first channel based on the first noise map; calculate a second array map of the reference frame image in the first channel based on the second Bayer array map; calculate a fourth noise map of the reference frame image in the first channel based on the second noise map; the first channel is any one of the n channels;
[0008] A first difference map of the target frame image relative to the reference frame image in the first channel is determined based on the first array map, the second array map, the third noise map, and the fourth noise map;
[0009] Calculate the first threshold of the first difference map;
[0010] The ghost region in the first difference map is determined based on the first threshold.
[0011] The ghost region of the target frame image in the first channel is determined based on the ghost region in the first difference map.
[0012] In one possible implementation, before determining the first difference map of the target frame image relative to the reference frame image in the first channel, the method further includes:
[0013] The first array image, the second array image, the third noise image, and the fourth noise image are respectively subjected to noise reduction processing to obtain the third array image of the target frame image in the first channel, the fourth array image of the reference frame image in the first channel, the fifth noise image of the target frame image in the first channel, and the sixth noise image of the reference frame image in the first channel.
[0014] Determining the first difference map of the target frame image relative to the reference frame image in the first channel based on the first array map, the second array map, the third noise map, and the fourth noise map includes:
[0015] The first difference map of the target frame image relative to the reference frame image in the first channel is determined based on the third array map, the fourth array map, the fifth noise map, and the sixth noise map.
[0016] In one possible implementation, the first channel is an R channel or a B channel, and before determining the first difference map of the target frame image relative to the reference frame image in the first channel, the method further includes:
[0017] The resolutions of the first array image, the second array image, the third noise image, and the fourth noise image are respectively converted to the resolution of the target frame image.
[0018] In one possible implementation, calculating the first threshold of the first difference map includes:
[0019] Cluster the pixel values of the pixels in the first difference map to obtain at least two classes of pixel values;
[0020] Calculate the mean distribution of pixel values for each category;
[0021] The first threshold is determined based on the mean of the distribution.
[0022] In one possible implementation, determining the first threshold based on the distribution mean includes: selecting the distribution mean from the distribution mean whose size arrangement is in a preset order as the first threshold.
[0023] In one possible implementation, determining the ghost region in the first difference map based on the first threshold includes:
[0024] The region formed by pixels whose pixel values are greater than the first threshold in the first difference map is defined as the ghost region in the first difference map.
[0025] In one possible implementation, converting the resolutions of the first array map, the second array map, the third noise map, and the fourth noise map to the resolution of the target frame image includes:
[0026] The first array image, the second array image, the third noise image, and the fourth noise image are upsampled by a preset multiple, respectively.
[0027] In one possible implementation, the noise reduction processing of the first array map, the second array map, the third noise map, and the fourth noise map includes:
[0028] Gaussian blurring with a first convolution kernel is performed on the first array image, the second array image, the third noise image, and the fourth noise image, respectively.
[0029] In one possible implementation, determining the first difference map of the target frame image relative to the reference frame image in the first channel based on the third array map, the fourth array map, the fifth noise map, and the sixth noise map includes:
[0030] The first difference map D is calculated using the following formula:
[0031]
[0032] in, This indicates that Gaussian blurring is applied using the second convolution kernel. This represents the third array diagram of the target frame image in the first channel. This represents the fourth array diagram of the target frame image in the first channel. This represents the fifth noise map of the target frame image in the first channel. c1 represents the sixth noise map of the target frame image in the first channel, where c1 represents the coefficient.
[0033] In one possible implementation, when n equals 1, determining the ghost region of the target frame image based on the ghost region of the target frame image in n channels includes:
[0034] The ghost region of the target frame image in one channel is defined as the ghost region of the target frame image.
[0035] In one possible implementation, when n is greater than 1, determining the ghost region of the target frame image based on the ghost region of the target frame image in n channels includes:
[0036] The ghost regions of the target frame image in n channels are merged to obtain the ghost region of the target frame image.
[0037] In a second aspect, embodiments of this application provide an electronic device, including: a processor; a memory; and one or more computer programs, wherein the computer programs are stored in the memory, and when executed by the processor, the electronic device performs the method described in any of the first aspects.
[0038] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method of any one of the first aspects.
[0039] Fourthly, this application provides a computer program that, when executed by a computer, performs the method of any one of the first aspects.
[0040] In one possible design, the program in the fourth aspect can be stored wholly or partially on a storage medium packaged with the processor, or it can be stored wholly or partially on a memory not packaged with the processor. Attached Figure Description
[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of this application;
[0043] Figure 2 This is a schematic flowchart of an image detection method provided in an embodiment of this application;
[0044] Figure 3 The embodiments of this application are based on Figure 2 A schematic diagram of one of the steps provided in the diagram;
[0045] Figure 4 This is a schematic diagram of the structure of a filter template provided in an embodiment of this application;
[0046] Figure 5 Another structural schematic diagram of the electronic device provided in the embodiments of this application;
[0047] Figure 6 A comparison image of HDR images obtained by fusing images obtained through and without the image detection method of this application, provided as an embodiment of the present application. Detailed Implementation
[0048] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.
[0049] A single LDR image cannot display the full dynamic range of a scene, and it easily loses details in low- or high-brightness areas in scenes with large brightness variations. HDR images, compared to LDR images, can represent more details in both bright and dark areas of a scene, playing a crucial role in film, media, medicine, and many other fields. One method of HDR image synthesis involves capturing multiple LDR images and fusing them to obtain the HDR image. However, this method can produce ghosting artifacts, especially in dynamic scenes, resulting in severe ghosting and affecting the quality of the HDR image. Therefore, removing ghosting artifacts from HDR images, particularly in dynamic scenes, is a key consideration in HDR image synthesis. The core of HDR image ghosting removal is ghost region detection, i.e., detecting ghost regions in the LDR image used for HDR image synthesis. In one example, ghost regions in the LDR image can be detected first, and then the ghost regions in the HDR image can be determined based on these ghost regions.
[0050] It should be noted that when synthesizing HDR images using multiple LDR images, one LDR image is generally selected as the reference LDR image, and the other LDR images can be referred to as the target LDR images. The detection of ghosting regions in the LDR images used for HDR image synthesis can be performed by detecting ghosting regions in each target LDR image.
[0051] For ease of description, in the following embodiments, the target frame LDR image is simply referred to as the target frame image, and the reference frame LDR image is simply referred to as the reference frame image.
[0052] Ghosting regions in a target frame image refer to areas in the target frame image that have changed relative to a reference frame image. Specifically, they can be regions formed by pixels in the target frame image whose pixel values have changed relative to the corresponding pixels in the reference frame image.
[0053] For ghosting region detection, a common algorithm is the pixel comparison method. In an example of the pixel comparison method, a joint histogram of pixel values in the reference frame image and the target frame image can be calculated. The histogram is then converted into a joint probability, and regions with a joint probability less than a fixed threshold are identified as ghosting regions in the HDR image. The pixel comparison method is simple and easy to implement, but the error rate of the detection results is high.
[0054] The image detection method provided in this application embodiment can more accurately detect ghosting regions in LDR images.
[0055] The image detection method provided in this application can be applied to electronic devices, such as mobile phones, tablet computers (PADs), personal computers (PCs), wearable devices, etc.
[0056] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. See also... Figure 1 The electronic device 100 may include: a processor 110, a memory 120, and a camera module 130.
[0057] Optionally, to further enhance the functionality of the electronic device 100, the electronic device 100 may also include: an antenna, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, a headphone jack, etc., which are not limited in the embodiments of this application.
[0058] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.
[0059] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0060] The memory 120 can be used to store computer executable program code, which includes instructions. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.). The data storage area may store data created during the use of the electronic device 600 (such as audio data, phonebook, etc.). Furthermore, the memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the memory 120 and / or instructions stored in memory disposed within the processor.
[0061] The camera module 130 is used to capture still images or videos. The camera module 130 may include a pixel array, a signal readout circuit, an ISP, and an interface circuit. The pixel array collects the light signal returned by the imaged object and converts it into an electrical signal. The strength of the electrical signal reflects the light image of the imaged object. The signal readout circuit reads the electrical signal output by each pixel. The ISP performs analog-to-digital conversion on the electrical signal output by the pixel array and outputs the image data of the imaged object. The interface circuit transmits the image data externally (e.g., to the processor 110 of the electronic device 100). Each pixel in the pixel array has a similar structure, and typically each pixel structure may include a lens (or microlens), a color filter, and a photosensitive element. The lens is located above the color filter, and the color filter is located above the photosensitive element. Light returning from the imaged object is focused by the lens, emitted from the lens exit area, filtered by the color filter, and then enters a photosensitive element such as a photodiode (PD), where the photosensitive element converts the light signal into an electrical signal. Depending on the type of light that different filters can transmit, pixels can include red pixels (hereinafter referred to as R pixels), green pixels (hereinafter referred to as G pixels), and blue pixels (hereinafter referred to as B pixels). R pixels refer to pixels that, after being filtered by the filter, only red light enters the photosensitive element. The principle of G pixels and B pixels is the same as that of R pixels, and will not be described in detail here.
[0062] To acquire color images, color filters with a specific color arrangement are placed above the array of photosensitive elements included in the pixel array; these can also be called color filter arrays (CFAs). Currently, for most pixel arrays, such as CCD and CMOS image sensors, the CFAs used employ the Bayer format based on the RGB three primary colors, also known as Bayer arrays. The Bayer format is characterized by its basic unit being a 2×2 four-pixel array, including one red pixel R, one blue pixel B, and two green pixels G, where the two green pixels G are arranged adjacent to each other at the same vertex.
[0063] In some embodiments, the electronic device 100 may include one or N camera modules 130, where N is a positive integer greater than 1. In some embodiments, the ISP in the camera module 130 may also be disposed in the electronic device 100 as a component of the processor 110.
[0064] In some embodiments, the electronic device 100 may not include the camera module 130, in which case the camera module can be externally connected to the electronic device through an interface to provide raw image data to the electronic device.
[0065] The software system in the electronic device 100 of this application embodiment can use a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. Taking Android as an example, the electronic device 100 uses... Taking the Android system as an example, it can be divided into five layers, from top to bottom: the application layer, the application framework layer, the system runtime library layer, the hardware abstraction layer, and the kernel layer; among them,
[0066] The application layer can include applications, such as camera applications.
[0067] The application framework layer provides application programming interfaces and programming frameworks for applications in the application layer, including predefined functions.
[0068] The system runtime library layer can include the Android runtime and system libraries.
[0069] The hardware abstraction layer is the interface layer located between the operating system kernel and the hardware circuitry.
[0070] The kernel layer is the layer between hardware and software, and can include camera drivers.
[0071] The image detection method of this application embodiment can be located at the application layer as a function provided in an application with shooting function (e.g., a camera application), or it can be located at the application framework layer as a function provided by the operating system of the electronic device that can be called by the aforementioned application with shooting function (e.g., a camera application).
[0072] Taking a camera application as an example, if the image data processing method of this application is provided as a function in the camera application, when the user clicks the shooting control on the image shooting interface provided by the camera application to instruct the camera application to shoot an image, the camera application can be triggered to execute the image detection method of this application during the process of generating an HDR image, to detect ghosting areas in the LDR image, and then further generate an HDR image based on the detection results, and display the HDR image to the user as the shooting result; similarly, when the camera application is triggered by the user to shoot video, the camera application can also execute the image detection method of this application during the process of processing video data, to detect ghosting areas in the LDR image, and then further generate an HDR image based on the detection results, and display the HDR image to the user as a frame of the video.
[0073] Taking a camera application as an example, if the image data processing method in this embodiment is a function provided by the operating system of an electronic device that can be called by the application, then when the user clicks the shooting control on the image shooting interface provided by the camera application to instruct the camera application to shoot an image, the camera application can be triggered to call this function during the process of generating an HDR image from an LDR image. Based on the detection results obtained by the function, an HDR image is generated and displayed to the user as the shooting result. Similarly, when the camera application is triggered by the user to shoot video, the camera application can also call this function during the process of processing video data to detect ghosting areas in the LDR image, and then generate an HDR image based on the detection results obtained by the function. The HDR image is then displayed to the user as a frame of the video.
[0074] In the image data processing method of this application embodiment, the raw image data captured by the camera module can be obtained from the camera driver located in the kernel layer. The raw image data can be an electrical signal converted from the light signal obtained by the camera module based on the RGB sensor, and then converted into digital image data; or, the raw image data can be an electrical signal converted from the light signal obtained by the camera module based on the RGB sensor, in which case the ISP in the electronic device needs to convert the electrical signal into digital image data. The pixels of each frame in the digital image data correspond to the pixels at the same position in the pixel array of the camera module, and it includes only one color. The color type is the same as the pixel type of the corresponding pixel in the pixel array. For example, if a pixel in the pixel array is an R pixel, then the pixel corresponding to that pixel in the image is also an R pixel, and that pixel only has an R value. Since the image data processing method of this application embodiment uses a Bayer array as an example, each frame of image data in the digital image data is referred to as a Bayer array map in this application embodiment.
[0075] The image detection method of this application embodiment will be described in more detail below with reference to the hardware and software structure of the above-mentioned electronic device.
[0076] Figure 2 This is a flowchart illustrating one embodiment of the image detection method of this application. The method can be executed by an electronic device, specifically by a processor in the electronic device, or more specifically by an image capturing application in the electronic device, such as a camera application, or by the image capturing application calling relevant services provided by the operating system of the electronic device.
[0077] like Figure 2 As shown, the method may include:
[0078] Step 201: Obtain the first Bayer array map of the target frame image and the second Bayer array map of the reference frame image.
[0079] The target frame image in this step can be any one of the one or more target frame images used for HDR image merging.
[0080] The target frame image is an LDR image generated based on the first Bayer array image, and the target frame image and the first Bayer array image have the same resolution; the reference frame image is an LDR image generated based on the second Bayer array image, and the reference frame image and the second Bayer array image have the same resolution. The resolution of the first Bayer array image and the second Bayer array image can be the same.
[0081] Optionally, in this step, the first Bayer array pattern and the second Bayer array pattern can be obtained from the camera driver.
[0082] In this step, after the upper-layer application obtains the multi-frame LDR images generated based on the Bayer array diagram, it then re-obtains the corresponding first Bayer array diagram and second Bayer array diagram from the camera driver based on the LDR images to be processed (i.e., the target frame image and reference frame image mentioned above).
[0083] Step 202: Calculate the ghost region of the target frame image in n channels based on the first Bayer array diagram and the second Bayer array diagram; n is greater than or equal to 1.
[0084] Optionally, the above n channels may include R channels, and / or B channels, and / or Y channels.
[0085] In one possible implementation, this step may include:
[0086] Calculate the first noise map of the first Bayer array diagram, and calculate the second noise map of the second Bayer array diagram;
[0087] The ghost region of the target frame image in n channels is calculated based on the first Bayer array diagram, the second Bayer array diagram, the first noise diagram, and the second noise diagram.
[0088] Optionally, the resolutions of the first Bayer array map, the second Bayer array map, the first noise map, and the second noise map can be the same.
[0089] The pixel value of each pixel in the first noise map is used to record the noise value of the corresponding pixel in the first Bayer array map; the pixel value of each pixel in the second noise map is used to record the noise value of the corresponding pixel in the second Bayer array map.
[0090] The noise map of the Bayer array diagram described above can be calculated using relevant noise map calculation methods, and the specific implementation method is not limited in the embodiments of this application.
[0091] In the above implementation, the ghost region of the target frame image is calculated in at least one channel based on the Bayer array diagram of the target frame image and the reference frame image, combined with the influence of noise, thereby achieving a better ghost region detection effect.
[0092] Step 203: Determine the ghost region of the target frame image based on the ghost region of the target frame image in n channels.
[0093] If n = 1, this step may include:
[0094] The ghost region of the target frame image in one channel is defined as the ghost region of the target frame image.
[0095] If n>1, this step may include:
[0096] The ghost regions of the target frame image in n channels are merged to obtain the ghost regions of the target frame image.
[0097] Taking n channels, including Y channel, R channel and B channel, as an example, the calculation formula is as follows: G = IU(G Y G R G B ); where G represents the ghost region of the target frame image, G Y G represents the ghosting region in the Y channel of the target frame image. R G represents the ghosting region of the target frame image in the R channel. B This indicates the ghosting region of the target frame image in the B channel, and IU represents the image and operation.
[0098] Figure 2In the method shown, the ghost region of the target frame image in at least one channel is calculated based on the Bayer array map of the target frame image and the reference frame image. Based on the ghost region of the target frame image in n channels, the ghost region of the target frame image is determined. The ghost region is detected directly based on the Bayer array map of the target frame image and the reference frame image. Furthermore, the ghost region is detected separately according to each channel. By calculating the ghost region by channel, the robustness of the ghost region detection is improved, thus achieving a relatively better ghost region detection effect.
[0099] Furthermore, the ghost region of the target frame image in n channels can be calculated based on the first Bayer array map, the second Bayer array map, the first noise map, and the second noise map. That is, the influence of noise is combined on the basis of the Bayer array maps of the target frame image and the reference frame image, so that the noise information under different illumination is added to the calculation process of the image difference in each channel, thereby improving the detection accuracy of the ghost region and achieving a better ghost region detection effect.
[0100] The specific implementation of calculating the ghost region of the target frame image in n channels based on the first Bayer array map, the second Bayer array map, the first noise map, and the second noise map in step 202 is illustrated by an example.
[0101] If the Y channel is included among the n channels, then the ghost region of the target frame image in the Y channel can be calculated based on the first Bayer array map, the second Bayer array map, the first noise map, and the second noise map. Figure 3 For the implementation of the process shown, please refer to [link / reference]. Figure 3 It can include:
[0102] Step 301: Calculate the first Bayer array diagram I respectively. alt First array diagram and second Bayer array diagram I under the Y channel ref The second array plot and the first noise plot σ under the Y channel alt The third and second noise plots under the Y channel σ ref The fourth noise plot in the Y channel.
[0103] Optionally, the filter template M of the Y channel can be preset, for example... Figure 4 As shown, the first array pattern Y of the first Bayer array pattern in the Y channel is calculated using this filter template. alt The second Bayer array diagram in the Y channel. ref The first noise map and the third noise map in the Y channel. The second noise map is the fourth noise map under the Y channel. The specific formula is as follows:
[0104] Y alt =M(Ialt )
[0105] Y ref =M(I ref )
[0106]
[0107]
[0108] It should be noted that, Figure 4 The filter template shown is only an example. Other filter modules can also be used to implement it. This application does not limit the implementation, as long as it can filter out the information of each image in the Y channel.
[0109] Optionally, the resolution of the first array plot and the first Bayer array plot can be the same, the resolution of the second array plot and the second Bayer array plot can be the same, the resolution of the first noise plot and the third noise plot can be the same, and the resolution of the second noise plot and the fourth noise plot can be the same; while the resolutions of the first Bayer array plot, the second Bayer array plot, the first noise plot and the second noise plot can be the same, therefore, the resolutions of the first array plot, the second array plot, the third noise plot and the fourth noise plot can be the same.
[0110] Step 302: Perform the first array diagram Y on each side. alt Second array diagram Y ref Third noise diagram and the fourth noise map After noise reduction processing, the third array pattern of the first Bayer array pattern in the Y channel is obtained. The second Bayer array diagram is the fourth array diagram under the Y channel. The fifth noise map under the Y channel of the first noise map. The second noise map is the sixth noise map under the Y channel.
[0111] Optionally, the noise reduction process in this step can be achieved by using Gaussian blurring with a first convolution kernel. For example, the first convolution kernel could be a 5x5 convolution kernel. In this case, the calculation formula for the noise reduction of each image in this step is as follows:
[0112]
[0113]
[0114]
[0115]
[0116] in, Used to indicate Gaussian blurring using the first convolution kernel.
[0117] The specific implementation of noise reduction processing for each image in this step will not be described in the embodiments of this application.
[0118] Optionally, the resolutions of the first and third array images can be the same, the resolutions of the second and fourth array images can be the same, the resolutions of the third and fifth noise images can be the same, and the resolutions of the fourth and sixth noise images can be the same; and the resolutions of the first, second, third, and fourth noise images can be the same, therefore, the resolutions of the third, fourth, fifth, and sixth noise images can be the same.
[0119] Step 302 is an optional step.
[0120] Step 303: According to the third array diagram Fourth array diagram Fifth noise diagram and the sixth noise diagram Determine the first difference map D of the target frame image in the Y channel. Y .
[0121] The first difference image is used to record the differences between the third and fourth array images. Optionally, the pixel value of each pixel in the first difference image is used to record the difference value between the corresponding pixel in the third array image and the corresponding pixel in the fourth array image.
[0122] Optionally, this step can use the following formula to calculate the first difference map D. Y :
[0123]
[0124] in, This indicates that Gaussian blur processing is performed using a second convolution kernel. Optionally, the second convolution kernel used in this step for Gaussian blur processing can be a 10*10 convolution kernel.
[0125] It should be noted that the Gaussian blurring process in this step... The second convolution kernel used and the first convolution kernel used in the Gaussian blurring process in step 302 can be the same or different, and this application embodiment does not limit them.
[0126] It should be noted that when using the above formula to calculate the first difference graph D... Y It can be calculated pixel by pixel; specifically, a third array image can be used. Fourth array diagram Fifth noise diagram and the sixth noise diagram Substituting the pixel values of pixels at the same position into the above formula, the first difference map D is calculated. Y The pixel values of pixels at the same position in the image. At this point, the first difference image D... Y Third array diagram Fourth array diagram Fifth noise diagram and the sixth noise diagram The resolutions can be the same.
[0127] In this step, noise information under different illumination conditions is added when calculating the first difference map. For pixels with high noise, the probability of the pixel being detected as a ghost region is reduced, and for pixels with low noise, the probability of the pixel being detected as a ghost region is increased. This dynamically considers the ghost region detection probability of pixels under different noise conditions, thereby improving the detection accuracy of ghost regions under low light conditions and enhancing the detection accuracy and robustness of ghost regions.
[0128] Step 304: Calculate the first threshold of the first difference map.
[0129] The first threshold calculated in this step is used to divide the first difference map into regions. Specifically, in subsequent steps, regions outside the static regions in the first difference map (hereinafter referred to as non-static regions) can be divided into ghost regions based on the first threshold.
[0130] In one possible implementation, this step may include:
[0131] Calculate the mean distribution of pixels in the first difference map;
[0132] Use the mean of this distribution as the first threshold.
[0133] To make the first threshold more accurate, and thus the ghosting region obtained in subsequent processing more accurate, this step may include, in another possible implementation:
[0134] Cluster the pixel values of the pixels in the first difference map to obtain at least two classes of pixel values;
[0135] Calculate the mean distribution of pixel values for each category;
[0136] The first threshold is determined based on the distribution mean.
[0137] When determining the first threshold based on the distribution mean, it can be based on the method for dividing ghost regions in the clustering described above. This allows subsequent steps to determine the ghost regions in the first difference map based on the first threshold. For example, if the pixel values in the first difference map are clustered into two classes, resulting in the distribution mean of the two classes of pixel values, the two distribution means can divide the first difference map into three regions. These three regions can be named, in ascending order of pixel value, as: static region, dynamic region, and overexposed region. Since both the dynamic region and the overexposed region are non-static regions, the smaller distribution mean of the two classes of pixel values can be determined as the first threshold.
[0138] If the pixel values of the pixels in the first difference map are clustered into 3 or more classes, the first difference map can be divided into 4 or more regions. At this time, the region with the smallest pixel value can be determined as the static region according to the actual application. Correspondingly, the distribution mean can be determined as the first threshold based on the division criteria of the static region.
[0139] Optionally, the above clustering can be implemented using a pre-defined Gaussian Mixture Model (GMM). The pre-defined Gaussian Mixture Model is used to cluster the pixel values of pixels in the first difference map. The pre-defined Gaussian Mixture Model can be pre-built; the method for building the model and its specific implementation will not be detailed in this embodiment.
[0140] For example, in this step, the first difference map D can be... Y All pixel values are clustered into two classes using a preset Gaussian mixture model. The mean values of the pixel values in the two classes are calculated and ranked from smallest to largest as follows: Then, the first difference diagram D can be... Y Medium pixel value less than The area composed of pixels is divided into static areas, and the pixel values are in and The region formed by pixels between a certain value is divided into dynamic regions, where the pixel value is greater than a certain value. The region consisting of pixels is defined as the overexposed region. Since the overexposed region will not be merged into the final HDR image during the final fusion of multi-frame LDR images, the dynamic region and the overexposed region can be uniformly classified as the ghost region. Therefore, in this step, the smallest distribution mean among the three distribution means can be selected. It was determined to be the first threshold.
[0141] The above calculation of the distribution mean of pixel values can be achieved using relevant methods for calculating the distribution mean, which will not be elaborated further in the embodiments of this application.
[0142] Step 305: Determine the ghost region in the first difference map based on the first threshold, and determine the ghost region of the target frame image in the Y channel based on the ghost region in the first difference map.
[0143] Optionally, the region consisting of pixels with pixel values greater than a first threshold in the first difference map can be defined as the ghost region of the first difference map.
[0144] Since the resolution of the first difference map is the same as that of the target frame image, once the ghost region of the first difference map is determined, the region at the same position in the target frame image can be determined as the ghost region of the target frame image in the Y channel.
[0145] If the n channels include the R channel, then the specific implementation of calculating the ghost region of the target frame image in the R channel based on the first Bayer array map, the second Bayer array map, the first noise map, and the second noise map in step 202 can be found in [reference needed]. Figure 3 The main difference in the steps shown is that... Figure 3 In steps 301 to 305, the Y channel is replaced with the R channel.
[0146] It should be noted that if n channels simultaneously include both Y and R channels, since the resolution of the first Bayer array image in the Y channel is higher than that in the R channel, similarly, there are also resolution differences between the second array image in the Y channel and the second array image in the R channel, the third noise image in the Y channel and the third noise image in the R channel, and the fourth noise image in the Y channel and the fourth noise image in the R channel. To ensure that the corresponding images have the same resolution, and thus ensure that the first difference image in the Y channel and the first difference image in the R channel obtained after subsequent processing have the same resolution, it is necessary to ensure that... To merge the ghosting regions in multiple channels, in the method for calculating the ghosting region of the target frame image in the R channel, after calculating the first Bayer array image in the R channel, the second Bayer array image in the R channel, the third noise image in the R channel, and the fourth noise image in the R channel, the first Bayer array image in the R channel, the second Bayer array image in the R channel, the third noise image in the R channel, and the fourth noise image in the R channel can be upsampled respectively, so that the resolution of the above images is the same as the resolution of the corresponding image in the Y channel. Optionally, the above upsampling can be achieved by performing bilinear interpolation on the above images respectively.
[0147] If the n channels include the B channel, then the specific implementation of calculating the ghost region of the target frame image in the B channel based on the first Bayer array map, the second Bayer array map, the first noise map, and the second noise map in step 202 can be found in [reference needed]. Figure 3 The main difference in the steps shown is that... Figure 3In steps 301 to 305, the Y channel is replaced with the B channel.
[0148] For reasons similar to those of the R channel, if the n channels simultaneously include the Y channel and the B channel, in the above method for calculating the ghost region of the target frame image in the B channel, after calculating the first array map, the second array map, the third noise map, and the fourth noise map in the B channel, the above images can be upsampled respectively, so that the resolution of the above images is the same as the resolution of the corresponding image in the Y channel.
[0149] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device 400 may include at least one of the following modules: a Y-channel ghost region detection module 410, an R-channel ghost region detection module 420, and a B-channel ghost region detection module 430. The electronic device 400 may also include a ghost region merging module 440.
[0150] The Y-channel ghost region detection module 410 is used to calculate the ghost region of the target frame image in the Y channel;
[0151] The R-channel ghost region detection module 420 is used to calculate the ghost region of the target frame image in the R channel;
[0152] The B-channel ghost region detection module 430 is used to calculate the ghost region of the target frame image in the B channel;
[0153] The ghost region merging module 440 is used to determine the ghost region of the target frame image based on the ghost region of the target frame image in the Y channel, and / or in the R channel, and / or in the B channel.
[0154] Figure 5 For more specific implementations of the modules in the embodiments shown, please refer to the above-described image detection method embodiments; these will not be repeated in the embodiments of this application.
[0155] like Figure 6 As shown, the left image is an example of an HDR fusion image after ghosting is removed using the ghost detection method of this application embodiment, and the right image is an example of an HDR fusion image without using the image detection method of this application embodiment;
[0156] The comparison shows that the left HDR image has less ghosting than the right HDR image. Specifically, there is almost no ghosting in the left HDR image, which verifies that the image detection method of this application can effectively reduce ghosting in HDR images and further illustrates the good performance of this application in detecting ghosting regions in LDR images.
[0157] This application also provides an electronic device, including a processor and a memory, wherein the processor is used to execute the method provided in any embodiment of this application.
[0158] This application also provides an electronic device, which includes a storage medium and a central processing unit. The storage medium may be a non-volatile storage medium, and a computer-executable program is stored in the storage medium. The central processing unit is connected to the non-volatile storage medium and executes the computer-executable program to implement the method provided in any embodiment of this application.
[0159] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method provided in any embodiment of this application.
[0160] This application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform the method provided in any embodiment of this application.
[0161] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.
[0162] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0163] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0164] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0165] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.
Claims
1. An image detection method characterized by, include: Obtain the first Bayer array diagram and the second Bayer array diagram; The first Bayer array diagram is a Bayer array diagram of a target frame image, which is obtained by converting the electrical signal output by the camera module when capturing the target frame image. The target frame image is an LDR image generated based on the first Bayer array diagram. The second Bayer array diagram is a Bayer array diagram of a reference frame image, which is obtained by converting the electrical signal output by the camera module when capturing the reference frame image. The reference frame image is an LDR image generated based on the second Bayer array diagram. Calculate the first noise map of the first Bayer array map; Calculate the second noise map of the second Bayer array map; The ghost region of the target frame image in n channels is calculated based on the first Bayer array diagram, the second Bayer array diagram, the first noise diagram, and the second noise diagram; n is greater than or equal to 1; The ghost region of the target frame image is determined based on the ghost region of the target frame image in n channels; The step of calculating the ghost region of the target frame image in n channels based on the first Bayer array map, the second Bayer array map, the first noise map, and the second noise map includes: Calculate a first array map of the target frame image in the first channel based on the first Bayer array map; calculate a third noise map of the target frame image in the first channel based on the first noise map; calculate a second array map of the reference frame image in the first channel based on the second Bayer array map; calculate a fourth noise map of the reference frame image in the first channel based on the second noise map. The first array image, the second array image, the third noise image, and the fourth noise image are respectively subjected to noise reduction processing to obtain the third array image of the target frame image in the first channel, the fourth array image of the reference frame image in the first channel, the fifth noise image of the target frame image in the first channel, and the sixth noise image of the reference frame image in the first channel; the first channel is any one of the n channels. A first difference map of the target frame image relative to the reference frame image in the first channel is determined based on the third array map, the fourth array map, the fifth noise map, and the sixth noise map; Calculate the first threshold of the first difference map; The ghost region in the first difference map is determined based on the first threshold. The ghost region of the target frame image in the first channel is determined based on the ghost region in the first difference map; Determining the first difference map of the target frame image relative to the reference frame image in the first channel based on the third array map, the fourth array map, the fifth noise map, and the sixth noise map includes: The image calculated based on the third array map, the fourth array map, the fifth noise map, and the sixth noise map is subjected to Gaussian blurring using a second convolution kernel to obtain a first difference map of the target frame image relative to the reference frame image in the first channel.
2. The method according to claim 1, characterized in that, The first channel is an R channel or a B channel. Before determining the first difference map of the target frame image relative to the reference frame image in the first channel, the method further includes: The resolutions of the first array image, the second array image, the third noise image, and the fourth noise image are respectively converted to the resolution of the target frame image.
3. The method according to claim 1 or 2, characterized in that, The calculation of the first threshold of the first difference map includes: Cluster the pixel values of the pixels in the first difference map to obtain at least two classes of pixel values; Calculate the mean distribution of pixel values for each category; The first threshold is determined based on the mean of the distribution.
4. The method of claim 3, wherein, Determining the first threshold based on the distribution mean includes: selecting the distribution mean whose size arrangement is in a preset order from the distribution mean as the first threshold.
5. The method according to claim 1 or 2, characterized in that, Determining the ghost region in the first difference map based on the first threshold includes: The region formed by pixels whose pixel values are greater than the first threshold in the first difference map is defined as the ghost region in the first difference map.
6. The method of claim 2, wherein, The step of converting the resolutions of the first array image, the second array image, the third noise image, and the fourth noise image to the resolution of the target frame image includes: The first array image, the second array image, the third noise image, and the fourth noise image are upsampled by a preset multiple, respectively.
7. The method according to claim 1 or 2, characterized in that, The noise reduction processing of the first array image, the second array image, the third noise image, and the fourth noise image includes: Gaussian blurring with a first convolution kernel is performed on the first array image, the second array image, the third noise image, and the fourth noise image, respectively.
8. The method of claim 1, wherein, The step of performing Gaussian blur processing on the image calculated based on the third array map, the fourth array map, the fifth noise map, and the sixth noise map using a second convolution kernel to obtain a first difference map of the target frame image relative to the reference frame image in the first channel includes: The first difference map D is calculated using the following formula: ； in, This indicates that Gaussian blurring is applied using the second convolution kernel. This represents the third array diagram of the target frame image in the first channel. This represents the fourth array diagram of the target frame image in the first channel. This represents the fifth noise map of the target frame image in the first channel. This represents the sixth noise map of the target frame image in the first channel. Represents the coefficient.
9. The method of claim 1 or 2, wherein, When n equals 1, determining the ghost region of the target frame image based on the ghost region of the target frame image in n channels includes: The ghost region of the target frame image in one channel is defined as the ghost region of the target frame image.
10. The method of claim 1 or 2, wherein, When n is greater than 1, determining the ghost region of the target frame image based on the ghost region of the target frame image in n channels includes: The ghost regions of the target frame image in n channels are merged to obtain the ghost region of the target frame image.
11. An electronic device, comprising: include: processor; Memory; And one or more computer programs, wherein the computer programs are stored in the memory, and when executed by the processor, the computer programs cause the electronic device to perform the method of any one of claims 1 to 10.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method described in any one of claims 1 to 10.