A ghost detection method and electronic device

Abstract

The present application provides a ghost detection method, which is applied to the field of image processing technology, comprising: obtaining a first type of exposure frame and a second type of exposure frame obtained by photographing a target object, wherein the type of the first type of exposure frame is a normal exposure frame or a long exposure frame, the type of the second type of exposure frame is a short exposure frame or an ultra-short exposure frame, and the number of exposure frames in the second type of exposure frame is at least two frames, determining a target ghost image corresponding to each exposure frame in the second type of exposure frame and a motion overexposure area, wherein the motion overexposure area is an area in the first type of exposure frame that is overexposed and the target object is in a motion state, and the target ghost image is used to characterize the relative texture difference between the motion overexposure area and the area corresponding to the motion overexposure area, and when the corresponding similarity between the two target ghost images is less than a similarity threshold, it can be determined that there are two frames of target ghost images with a large relative texture difference, and in this case it can be determined that the synthesized high dynamic range image has ghosts.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a ghost detection method and electronic equipment. Background Art

[0002] Currently, camera applications are installed in the operating systems of electronic devices such as mobile phones. The camera application can provide multiple shooting modes (such as normal mode, portrait mode, and telephoto mode, etc.) for users to choose from, so that the electronic device can capture images using the shooting mode selected by the user.

[0003] In some possible implementations, the electronic device can turn on the HDR function by default in the normal photo mode, that is, the normal photo mode is also the HDR mode. The electronic device can obtain a set of image frames obtained by photographing the target object in this mode, wherein the image frame set includes at least a first type of exposure frame and a second type of exposure frame, the first type of exposure frame can be a normal exposure frame or a long exposure frame, and the second type of exposure frame can be a short exposure frame or an ultra-short exposure frame. Then, the first type of exposure frame and the second type of exposure frame are fused to obtain a fused HDR image, which can improve the dynamic range of the imaging and restore the details.

[0004] When there is object motion in the overexposed area of ​​the first exposure frame, since the sensor sequentially outputs the first exposure frame and the second exposure frame, there will be a certain timing difference between the image frames. When the object motion is large and the second exposure frame includes multiple exposure frames, it may cause the regional pixel differences corresponding to the overexposed area between different exposure frames in the second exposure frame to be large. In this way, after the first exposure frame and the second exposure frame are fused, a "ghosting" phenomenon will appear in the overexposed area where the object motion is large. Therefore, how to detect whether ghosting is likely to occur before synthesizing an HDR image is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0005] The present application provides a ghost detection method to detect whether a ghost may appear in an HDR image before synthesizing the image.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a ghost detection method, which may include:

[0008] A first type of exposure frame and a second type of exposure frame obtained by photographing the target object are obtained, wherein the type of the first type of exposure frame can be a normal exposure frame or a long exposure frame, and the type of the second type of exposure frame can be a short exposure frame or an ultra-short exposure frame, and the number of exposure frames in the second type of exposure frame is at least two, and then a target ghost image of the area corresponding to the motion overexposure area in each exposure frame of the second type of exposure frame can be determined, wherein the motion overexposure area is an area in the first type of exposure frame that is overexposed and the target object is in motion, and the target ghost image is used to characterize the relative texture difference between the area in which the target object is in motion and the motion overexposure area, and when the similarity between the two target ghost images is less than a similarity threshold, it can be determined that the high dynamic range image synthesized by the first type of exposure frame and the second type of exposure frame has ghosts. Subsequent processing can be performed.

[0009] Exemplarily, the second type of exposure frame may include a first short exposure frame and a second short exposure frame. When the similarity between the two target ghost images is less than a similarity threshold, it is determined that the synthesized high dynamic range image has ghosts. Specifically, the electronic device can traverse the pixel values ​​of each pixel in the first target ghost image and the second target ghost image. When the pixel values ​​of a group of pixels at the same pixel position in the two target ghost images are all 1, the number of intersection pixels of the first target ghost image and the second target ghost image is increased by one. The more the number of intersection pixels, the greater the overlap between the first target ghost image and the second target ghost image. The larger the overlap, the more similar it is, that is, the higher the similarity. When the number of intersection pixels is less than the first similarity threshold, it is determined that the synthesized high dynamic range image has ghosts.

[0010] In some possible implementations, the present application can also calculate the union pixel points between the first target ghost image and the second target ghost image, that is, the electronic device can traverse the pixel value of each pixel point in the first target ghost image and the second target ghost image. When the pixel value of at least one pixel point in a group of pixels at the same pixel position in the two target ghost images is 1, the number of union pixel points is increased by one, and then the ratio of the intersection pixel points to the union pixel points can be calculated, and then the ratio is used to compare with the second similarity threshold. When the ratio is greater than or equal to the second similarity threshold, it can be determined that the similarity between the first target ghost image and the second target ghost image is high, indicating that the motion amplitude of the photographed object between the first target ghost image and the second target ghost image is small. At this time, it can be determined that there is no ghost in the high dynamic range image synthesized by the first type of exposure frame and the second type of exposure frame, and then post-processing is performed to synthesize the high dynamic range image of the first type of exposure frame and the second type of exposure frame.

[0011] When the ratio is less than the second similarity threshold, it can be determined that the similarity between the first target ghost image and the second target ghost image is low, indicating that the subject has a large movement amplitude between the first target ghost image and the second target ghost image. At this time, it can be determined that the high dynamic range image synthesized by the first type of exposure frames and the second type of exposure frames has ghosts, and post-processing can be prohibited, that is, the electronic device can delete the second type of exposure frames in the image frame set, so that the high dynamic range image is not synthesized and only the first type of exposure frames are output.

[0012] In some possible implementations, determining a target ghost image of an area corresponding to the motion overexposed area in each exposure frame of the second type of exposure frames may specifically be:

[0013] According to the preset window, the local standard deviations of the first type of exposure frame, the first short exposure frame and the second short exposure frame are respectively determined; the first ghost image of the first short exposure frame relative to the first type of exposure frame is determined according to the local standard deviation of the first type of exposure frame and the local standard deviation of the first short exposure frame; the second ghost image of the second short exposure frame relative to the first type of exposure frame is determined according to the local standard deviation of the first type of exposure frame and the local standard deviation of the second short exposure frame; finally, the first target ghost image corresponding to the motion overexposed area in the first ghost image and the second target ghost image corresponding to the motion overexposed area in the second ghost image can be determined.

[0014] Specifically, the specific formula for determining the first ghost image according to the local standard deviation of the first type of exposure frame and the local standard deviation of the first short exposure frame may be as follows:

[0015]

[0016] Among them, X1 is the local standard deviation of the first type of exposure frame, X2 is the local standard deviation of the first short exposure frame, C is used to increase the robustness of the calculation result, and map is each sub-ghost image in the first ghost image.

[0017] In some possible implementations, the first ghost image is determined based on a local standard deviation of the first type of exposure frames and a local standard deviation of the first short exposure frame. Specifically, the first initial ghost image of the first short exposure frame relative to the first type of exposure frames is determined based on the local standard deviation of the first type of exposure frames and the local standard deviation of the first short exposure frame, and the first initial ghost image is binarized and morphologically processed (the morphological processing may include, for example, corroding, dilating, and removing discrete points of the binary image) to determine the first ghost image.

[0018] The second ghost image is determined according to the local standard deviation of the first type of exposure frame and the local standard deviation of the second short exposure frame, which can be specifically:

[0019] The second initial ghost image of the second short exposure frame relative to the first exposure frame is determined based on the local standard deviation of the first exposure frame and the local standard deviation of the second short exposure frame, and then the second initial ghost image is binarized and morphologically processed to determine the second ghost image. In this way, the binary image has only two pixel values ​​(usually 0 and 1), which greatly simplifies the data volume of the image frame. The binarized image is easier to process, which can improve the calculation efficiency and reduce the storage space. Morphological processing after binarization can eliminate noise points and breakpoints in the binary image.

[0020] In some possible implementations, determining a target ghost image of an area corresponding to the motion overexposed area in each exposure frame of the second type of exposure frames may specifically be:

[0021] The exposure frame with the shortest exposure time in the second category is determined as the reference exposure frame, and the first category exposure frames and other exposure frames in the second category exposure frames except the reference exposure frame are exposure aligned to the reference exposure frame to ensure that the brightness of the image frames in the image frame set is at the same reference, thereby improving the accuracy of subsequent processing results. Then, a target ghost image of the area corresponding to the motion overexposure area of ​​each exposure frame in the second category exposure frames after exposure alignment is determined.

[0022] In a second aspect, the present application provides an electronic device, including: a processor and a memory;

[0023] One or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device performs the method described in any one of the first aspects.

[0024] In a third aspect, the present application provides a computer storage medium, comprising computer instructions, which, when executed on a mobile terminal, enables the electronic device to execute the method described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of opening a camera application provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of a shooting preview interface of a camera application provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a shooting preview interface of another camera application provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a first type of exposure frame and a second type of exposure frame provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram showing a synthesized HDR image provided in an embodiment of the present application;

[0030] Figure 6 A flow chart of a ghost detection method provided in an embodiment of the present application;

[0031] Figure 7 An architectural diagram of another ghost detection method provided in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of calculating a local standard deviation of an image frame is provided for an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of determining a target ghost image provided by an embodiment of the present application;

[0034] Fig.10 A structural diagram of an electronic device is provided for an embodiment of the present application;

[0035] Fig.11 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0037] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0038] The following is an introduction to the technical terms involved in this application:

[0039] High Dynamic Range (HDR) refers to an image that can record and represent a wider range of brightness than ordinary images (usually called low dynamic range images, LDR). HDR images can more realistically reflect the scenes seen by the human eye in the real world because they can capture more details from the darkest to the brightest.

[0040] The technical solution of the present application is introduced below in combination with technical terms.

[0041] Currently, camera applications are installed in the operating systems of electronic devices such as mobile phones, and users can use the camera application to trigger a photo event to take photos, such as taking photos of people or scenery.

[0042] like Figure 1 As shown, Figure 1 A schematic diagram of opening a camera application is shown. For example, the user touch operation is to open the camera. The touch sensor can receive the user's touch operation on the camera icon and report the touch operation on the camera icon to the processor. After receiving the touch operation on the camera icon, the processor can respond to the touch operation and start the application corresponding to the camera icon (which can be referred to as the camera application). In some possible implementation methods, the user can also enable the electronic device to start the camera application in other ways. For example, through voice commands or other preset gestures, etc., which are not limited in the embodiments of the present application.

[0043] When the electronic device opens the camera application, the lens receives the light reflected from the target scene (the scene to be photographed) and projects the light onto the image sensor. The sensor generates an optical image based on the photography parameters sent by the HAL layer or the pre-set photography parameters (such as exposure time, white balance parameters, focus parameters, etc.). The sensor then converts the optical image into an electrical signal and sends it to the image signal processor ISP. This can then be presented as follows: Figure 2 The shooting preview interface shown.

[0044] In some possible implementations, after the user opens the camera application, the camera application automatically enters the photo taking mode. The user can trigger the photo taking operation by touching the photo taking key in the photo taking preview interface of the camera application. When the user triggers the photo taking operation, the touch sensor can obtain the photo taking operation triggered by the user and report the photo taking operation to the processor. The processor can control the camera application to take the photo and display it on the Figure 2 In the interface shown.

[0045] In some possible implementations, the electronic device may provide multiple photo taking modes, such as Figure 2 As shown, the user can trigger the camera application to enter different shooting modes, and the shooting modes may be, for example, normal shooting mode, HDR mode, enhanced portrait mode, and night scene mode.

[0046] In some possible implementations, the electronic device may enable the HDR function by default in the normal camera mode, that is, the normal camera mode is also the HDR mode.

[0047] The following is an introduction to the HDR function of an electronic device in normal shooting mode. After the electronic device starts the camera application and enters the normal shooting mode, the electronic device can obtain the video stream through the camera, that is, obtain the preview frame sequence of the target object, and store it in the preliminary buffer, and then preview it in the shooting preview interface. Among them, the target object can be a person, a landscape, a building, etc.

[0048] During the process of presenting a preview frame sequence of a target object in a shooting preview interface, the user can click the touch shooting key 201 to trigger a photo shooting operation, triggering a photo shooting event. After the electronic device detects the user's click operation on the touch shooting key 201, it can obtain a set of image frames obtained by shooting the target object, so as to fuse the image frames in the image frame set into an HDR image. It should be noted that the purpose of synthesizing HDR images is to obtain better imaging effects. Multi-exposure frame fusion is usually used to restore the contents of dark and bright areas, that is, the sensor of the same camera continuously frames multiple times to obtain multiple exposure frames, wherein the image frame set includes at least two types of exposure frames, the first type of exposure frame type can be a normal exposure frame or a long exposure frame, and the second type of exposure frame type can be a short exposure frame or an ultra-short exposure frame. Then, the first type of exposure frame and the second type of exposure frame are fused to obtain a fused HDR image. The first type of exposure frame is intended to restore the content of the dark area, and the second type of exposure frame is intended to suppress the overexposed area.

[0049] In this way, the HDR image obtained by fusing the first type of exposure frames and the second type of exposure frames can improve the dynamic range of imaging and restore details. In some possible implementations, the second type of exposure frames may include multiple short exposure frames or ultra-short exposure frames, so that the first type of exposure frames are fused with multiple short exposure frames or ultra-short exposure frames included in the second type of exposure frames, which can further improve the dynamic range of imaging and restore more details.

[0050] When there is object motion in the overexposed area of ​​the first exposure frame, since the sensor sequentially outputs the first exposure frame and the second exposure frame, there will be a certain timing difference between the image frames. When the object moves greatly, it may cause the pixels in the area corresponding to the overexposed area between different exposure frames in the second exposure frame to be greatly different. In this way, after the first exposure frame and the second exposure frame are fused, a "ghosting" phenomenon will appear in the overexposed area where the object moves greatly. For example, Figure 3 In the shooting scene, the first area 3011 is an overexposed area. The man's arm in the picture swings up and down in the first area 3011 when taking a photo. When the multi-exposure synthesis technology is used to take a photo of the scene, the sensor of the electronic device can obtain an image frame as shown in 4. The obtained first type of exposure frame (taking the first type of exposure frame as a normal exposure frame as an example) can be Figure 4As shown in (a) in FIG. 1 , the obtained second type of exposure frame (taking the second type of exposure frame including two short exposure frames as an example) can be Figure 4 (b) and Figure 4 As shown in (c) in the figure, the first exposure frame and the second exposure frame are fused to generate Figure 5 The image frame shown is displayed on the display screen of the electronic device. Figure 5 It can be seen that in the first area 3011, the man's arm has a moving "ghost" (an afterimage appears around the man's arm). Therefore, how to detect whether a ghost image may appear before synthesizing an HDR image is a technical problem that needs to be solved in this field.

[0051] In view of this, the present application provides a ghost detection method, including: obtaining a first type of exposure frame and a second type of exposure frame obtained by shooting a target object, wherein the type of the first type of exposure frame can be a normal exposure frame or a long exposure frame, the type of the second type of exposure frame can be a short exposure frame or an ultra-short exposure frame, and the number of exposure frames in the second type of exposure frame is at least two frames, and then determining a ghost image corresponding to each exposure frame in the second type of exposure frame and a motion overexposed area, wherein the motion overexposed area is an area in the first type of exposure frame that is overexposed and the target object is in a motion state, and the target ghost image is used to characterize the relative texture difference between the motion overexposed area and the area corresponding to the motion overexposed area, and when the corresponding similarity between the two ghost images is less than a similarity threshold, it is determined that a high dynamic range image synthesized by the first type of exposure frame and the second type of exposure frame has a ghost. In this way, by comparing the similarities between the target ghost images, the relative texture differences between the target ghost images can be determined, thereby reflecting the signal differences between the second type of exposure frames. When the similarity between two target ghost images is less than the similarity threshold, it can be determined that there are two frames of target ghost images with large relative texture differences, that is, it can be determined that there are exposure frames with large signal differences in the second type of exposure frames, that is, there are exposure frames with large motion amplitudes of the photographed objects in the area corresponding to the overexposed area. In this case, it can be determined that the high dynamic range image synthesized by the first type of exposure frames and the second type of exposure frames has ghosts.

[0052] In order to make the technical solution of the present application clearer and easier to understand, the ghost detection method provided by the embodiment of the present application is introduced below in combination with the above embodiments and corresponding drawings, taking an electronic device as the execution subject, and the operating system in the electronic device as an Android operating system as an example, which can be referred to as Embodiment 1. Figure 6 As shown, the ghost detection method provided in the first embodiment of the present application may include:

[0053] S601: In response to a user opening a camera operation, starting a camera application.

[0054] The electronic device launches a camera application in response to a user opening a camera operation.

[0055] Specifically, the electronic device receives a user's touch operation on the camera icon and reports the touch operation on the camera icon to the processor. After receiving the touch operation on the camera icon, the processor can respond to the touch operation and start the application corresponding to the camera icon (which may be referred to as the camera application). In some possible implementations, the user can also enable the electronic device to start the camera application in other ways. For example, through voice commands or other preset gestures, etc., which are not limited in the embodiments of the present application.

[0056] After the electronic device starts the camera application, it can enter the normal photo mode. The following is an introduction to the HDR function being turned on by default in the normal photo mode of the electronic device.

[0057] S602: In response to triggering a photographing event, obtaining a set of image frames obtained by photographing a target object.

[0058] The electronic device can obtain the video stream through the camera and preview it in the shooting preview interface.

[0059] The user can click the touch shooting key 201 to trigger the photo shooting operation and the photo shooting event. After the electronic device detects the user's click operation on the touch shooting key 201, it can obtain a set of image frames obtained by shooting the target object, wherein the target object here can be, for example, a person, a landscape, and a building, etc. It should be noted that the image frame set includes at least two types of exposure frames, the first type of exposure frame can be a normal exposure frame or a long exposure frame, and the second type of exposure frame can be a short exposure frame or an ultra-short exposure frame. The number of synthesized HDR image frames, that is, the number of the first type of exposure frames and the number of the second type of exposure frames in the acquired image frame set can be preset according to demand.

[0060] For example, in this embodiment, an image frame set includes a first-type exposure frame and the first-type exposure frame is a normal exposure frame, and an image frame set includes two short-exposure frames. That is, when the electronic device detects a user's click operation on the touch capture key 201, three image frames of the target object can be obtained, and the image frames in the obtained image frame set are one normal exposure frame and two short exposure frames. For example, the obtained image frames can be as shown in Table 1:

[0061] Table 1

[0062]

[0063] Among them, N (Normal) indicates that the image frame is a normal exposure frame, and the normal exposure frame can be used as a reference frame of the image. It should be noted that the normal exposure frame, that is, the reference frame, can be the image frame corresponding to the moment when the electronic device triggers a photo event, or it can be the image frame corresponding to the moment when the user clicks and touches the shooting key 201. Here, it can be considered that the moment when the user clicks and touches the shooting key 201 is the same as the moment when the electronic device triggers a photo event. The exposure time corresponding to the normal exposure frame can be a preset standard exposure time, for example, it can be 10ms, which can be represented by exposure value (exposure values, EV) = 0. The EV value of zero can also be a standard EV value, where the EV value is a value used to reflect the exposure time.

[0064] After determining the reference frame of the image, the electronic device can obtain other image frames with different exposure times.

[0065] S (Short) can indicate that the image frame is a short exposure frame, that is, the exposure time corresponding to the short exposure frame is shorter than the standard exposure time, which can be represented by the EV value reduced by a certain value on the standard EV value. For example, the image frame 2 in Table 1 can be reduced by 0.5 in EV value. When the EV value is reduced by 0.5, the exposure time corresponding to the image frame 3 can be 7.07ms, for example. For example, the image frame 3 in Table 1 can be reduced by 0.4 in EV value. When the EV value is reduced by 0.4, the exposure time corresponding to the image frame 3 will also be shorter than the normal exposure frame. It should be noted that the above is only an example introduction, and of course, the embodiments of the present application are not limited to this. The first type of exposure frame included in the image frame set can also be a long exposure frame (the corresponding exposure time is longer than the standard exposure time, which can be represented by the EV value increasing a certain value on the standard EV value), and the second type of exposure included in the image frame set can also be an ultra-short exposure frame (that is, the exposure time corresponding to the ultra-short exposure frame is shorter than the exposure time of the short exposure frame, which can be represented by the EV value reduced by a certain value on the standard EV value), and again there is no limitation.

[0066] S603: Calculate a ghost image of the second type of exposure frames in the image frame set relative to the first type of exposure frames.

[0067] After acquiring a set of image frames obtained by photographing a target object, the electronic device can detect a ghost map of a second type of exposure frame in the image frame set relative to a first type of exposure frame, wherein the ghost map is used to characterize the double or ghosting effects produced by moving objects in a scene in an image sequence with different exposure times.

[0068] In some possible implementations, the first type of exposure frame images and the second type of exposure frame images may be firstly exposed and aligned to ensure that the brightness of the image frames in the image frame set is at the same reference, thereby improving the accuracy of subsequent processing results.

[0069] First, the baseline exposure frame can be determined. In this embodiment, in order to ensure that the overexposed area of ​​each image frame has more information and facilitate subsequent calculations, the second type of exposure frame (here taking two short exposure frames as an example) with the shortest exposure time can be determined as the baseline exposure frame, and then the exposure alignment is performed on the other exposure frames in the image frame set.

[0070] The electronic device may calculate a ghost image of the second-type exposure frame after exposure alignment relative to the first-type exposure frame.

[0071] Exemplarily, the electronic device may include a ghost detection module, such as Figure 7 As shown in FIG. 1 , the second type of exposure frame and the first type of exposure frame after exposure alignment can be input into the ghost detection module (the ghost detection module is in FIG. 1 ). Figure 7 In the example, the first type of exposure frame includes one normal exposure frame, and the second type of exposure frame includes two short exposure frames (for example, including a first short exposure frame and a second short exposure frame). Then, the ghost detection module can output a first ghost image of the first short exposure frame relative to the normal exposure frame (the first ghost image is in Figure 7 Ghost map 1 in the figure) and the second ghost map of the second short exposure frame relative to the normal exposure frame (the second ghost map is in Figure 7 It should be noted that the absolute texture complexity index (i.e., the absolute ghost map obtained by direct difference) is related to the texture complexity of the input image itself. Even if the absolute texture complexity index is similar, the underlying semantics are different and the visual quality is different. Therefore, the relative texture difference index (i.e., the ghost map of the second exposure frame relative to the first exposure frame) that is independent of the original texture change can more accurately reflect the signal difference between frames.

[0072] In some possible implementations, the ghost detection module may include a ghost complexity calculation module (the ghost complexity calculation module is in Figure 7 The Ghost complexity calculation module in the figure) and the relative ghost calculation module (the relative ghost calculation module in the figure) Figure 7 (in the figure is the relative Ghost calculation module).

[0073] After the first type of exposure frames and the second type of exposure frames are input to the ghost detection module, the ghost complexity calculation module in the ghost detection module can calculate the local standard deviation of each input image. It should be noted that a high local standard deviation usually indicates that there is a large grayscale change in the image area, that is, the texture is relatively rough, and a low local standard deviation indicates that the grayscale change is small, that is, the texture is relatively smooth. The local standard deviation can provide information about the local structure of the image. The ghost area may have a large grayscale change, resulting in a high local standard deviation. Then, by analyzing the local standard deviation of different exposure frames, the area where ghosts may exist can be identified.

[0074] The following is an introduction using the calculation of the local standard deviation of a frame of input image as an example.

[0075] Exemplarily, it is first necessary to determine the local window size n of the frame image, where the window is used to analyze or process the local area of ​​the image, and its size and shape can vary according to the needs of different applications. The number of pixels included in a window depends on the window size n, and n can be set according to actual needs. For example, the window size can be 3*3 (including 9 pixels) and 5*5 (including 25 pixels), etc.

[0076] The ghost complexity calculation module can place the window on the first center pixel of the frame image and calculate the standard deviation of the pixel value of each pixel included in the window. After the standard deviation calculation of the current window is completed, the window is moved to the right to the next center pixel, and the standard deviation calculation is repeated until the last center pixel at the end of the row is reached.

[0077] When the standard deviation of the last center pixel at the end of the row is calculated, it can be moved to the first center pixel at the beginning of the next row and the sliding window process is repeated until the entire image is covered.

[0078] Specifically, the formula for calculating the local standard deviation of a window of size n*n at the center pixel point (i, j) of the image can be shown as formula (1):

[0079] X(i,j)=sd(img(i-(n-1) / 2: i+(n-1) / 2, j-(n-1) / 2: j+(n-1) / 2))(1)

[0080] Where img is the image matrix, sd(.) represents the standard deviation operator, n represents the local window size, i-(n-1) / 2 is used to calculate the pixel value of the upper left corner of the window in the vertical direction, i+(n-1) / 2 is used to calculate the pixel value of the lower right corner of the window in the vertical direction. j-(n-1) / 2 is used to calculate the pixel value of the upper left corner of the window in the horizontal direction, and j+(n-1) / 2 is used to calculate the pixel value of the lower right corner of the window in the horizontal direction.

[0081] like Figure 8 As shown in (a), for example, if the image frame is a RAW image in RGBC format, n=2, then the window is a 2x2 area, then the formula for the local standard deviation of the window at the center pixel point (i, j) of the image can be: , which can represent the standard deviation of the pixel values ​​of the 9 pixels from (i-1, j-1) to (i+1, j+1). Among them, the 9 pixels from (i-1, j-1) to (i+1, j+1) include (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j), (i, j+1), (i+1, j-1), (i+1, j) and (i+1, j+1).

[0082] The ghost complexity calculation module can calculate the local standard deviation of each input image using formula (1), such as Figure 8 As shown in (b), multiple local standard deviations can be calculated in each frame of the image (one window corresponds to one local standard deviation). For example, each local standard deviation of the normal exposure frame can be represented by X1, each local standard deviation of the first short exposure frame of the two short exposure frames can be represented by X2, and each local standard deviation of the second short exposure frame can be represented by X3. The relative ghost calculation module can calculate the ghost image of the second type of exposure frame relative to the first type of exposure frame based on the calculated X1, X2 and X3.

[0083] For example, the formula for calculating the first ghost image of the first short exposure frame relative to the normal exposure frame can be shown as formula (2):

[0084] (2)

[0085] Wherein, map is a sub-ghost image in the first ghost image. Each X1 in the normal exposure frame and the corresponding X2 in the first short exposure frame can generate a sub-ghost image according to formula (2), and then the first ghost image can be generated according to each sub-ghost image.

[0086] like Figure 7 As shown, based on the above principle, a first ghost image of the first short exposure frame relative to the normal exposure frame and a second ghost image of the second short exposure frame relative to the normal exposure frame can be generated.

[0087] In some possible implementations, a constant C may be introduced into formula (2) to increase the robustness of the calculation result, as shown in formula (3):

[0088] (3)

[0089] The value of C can be set according to actual needs, and no unnecessary restrictions are imposed.

[0090] In some possible implementations, such as Figure 7 The ghost detection module shown may also include a ghost Figure 2 Value module (ghost Figure 2 The value module is Figure 7 The Ghost map binarization module in the figure) and the morphological image processing module (the morphological image processing module in the figure) Figure 7 (Figure 2 shows the binary Ghost map post-processing module).

[0091] Example, ghost Figure 2 The value module can use the threshold segmentation method to binarize the ghost image output by the relative ghost calculation module (here the ghost image output by the ghost calculation module can also be called the initial ghost image), and obtain the binary image corresponding to each initial ghost image (the initial ghost image of the first short exposure frame relative to the first type of exposure frame can be called the first initial ghost image, and the initial ghost image of the second short exposure frame relative to the first type of exposure frame can be called the second initial ghost image), where the threshold in the threshold segmentation is a value that can be adjusted according to the brightness. Then the pixel value in the grayscale image is compared with this threshold. Pixels above the threshold are marked as True (or 1), and pixels below the threshold are marked as False (or 0). Then, this mask can be applied to the original image, setting the pixels corresponding to True to a certain value (usually 255, indicating white), and setting the pixels corresponding to False to another value (usually 0, indicating black). Here, the pixel value of the pixel above the threshold is marked as 1 as an example. Since the binary image has only two pixel values ​​(usually 0 and 1), this greatly simplifies the amount of data in the image frame. The binarized image is easier to process, which can improve computational efficiency and reduce storage space.

[0092] The electronic device can input the binarized image output by the binarization module into the morphological image processing module for morphological processing to eliminate noise points and breakpoints in the binary image. The morphological processing can, for example, erode, dilate, and remove discrete points on the binary image, and then output a ghost image of the second type of exposure frame relative to the first type of exposure frame (for example, the output here can be a first ghost image of the first short exposure frame relative to the normal exposure frame and a second ghost image of the second short exposure frame relative to the normal exposure frame).

[0093] S604: Determine, according to the ghost image, a target ghost image corresponding to the motion overexposed region of the first type of exposure frame.

[0094] The electronic device can determine the target ghost image corresponding to the motion overexposed area of ​​the first type of exposure frame according to the ghost image, such as Figure 7 As shown, the mask of the motion overexposed area in the first ghost image is determined according to the mask of the motion overexposed area (the mask of the motion overexposed area is in Figure 7 Overesop mask in the figure) corresponds to the first target ghost image (the first target ghost image is in Figure 7 In the figure, Overesop ghost1 is defined, and the first target ghost image corresponding to the mask of the motion overexposed area in the first ghost image is determined according to the mask of the motion overexposed area (the second target ghost image is defined in Figure 7 Overesop ghost2), where the overexposed area refers to the area where the brightness of some pixels is too high due to excessive light or too long exposure time. The area composed of these pixels with too high brightness is the overexposed area. The overexposed area usually shows no texture and no color change. The motion overexposed area refers to the area in the overexposed area where the subject is in motion.

[0095] Furthermore, the electronic device may first determine the overexposed area and the object motion area of ​​the first type of exposure frame (taking the normal exposure frame as an example in this embodiment) before determining the target ghost image corresponding to the motion overexposed area of ​​the first type of exposure frame.

[0096] Exemplarily, determining the overexposed area of ​​the first type of exposure frame may be performed as follows:

[0097] The electronic device can calculate the brightness value (Lighting Value, LV) of each pixel in the normal exposure frame. If the LV value of the pixel is greater than or equal to the brightness threshold, the pixel is judged to be overexposed, and the pixel can be marked as an overexposed pixel (for example, the pixel is marked as 1). If the LV value of the pixel is less than the brightness threshold, the pixel is judged not to be overexposed, and the pixel is not marked as an overexposed pixel (for example, the pixel can be marked as 0), wherein the brightness threshold can be obtained from empirical values, historical data, or experimental data, and the embodiments of the present application do not limit this. In this way, the area corresponding to each overexposed pixel can be determined as an overexposed area.

[0098] The object motion region of the first type of exposure frame may be determined as follows:

[0099] The floating point number representing the degree of optical flow motion is calculated for each pixel. When the floating point number is 0, there is no object motion in the area corresponding to the pixel. When the floating point number is not 0, there is object motion in the area corresponding to the pixel.

[0100] Exemplarily, in this embodiment, a floating point threshold may be set, for example, the floating point threshold is set to 1, then pixels with floating point numbers greater than or equal to 1 may be marked as 1, and pixels with floating point numbers less than 1 may be marked as 0. Thus, the electronic device may determine the area corresponding to each pixel with object motion as the object motion area.

[0101] After determining the overexposed area and the object motion area, the electronic device can perform a "logical AND" operation on the overexposed area and the object motion area to obtain the intersection of the overexposed area and the object motion area, so as to determine the motion overexposed area of ​​the first type of exposure frame.

[0102] After determining the motion overexposure area of ​​the first type of exposure frame, the electronic device can determine the target ghost image corresponding to the motion overexposure area in the second type of exposure frame, such as Fig. 9 As shown, Fig. 9 910 in (a) is the first target ghost image corresponding to the motion overexposed area in the first ghost image. Fig. 9 911 in (b) is a second target ghost image corresponding to the motion overexposed area in the second ghost image.

[0103] S605: Determine the similarity between each target ghost image respectively.

[0104] After determining the target ghost image corresponding to each second-category exposure frame, the electronic device may calculate the similarity between each target ghost image.

[0105] Exemplarily, for example, in this embodiment, the second type of exposure frame includes two short exposure frames (one frame is T1 and one frame is T2). The electronic device can determine the similarity between the first target ghost image corresponding to T1 and the second target ghost image corresponding to T.

[0106] Specifically, the electronic device may traverse the pixel values ​​of each pixel point in the first target ghost image and the second target ghost image. When the pixel values ​​of a group of pixel points at the same pixel position in the two target ghost images are all 1, the intersection pixel point of the first target ghost image and the second target ghost image (in Figure 7 The greater the number of intersection pixels, the greater the overlap between the first target ghost image and the second target ghost image. The greater the overlap, the more similar they are, that is, the higher the similarity.

[0107] S606: Perform ghost detection according to the similarity between the target ghost images.

[0108] The electronic device can perform ghost detection based on the similarity between target ghost images.

[0109] In some possible implementations, the number of intersection pixels may be compared with a first similarity threshold. When the number of intersection pixels is greater than or equal to the first similarity threshold, it may be determined that the first target ghost image and the second target ghost image are highly similar, indicating that the motion amplitude of the photographed object between the first target ghost image and the second target ghost image is small. In this case, it may be determined that the high dynamic range image synthesized by the first type of exposure frames and the second type of exposure frames does not have ghosts.

[0110] When the number of intersection pixels is less than the first similarity threshold, it can be determined that the first target ghost image and the second target ghost image are less similar, indicating that the photographed object (also called the target object) has a small movement range between the first target ghost image and the second target ghost image. At this time, it can be determined that the high dynamic range image synthesized by the first type of exposure frame and the second type of exposure frame has ghosts, and the electronic device can delete the second type of exposure frame in the image frame set, so that only the first type of exposure frame is output in the end.

[0111] In some possible implementations, such as Figure 7 As shown, the union pixel points between the first target ghost image and the second target ghost image can be calculated (in Figure 7 Og1∪Og2), that is, the electronic device can traverse the pixel value of each pixel in the first target ghost image and the second target ghost image. When the pixel value of at least one pixel in a group of pixels at the same pixel position in the two target ghost images is 1, the number of union pixels is increased by one, and then the ratio of the intersection pixel to the union pixel can be calculated, and then the ratio is compared with the second similarity threshold. When the ratio is greater than or equal to the second similarity threshold, it can be determined that the first target ghost image and the second target ghost image are highly similar, indicating that the motion amplitude of the photographed object between the first target ghost image and the second target ghost image is small. At this time, it can be determined that the high dynamic range image synthesized by the first type of exposure frame and the second type of exposure frame does not have a ghost, and then post-processing is performed (in Figure 7 The postprocess is a high dynamic range image that is synthesized by the first type of exposure frame and the second type of exposure frame.

[0112] When the ratio is less than the second similarity threshold, it can be determined that the similarity between the first target ghost image and the second target ghost image is low, indicating that the motion amplitude of the photographed object is large between the first target ghost image and the second target ghost image. In this case, it can be determined that the high dynamic range image synthesized by the first type of exposure frame and the second type of exposure frame has ghosts, and post-processing is prohibited (in Figure 7Drop post process) means that the electronic device can delete the second type of exposure frames in the image frame set, so that only the first type of exposure frames are outputted as the synthesized high dynamic range image.

[0113] In this embodiment of the present application, a first type of exposure frame and a second type of exposure frame obtained by photographing the target object can be obtained, wherein the type of the first type of exposure frame can be a normal exposure frame or a long exposure frame, the type of the second type of exposure frame can be a short exposure frame or an ultra-short exposure frame, and the number of exposure frames in the second type of exposure frame is at least two frames, and then a ghost image corresponding to each exposure frame in the second type of exposure frame and the motion overexposed area is determined, wherein the motion overexposed area is an area in the first type of exposure frame that is overexposed and the target object is in a motion state, and the target ghost image is used for the relative texture difference between the motion overexposed area and the area corresponding to the motion overexposed area, and when the corresponding similarity between the two target ghost images is less than a similarity threshold, it is determined that the high dynamic range image synthesized by the first type of exposure frame and the second type of exposure frame has ghosts. In this way, by comparing the similarities between the target ghost images, the relative texture differences between the target ghost images can be determined, thereby reflecting the signal differences between the second type of exposure frames. When the similarity between two target ghost images is less than the similarity threshold, it can be determined that there are two frames of target ghost images with large relative texture differences, that is, it can be determined that there are exposure frames with large signal differences in the second type of exposure frames, that is, there are exposure frames with large motion amplitudes of the photographed objects in the area corresponding to the overexposed area. In this case, it can be determined that the high dynamic range image synthesized by the first type of exposure frames and the second type of exposure frames has ghosts.

[0114] The above embodiment is introduced by taking the second type of exposure frame including two exposure frames as an example, and the following is introduced by taking the second type of exposure frame including three exposure frames as an example (for example, T1, T2 and T3). It should be noted that the three exposure frames here can all be short exposure frames, or all be ultra-short exposure frames, or some can be short exposure frames and some can be ultra-short exposure frames. This embodiment can be called embodiment 2, and embodiment 2 is described in detail below:

[0115] In this embodiment, a first type of exposure frame and a second type of exposure frame obtained by photographing the target object can be obtained, wherein the type of the first type of exposure frame can be a normal exposure frame or a long exposure frame, and the second type of exposure frame here takes three exposure frames as an example, and then step S603 is executed to obtain a ghost image of each exposure frame in the second type of exposure frame relative to the first type of exposure frame, and then step S604 is executed to determine the target ghost image corresponding to the motion overexposed area of ​​the first type of exposure frame according to the ghost image, and then step S605 is executed to determine the similarity between each target ghost image (the target ghost image corresponding to T1 can be called the first target ghost image, the target ghost image corresponding to T2 can be called the second target ghost image, and the target ghost image corresponding to T3 can be called the third target ghost image). In some possible implementation methods, it is not necessary to determine the similarity between each target ghost image separately, and the first similarity between the first target ghost image and the second target ghost image and the second similarity between the second target ghost image and the third target ghost image can be calculated. Finally, step S606 is executed to perform ghost detection. For example, when it is determined that the first similarity is greater than or equal to the similarity threshold (which may include the first similarity threshold or the second similarity threshold) and the second similarity is greater than or equal to the similarity threshold, it can be determined that the high dynamic range image synthesized by the first type of exposure frames and the second type of exposure frames has ghosts. In this case, all the second type of exposure frames in the image frame set can be deleted, so that only the first type of exposure frames are finally output.

[0116] When it is determined that the first similarity is greater than or equal to the similarity threshold and the second similarity is less than the similarity threshold, it can also be determined that the high dynamic range image synthesized by the first type of exposure frames and the second type of exposure frames has ghosts. In this case, T1 in the second type of exposure frames can be deleted, and finally T2, T3 and the first type of exposure frames are used for fusion, and finally the fused high dynamic range image is output.

[0117] When it is determined that the first similarity is smaller than the similarity threshold and the second similarity is greater than or equal to the similarity threshold, it can also be determined that the high dynamic range image synthesized by the first type of exposure frames and the second type of exposure frames has ghosts. In this case, T3 in the second type of exposure frames can be deleted, so that T1, T2 and the first type of exposure frames are finally used for fusion, and finally the fused high dynamic range image is output.

[0118] The method provided in the present application can be executed on an electronic device. In some embodiments, the electronic device can be a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, etc. The present application does not impose any special restrictions on the specific form of the above electronic devices. In this embodiment, the structure of the electronic device can be as follows: Fig.10 As shown, Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0119] like Fig.10 As shown, the electronic device may include a processor 1011, a sensor module 1012, a camera 1013, a display screen 1014, etc. The sensor module 1012 may include a touch sensor 1012A, etc.

[0120] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0121] The processor 1011 may include one or more processing units, for example, the processor 1011 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0122] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0123] In some embodiments, the processor 1011 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, and a mobile industry processor interface (MIPI).

[0124] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 1011 may include multiple groups of I2C buses. The processor 1011 may be coupled to the touch sensor 1012A, the charger, the flash, the camera 1013, etc. through different I2C bus interfaces. For example: the processor 1011 may be coupled to the touch sensor 1012A through the I2C interface, so that the processor 1011 communicates with the touch sensor 1012A through the I2C bus interface to realize the touch function of the electronic device.

[0125] The MIPI interface can be used to connect the ISP to the processor 1011, and to connect the processor 1011 to peripheral devices such as the display screen 1014. The MIPI interface can include a camera serial interface (CSI), a display serial interface (DSI), and the like.

[0126] In some embodiments, the ISP can transmit the processed digital image signal to the processor 1011 through the CSI interface in the MIPI standard interface to realize the shooting function of the electronic device. The processor 1011 and the display screen 1014 communicate through the DSI interface, and finally convert it into an image displayed on the screen of the electronic device to realize the display function of the electronic device.

[0127] The electronic device implements the display function through a GPU, a display screen 1014, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1014 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1011 may include one or more GPUs, which execute program instructions to generate or change display information.

[0128] The display screen 1014 is used to display images, videos, etc. The display screen 1014 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 1014, where N is a positive integer greater than 1.

[0129] A series of graphical user interfaces (GUIs) can be displayed on the display screen 1014 of the electronic device, and these GUIs are the main screen of the electronic device. Generally speaking, the size of the display screen 1014 of the electronic device is fixed, and only limited controls can be displayed on the display screen 1014 of the electronic device. A control is a GUI element, which is a software component included in an application program, and controls all data processed by the application program and interactive operations on the data. Users can interact with the control through direct manipulation to read or edit relevant information of the application program. Generally speaking, a control can include visual interface elements such as icons, controls, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0130] The electronic device can realize the shooting function through ISP, camera 1013, video codec, GPU, display screen 1014 and application processor.

[0131] The ISP is used to process the data fed back by the camera 1013. For example, when taking a photo, the shutter is opened, and the ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 1013.

[0132] The camera 1013 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto a photosensitive element (also called an image sensor). The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 1013, where N is a positive integer greater than 1.

[0133] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0134] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0135] NPU is a neural network (NN) computing processor. It can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of electronic devices, such as image recognition.

[0136] The touch sensor 1012A is also called a "touch control device". The touch sensor 1012A can be set on the display screen 1014, and the touch sensor 1012A and the display screen 1014 form a touch screen, also called a "touch control screen". The touch sensor 1012A is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 1014. In other embodiments, the touch sensor 1012A can also be set on the surface of the electronic device, which is different from the position of the display screen 1014.

[0137] The software architecture of the present application is introduced below. The software architecture of the electronic device of the present application can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example for exemplary description. Fig.11 As shown:

[0138] like Fig.11 As shown, the software architecture of the electronic device of the present application can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example for exemplary description.

[0139] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, application framework layer, Android runtime, system library and kernel layer.

[0140] The application layer may include a series of application packages, for example, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0141] The application framework layer provides an application programming interface (API) and a programming framework for the application. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework may include a view system and a display system Android interface definition language (AIDL) interface and an activity recognition management service ARMS.

[0142] The application framework layer may include a ghost detection module, and the ghost detection module is used to calculate a ghost image of the second type of exposure frame relative to the first type of exposure frame.

[0143] Specifically, the ghost detection module may include a ghost complexity calculation module and a relative ghost calculation module, wherein the ghost complexity calculation module is used to calculate the local standard deviation of each input image in the first type of exposure frame and the second type of exposure frame. The relative ghost calculation module is used to determine the ghost image of the second type of exposure frame relative to the first type of exposure frame according to the calculated local standard deviation of the image frame.

[0144] In some possible implementations, the ghost detection module may also include a ghost Figure 2 The value module and the morphological image processing module. Figure 2The valuing module is used to perform binarization processing on the ghost image output by the relative ghost calculation module (the ghost image output by the ghost calculation module can also be called the initial ghost image) by using the threshold segmentation method. The morphological image processing module is used to perform morphological processing on the binary image output by the binarization module. For example, the morphological processing can erode, dilate and remove discrete points of the binary image, and then output the ghost image of the second type of exposure frame relative to the first type of exposure frame.

[0145] The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. The display interface can be composed of one or more views. For example, the display interface of a camera application can include a view for displaying text and a view for displaying images.

[0146] The application layer and framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0147] The kernel layer is a layer between hardware and software. The kernel layer provided in the embodiment of the present application includes a display driver, an audio driver, and a sensor driver.

[0148] The technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, etc., various media that can store program codes.

[0149] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A ghost detection method, characterized in that: include: Acquire a first type of exposure frame and a second type of exposure frame obtained by photographing the target object, wherein the first type of exposure frame is a normal exposure frame or a long exposure frame, the second type of exposure frame is a short exposure frame or an ultra-short exposure frame, and the number of exposure frames in the second type of exposure frame is at least two frames; Determine a target ghost image of an area corresponding to a motion overexposed area in each exposure frame of the second type of exposure frames, wherein the motion overexposed area is an area in the first type of exposure frames where the overexposed area is in a state of motion of the target object, and the target ghost image is used to characterize a relative texture difference between the motion overexposed area and an area corresponding to the motion overexposed area in the second type of exposure frames; When the similarity between the two target ghost images is less than the similarity threshold, it is determined that the high dynamic range image synthesized by the first-type exposure frames and the second-type exposure frames has ghosts.

2. The method according to claim 1, characterized in that: The second type of exposure frames includes a first short exposure frame and a second short exposure frame. When the similarity between the two target ghost images is less than a similarity threshold, determining that the synthesized high dynamic range image has a ghost includes: Determine the number of intersection pixels between the first short exposure frame and the second short exposure frame, where the intersection pixels are a group of pixels in the first short exposure frame and the second short exposure frame that are at the same pixel position and have pixel values ​​of 1; When the number of the intersection pixels is less than the first similarity threshold, it is determined that the synthesized high dynamic range image has ghosts.

3. The method according to claim 1, characterized in that The second type of exposure frames includes a first short exposure frame and a second short exposure frame. When the similarity between the two target ghost images is less than a similarity threshold, determining that the synthesized high dynamic range image has a ghost includes: Determine the number of intersection pixels and the number of union pixels of the first short exposure frame and the second short exposure frame, wherein the intersection pixels are a group of pixels at the same pixel position and whose pixel values ​​are all 1, and the union pixels are a group of pixels at the same pixel position in the first short exposure frame and the second short exposure frame and at least one pixel has a pixel value of 1; When the ratio of the number of the intersection pixels to the number of the union pixels is less than a second similarity threshold, it is determined that the synthesized high dynamic range image has ghosts.

4. The method according to claim 2 or 3, characterized in that: The step of determining a target ghost image of an area corresponding to the motion overexposure area in each exposure frame of the second type of exposure frames includes: Determine the local standard deviations of the first type of exposure frame, the first short exposure frame, and the second short exposure frame respectively according to the preset window; Determine a first ghost image of the first short exposure frame relative to the first type of exposure frame according to the local standard deviation of the first type of exposure frame and the local standard deviation of the first short exposure frame; Determine a second ghost image of the second short exposure frame relative to the first type of exposure frame according to the local standard deviation of the first type of exposure frame and the local standard deviation of the second short exposure frame; A first target ghost image corresponding to the motion overexposed area in the first ghost image is determined, and a second target ghost image corresponding to the motion overexposed area in the second ghost image is determined.

5. The method according to claim 4, characterized in that The specific formula for determining the first ghost image according to the local standard deviation of the first type of exposure frame and the local standard deviation of the first short exposure frame is as follows: ； Among them, X1 is the local standard deviation of the first type of exposure frame, X2 is the local standard deviation of the first short exposure frame, C is used to increase the robustness of the calculation result, and map is each sub-ghost image in the first ghost image.

6. The method according to claim 4, characterized in that Determining the first ghost image according to the local standard deviation of the first type of exposure frames and the local standard deviation of the first short exposure frame includes: Determine a first initial ghost image of the first short exposure frame relative to the first type of exposure frame according to a local standard deviation of the first type of exposure frame and a local standard deviation of the first short exposure frame; Performing binarization and morphological processing on the first initial ghost image to determine the first ghost image; Determining the second ghost image according to the local standard deviation of the first type of exposure frames and the local standard deviation of the second short exposure frames includes: Determine a second initial ghost image of the second short exposure frame relative to the first type of exposure frame according to a local standard deviation of the first type of exposure frame and a local standard deviation of the second short exposure frame; The second initial ghost image is subjected to binarization processing and morphological processing to determine the second ghost image.

7. The method according to claim 1, characterized in that The step of determining a target ghost image of an area corresponding to the motion overexposed area in each exposure frame of the second type of exposure frames includes: Determine the exposure frame with the shortest exposure time among the second type of exposure frames as the reference exposure frame; Performing exposure alignment on the first type of exposure frames and other exposure frames in the second type of exposure frames except the reference exposure frame to the reference exposure frame; A target ghost image of an area corresponding to the motion overexposed area in each exposure frame of the second type of exposure frames after exposure alignment is determined.

8. The method according to claim 1, characterized in that: In the case where it is determined that a high dynamic range image synthesized by the first type of exposure frames and the second type of exposure frames has a ghost, the method further includes: The second-type exposure frames are deleted, and the first-type exposure frames are output.

9. An electronic device, characterized in that: include: Processor and memory; One or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The method comprises computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 8.

Images