Image processing method, image processing device, storage medium and electronic device

Acquire continuous multi-frame images through a smartphone and perform maximum pixel value fusion, solving the problems of ghosting and image blur in long exposure shooting, achieving clear motion trajectory effects and high-quality images.

CN112767295BActive Publication Date: 2024-11-15GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202110049901.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-14
Publication Date
2024-11-15
Estimated Expiration
2041-01-14

AI Technical Summary

Technical Problem

Electronic devices such as smartphones are prone to ghosting, overexposure and image blurring during long exposure shooting, making it difficult to effectively capture the effect of clear motion trajectory.

Method used

By acquiring continuous multi-frame images, extracting the foreground area images and fusing the maximum pixel value, generating the target image, avoiding the problem caused by inappropriate exposure time in traditional long exposure methods, and using the background area for registration and fusion to reduce noise Influence.

Benefits of technology

A simulated long exposure effect is achieved, avoiding ghosting and overexposure, improving image quality, retaining clear motion trajectories, and reducing noise impact.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112767295B_ABST
    Figure CN112767295B_ABST
Patent Text Reader

Abstract

The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, and relates to the field of image processing technology. Among them, an image processing method of the present disclosure includes: acquiring multiple consecutive frames of images; extracting foreground area images of at least two frames of images; fusing each foreground sub-region according to the maximum pixel value of each foreground sub-region in each foreground area image to form a foreground fused image, wherein the foreground sub-region is a sub-region of the foreground area image; and generating a target image based on the foreground fused image and the background area image of at least one frame of image. The present disclosure implements a technical solution for simulating a long exposure effect through the fusion of multiple frames of images, which is suitable for electronic devices such as smart phones, and can retain a clear and continuous motion trajectory in the final output target image while reducing the influence of noise.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Long exposure refers to a shooting method that maintains the lens exposure for a long time to obtain special effects. Generally, long exposure images can capture the motion trajectory of moving objects. For example, long exposure shooting of moving light sources in night scenes (such as car lights and meteors) can produce the effect of continuous light tracks (i.e., streamer images), and long exposure shooting of water flows (such as waterfalls and rivers) can produce images with the dynamic effect of rushing water.

[0003] Currently, long-exposure photography is generally suitable for SLR cameras. On electronic devices such as smartphones, due to the small camera lens, when using long-exposure photography, if the exposure time is insufficient, the moving subject may have ghost images (such as the ghost image of the car body). If the exposure time is too long, the image may be overexposed. In addition, the long exposure process is easily affected by the shaking of the device body, resulting in blurred images. Summary of the invention

[0004] The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium and an electronic device, thereby achieving a simulated long-exposure image shooting effect at least to a certain extent.

[0005] According to a first aspect of the present disclosure, there is provided an image processing method, comprising: acquiring a plurality of consecutive frames of images; extracting foreground area images of at least two frames of images; fusing each foreground sub-region according to a maximum pixel value of each foreground sub-region in each foreground area image to form a foreground fused image, wherein the foreground sub-region is a sub-region of the foreground area image; and generating a target image based on the foreground fused image and a background area image of at least one frame of images.

[0006] According to a second aspect of the present disclosure, there is provided an image processing method, comprising: determining a reference frame image in a plurality of continuous frame images, and determining at least one frame image other than the reference frame image as an image to be fused; determining a foreground area of ​​the image to be fused; extracting at least one sub-area from the image to be fused; and fusing the sub-areas in the image to be fused to the reference frame image respectively, wherein, when the sub-area is located in the foreground area, the larger pixel value of the sub-area in the image to be fused and the reference frame image is used as the pixel value of the sub-area after fusion.

[0007] According to a third aspect of the present disclosure, there is provided an image processing device, comprising: an image acquisition module, configured to acquire multiple consecutive frames of images; an image segmentation module, configured to extract foreground area images of at least two frames of images; a foreground fusion module, configured to fuse each foreground sub-region according to the maximum pixel value of each foreground sub-region in each foreground area image to form a foreground fused image, wherein the foreground sub-region is a sub-region of the foreground area image; and a target fusion module, configured to generate a target image based on the foreground fused image and the background area image of at least one frame of image.

[0008] According to a fourth aspect of the present disclosure, there is provided an image processing device, comprising: an image determination module, configured to determine a reference frame image in a plurality of consecutive frame images, and determine at least one frame image other than the reference frame image as an image to be fused; an image segmentation module, configured to determine a foreground area of ​​the image to be fused; a sub-area extraction module, configured to extract at least one sub-area from the image to be fused; and a sub-area fusion module, configured to fuse the sub-areas in the image to be fused to the reference frame image respectively, wherein, when the sub-area is located in the foreground area, the larger pixel value of the sub-area in the image to be fused and the reference frame image is used as the pixel value of the sub-area after fusion.

[0009] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method of the first or second aspect and possible implementation methods thereof are implemented.

[0010] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the image processing method of the first or second aspect and possible implementation thereof by executing the executable instructions.

[0011] The technical solution disclosed in this disclosure has the following beneficial effects:

[0012] A technical solution is provided for simulating a long exposure effect by fusing multiple frames of images. On the one hand, this solution is applicable to electronic devices such as smart phones. In practical applications, normal exposure can be used to shoot multiple frames of images. Compared with the long exposure shooting method, it can avoid problems such as ghosting and overexposure caused by inappropriate exposure time, and is less affected by the shaking of the device body, thereby improving the image blur problem. On the other hand, this solution uses the maximum pixel value fusion for the foreground area image, which is equivalent to fusing the actual image of the moving light source or moving object, which can retain a clear and continuous motion trajectory while reducing the impact of noise, thereby improving the quality of the final output target image.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown;

[0015] Figure 2 A flowchart showing a method for image processing in this exemplary embodiment is shown;

[0016] Figure 3 A flowchart of a process for determining exposure conditions in this exemplary embodiment is shown;

[0017] Figure 4 A flowchart of extracting a foreground area image in this exemplary embodiment is shown;

[0018] Figure 5 A schematic diagram showing a method of extracting a foreground area image in this exemplary embodiment is shown;

[0019] Figure 6 A schematic diagram showing a method of determining a sub-region pixel value in this exemplary embodiment is shown;

[0020] Figure 7 A flowchart of an image registration process in this exemplary embodiment is shown;

[0021] Figure 8 A flowchart of a process for generating a target image in this exemplary embodiment is shown;

[0022] Fig. 9 A flow chart showing an image processing method in this exemplary embodiment;

[0023] Fig.10 A flowchart of another image processing method according to the exemplary embodiment is shown;

[0024] Fig.11 A flowchart showing another image processing method according to the exemplary embodiment;

[0025] Fig.12 A flowchart of another image processing method according to the exemplary embodiment is shown;

[0026] Fig.13 Continuous multiple frame images in this exemplary embodiment are shown;

[0027] Fig.14 Shown by Fig.13 The target image is obtained by fusing continuous multiple frames of images;

[0028] Fig.15 A schematic diagram showing the structure of an image processing device in this exemplary embodiment is shown;

[0029] Fig.16 A schematic diagram showing the structure of another image processing device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0031] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0033] The exemplary embodiments of the present disclosure first provide an image processing method, and its application scenarios include but are not limited to: in night scene shooting, continuously shooting multiple frames of a moving light source, and obtaining a streamer image with a continuous light track effect through fusion; continuously shooting multiple frames of water flow, and obtaining a waterfall or river image with a dynamic sense of rushing water through fusion.

[0034] The exemplary embodiment of the present disclosure also provides an electronic device for executing the image processing method of the exemplary embodiment. The electronic device can be a terminal device with a shooting function, such as a smart phone, a tablet computer, a wearable device, a drone, etc., or a computing device with an image processing function. For example, the terminal device sends the captured continuous multi-frame images to a server, which processes the images and returns the obtained target image to the terminal device. In general, the electronic device includes a processor and a memory. The memory is used to store executable instructions of the processor, and can also store application data, such as image data, video data, etc.; the processor is configured to execute the image processing method of the exemplary embodiment by executing executable instructions.

[0035] Below Figure 1 Taking the mobile terminal 100 in FIG. 1 as an example, the structure of the above electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 1 The construction in can also be applied to fixed type equipment.

[0036] like Figure 1 As shown, the mobile terminal 100 may specifically include: a processor 110, an internal memory 121, an external memory interface 122, a USB (Universal Serial Bus) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 171, a receiver 172, a microphone 173, an earphone interface 174, a sensor module 180, a display screen 190, a camera module 191, an indicator 192, a motor 193, a button 194 and a SIM (Subscriber Identification Module) card interface 195, etc.

[0037] The processor 110 may include one or more processing units. For example, the processor 110 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0038] The encoder can encode (i.e. compress) the image or video data, for example, encode the continuous multi-frame images captured to form corresponding code stream data to reduce the bandwidth occupied by data transmission; the decoder can decode (i.e. decompress) the code stream data of the image or video to restore the image or video data, for example, decode the code stream data of the continuous multi-frame images to obtain complete image data, so as to facilitate the execution of the image processing method of this exemplary embodiment. The mobile terminal 100 can support one or more encoders and decoders. In this way, the mobile terminal 100 can process images or videos in a variety of encoding formats, such as: JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap) and other image formats, MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding) and other video formats.

[0039] In one implementation, the processor 110 may include one or more interfaces, and may be connected to other components of the mobile terminal 100 via different interfaces.

[0040] The internal memory 121 may be used to store computer executable program codes, which include instructions. The internal memory 121 may include volatile memory and non-volatile memory. The processor 110 executes various functional applications and data processing of the mobile terminal 100 by running the instructions stored in the internal memory 121.

[0041] The external memory interface 122 can be used to connect an external memory, such as a Micro SD card, to expand the storage capacity of the mobile terminal 100. The external memory communicates with the processor 110 through the external memory interface 122 to implement data storage functions, such as storing audio, video and other files.

[0042] The USB interface 130 is an interface that complies with USB standard specifications and can be used to connect a charger to charge the mobile terminal 100 , or to connect headphones or other electronic devices.

[0043] The charging management module 140 is used to receive charging input from a charger. While charging the battery 142, the charging management module 140 can also power the device through the power management module 141; the power management module 141 can also monitor the status of the battery.

[0044] The wireless communication function of the mobile terminal 100 can be implemented by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the mobile terminal 100. The wireless communication module 160 can provide wireless communication solutions including WLAN (Wireless Local Area Networks) (such as Wi-Fi (Wireless Fidelity) network), BT (Bluetooth), GNSS (Global Navigation Satellite System), FM (Frequency Modulation), NFC (Near Field Communication), IR (Infrared) and the like applied to the mobile terminal 100.

[0045] The mobile terminal 100 can implement a display function and display a user interface through the GPU, the display screen 190 and the AP, etc. For example, when the user turns on the shooting function, the mobile terminal 100 can display a shooting interface and a preview image on the display screen 190 .

[0046] The mobile terminal 100 can realize the shooting function through the ISP, the camera module 191, the encoder, the decoder, the GPU, the display screen 190 and the AP.

[0047] The mobile terminal 100 can implement audio functions through the audio module 170, the speaker 171, the receiver 172, the microphone 173, the earphone interface 174 and the AP.

[0048] The sensor module 180 may include a depth sensor 1801 , a pressure sensor 1802 , a gyroscope sensor 1803 , an air pressure sensor 1804 , etc., to implement corresponding sensing detection functions.

[0049] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power change, message, missed call, notification, etc. The motor 193 may generate a vibration prompt, or may be used for touch vibration feedback, etc. The buttons 194 include a power button, a volume button, etc.

[0050] The mobile terminal 100 may support one or more SIM card interfaces 195 for connecting a SIM card to implement functions such as calls and mobile communications.

[0051] Combine the following Figure 2The image processing method of this exemplary embodiment is described. Figure 2 An exemplary process of an image processing method is shown, which may include:

[0052] Step S210, acquiring multiple consecutive frames of images;

[0053] Step S220, extracting foreground area images of at least two frames of images;

[0054] Step S230, fusing each foreground sub-region according to the maximum pixel value of each foreground sub-region in each foreground region image to form a foreground fused image, wherein the foreground sub-region is a sub-region of the foreground region image;

[0055] Step S240, generating a target image based on the foreground fused image and the background area image of at least one frame of image.

[0056] based on Figure 2 The method provides a technical solution for simulating long exposure effects by fusing multiple frames of images. On the one hand, this solution is suitable for electronic devices such as smart phones. In practical applications, it is sufficient to shoot multiple frames of images with normal exposure. Compared with the long exposure shooting method, it can avoid problems such as ghosting and overexposure caused by inappropriate exposure time, and is less affected by the shaking of the device body, thereby improving the image blur problem. On the other hand, this solution uses the maximum pixel value fusion for the foreground area image, which is equivalent to fusing the actual image of the moving light source or moving object, which can retain a clear and continuous motion trajectory while reducing the impact of noise, thereby improving the quality of the final output target image.

[0057] Below Figure 2 Each step in the procedure is described in detail.

[0058] In step S210, multiple consecutive image frames are acquired.

[0059] The continuous multiple frames of images may be images obtained by continuously shooting a certain area or object with a lens, and the areas shot by the continuous multiple frames of images are basically the same, but may be slightly offset due to the shaking of the camera body during shooting.

[0060] In one implementation, step S210 may include the following steps:

[0061] Determine exposure conditions based on ambient lighting parameters;

[0062] Acquire multiple frames of images captured continuously under the above exposure conditions.

[0063] Among them, the ambient light parameter is a parameter that characterizes the intensity of the ambient light. It can be detected by an ambient light sensor configured by the electronic device, or determined by detecting the brightness of a preview image. For example, when shooting, a preview image is captured with a predetermined exposure time (Exp_time) and sensitivity (ISO), and the brightness level of the preview image is detected to estimate the ambient light level and obtain the corresponding ambient light parameter.

[0064] Generally, the larger the ambient light parameter value, the stronger the ambient light. In order to obtain the appropriate exposure, you can set lower exposure conditions, such as shorter exposure time, lower sensitivity, etc., to prevent overexposure, underexposure and other adverse conditions. For example, the ambient light parameters can be divided into three brightness ranges, representing darker, medium brightness and brighter environments respectively; each brightness range corresponds to a reference exposure time, such as a reference exposure time of 1.5s for a darker environment, 1s for a medium brightness environment, and 0.5s for a brighter environment; after obtaining the current ambient light parameters, determine the corresponding reference exposure time according to the brightness range in which it is located.

[0065] After determining the exposure conditions that are suitable for the ambient lighting parameters, the exposure conditions are used for shooting to obtain continuous multiple-frame images.

[0066] In one embodiment, reference Figure 3 As shown, the above-mentioned determination of exposure conditions according to the ambient light parameters may include the following steps:

[0067] Step S310, determining a corresponding first exposure time according to an ambient light parameter;

[0068] Step S320, obtaining a second exposure time corresponding to the minimum sensitivity;

[0069] Step S330, determining the exposure condition according to the maximum value of the first exposure time and the second exposure time, and the minimum sensitivity.

[0070] The first exposure duration may be an exposure duration adapted to the ambient lighting parameters, such as the reference exposure duration described above. The system may pre-configure a corresponding relationship between the ambient lighting parameters and the exposure duration, and the two are generally positively correlated, such as linear positive correlation, interval positive correlation, etc., so that the current ambient lighting parameters may be mapped to the corresponding first exposure duration.

[0071] The sensitivity is essentially the speed of light sensitivity of the photosensitive element. The minimum sensitivity is the minimum speed of light sensitivity, which is usually determined by the performance of the camera itself. For example, the minimum sensitivity of the camera on a smartphone can be 50, 100, etc. Sensitivity and exposure time are two related parameters. For example, a fixed exposure value (EV, such as setting EV = 0) can be set, and the corresponding exposure time can be calculated by the sensitivity. The higher the sensitivity, the shorter the exposure time. This exemplary embodiment determines to use the minimum sensitivity as the sensitivity in actual shooting, and the corresponding exposure time is the longest, which is the second exposure time.

[0072] The system can pre-configure the correspondence between sensitivity and exposure time. The two are generally negatively correlated. Different correspondence tables can be configured for different ambient lighting parameters. For example, as the ambient lighting parameters increase, the corresponding exposure time for the same sensitivity is shortened.

[0073] This exemplary embodiment determines to use the maximum value of the first exposure time and the second exposure time as the exposure time in actual shooting. If the first exposure time is greater than the second exposure time, the first exposure time is used, otherwise the second exposure time is used.

[0074] In actual shooting, if a higher sensitivity is used, a shorter exposure time needs to be set, otherwise overexposure is likely to occur. A certain amount of time needs to be left between the exposures of two adjacent frames, which can easily cause the position of an object in the two adjacent frames to jump, which is not conducive to subsequent image fusion. Figure 3 The exposure condition determined by the method can ensure a longer exposure time when shooting a single-frame image, which is conducive to forming the effects of smear and light tracks in the single-frame image, and the continuity between different frame images is stronger, which is conducive to the subsequent fusion of high-quality target images.

[0075] Continue to refer Figure 2 In step S220, foreground area images of at least two frames of images are extracted.

[0076] For the sake of convenience and to distinguish from the images generated in the subsequent processing, the continuous frame images obtained in step S210 are recorded as the original images. Step S220 is essentially to segment the original image into a foreground area and a background area. The foreground area can be the moving part in the original image, and correspondingly, the background area can be the static part in the original image.

[0077] The at least two frames of images mentioned above may be any two or more frames of images in a continuous plurality of frames of original images. In one embodiment, at least two frames of original images with higher quality may be selected by screening according to image brightness, image clarity, etc. For example, the gradient of each frame of original images is calculated, images with gradients below a certain threshold are removed, and foreground area images are extracted from the remaining original images, thereby improving the quality of subsequent image fusion. In one embodiment, a foreground area image may be extracted for each frame of the continuous plurality of frames of original images, that is, each frame of the image captured participates in the fusion of subsequent foreground area images, thereby ensuring that the information for subsequent image fusion is relatively sufficient.

[0078] For each frame of the original image, the moving object can be detected to segment the foreground and background, thereby extracting the foreground area image. For example, the original image can be semantically segmented to determine the semantic classification results corresponding to each part, and to determine whether each semantic is a moving object, thereby determining which parts of the image are moving objects.

[0079] In one embodiment, reference Figure 4 As shown, step S220 may include the following steps:

[0080] Step S410, subtracting at least two frames of the to-be-segmented images in the above-mentioned continuous multiple frames of images from the reference frame images in the above-mentioned continuous multiple frames of images respectively, to obtain a difference image corresponding to each frame of the to-be-segmented images;

[0081] Step S420, generating a foreground mask image corresponding to each frame of the image to be segmented according to the comparison result between each pixel value in the difference image and a preset threshold;

[0082] Step S430: extracting a foreground area image from the image to be segmented using the foreground mask image.

[0083] Among them, the image to be segmented is the original image used for foreground fusion later, that is, the at least two frames of images mentioned above. The reference frame image provides reference information for moving object detection, and can be any frame image in a continuous plurality of original frames, for example, it can be the first frame image therein. After the image to be segmented is subtracted from the reference frame image, the pixel difference of the part where the moving object is located is larger, and the pixel difference of the background part is smaller, and the pixel value of each pixel point in the obtained difference image is the pixel difference of the two images; it should be noted that in the case where the pixel difference has a negative value, the absolute value of the pixel difference can be used in this exemplary embodiment as the pixel value in the difference image. The preset threshold is the pixel difference standard for dividing the moving object and the stationary object, which can be determined based on experience or actual scenes. In the difference image, if the pixel value of a certain pixel point is greater than the preset threshold, it belongs to the foreground area, otherwise it belongs to the background area, thereby realizing the segmentation of the foreground and the background, and obtaining the foreground mask image. Step S420 is actually equivalent to binarizing the difference image. Finally, the foreground mask image is multiplied by the image to be segmented to obtain the foreground area image, which retains the complete information of the foreground part and the pixel values ​​of the background part are all 0.

[0084] Below through Figure 5 right Figure 4 An example of the process is to obtain k consecutive frames of original image I 0 , I 1 , I 2 ,…,I k-1 ; Select the first frame I 0 is the reference frame image, and each frame image is used as the image to be segmented; 1 , I 2 ,…,I k-1 with I 0 Subtract and get the difference image D 1 , D 2 , …, D k-1 ; The difference image D 1 , D 2 , …, D k-1 The pixel value of each pixel in D is compared with the preset threshold. If the pixel value is greater than the preset threshold, the mask value of the pixel is 1, otherwise it is 0. This process is equivalent to 1 , D 2 , …, D k-1 Binarization is performed to generate a foreground mask image M 1 , M 2 , …, M k-1 ; Finally, multiply each frame of the foreground mask image with the corresponding image to be segmented to obtain the foreground area image F 1 , F 2 , …, F k-1 .

[0085] Let i represent any frame from 1 to k-1, then the above process can be expressed by the following formula:

[0086]

[0087] F i (x,y)=M i (x,y)·I i (x,y) (2)

[0088] (x, y) represents any pixel in the i-th frame image; T is the preset threshold. Formula (1) means that after subtracting the image to be segmented from the reference frame image, the pixel value difference is compared with the preset threshold T to determine whether the mask value of each pixel is 1 or 0, thereby obtaining the foreground mask image M. Formula (2) means that the foreground mask image is multiplied by the image to be segmented to obtain the foreground area image.

[0089] For different frames of images to be segmented, after subtracting them from the reference frame images to obtain a difference image, a unified preset threshold can be used to compare the pixel values ​​in the difference image to determine the foreground mask image, or different preset thresholds can be used. In one embodiment, after obtaining the difference image, the distribution of pixel values ​​in the difference image can be statistically analyzed to determine the preset threshold. For example, the preset threshold is determined by using the histogram bimodal method, the maximum inter-class variance method, the two-dimensional entropy threshold segmentation method, the local threshold method, etc., and then the corresponding foreground mask image is generated according to the preset threshold. As a result, different frames of images to be segmented use different preset thresholds, which increases the flexibility of image segmentation and is conducive to achieving more accurate foreground area segmentation.

[0090] Taking into account the Figure 4 The obtained foreground mask image and foreground area image may have edge burrs and the like. In one embodiment, the foreground mask image may also be subjected to morphological processing, such as image corrosion and dilation, so that the edge of the foreground mask image is smooth and the morphology is more complete, thereby further improving the quality of the extracted foreground area image.

[0091] It should be added that Figure 4 In the process, if the image to be segmented includes a reference frame image, the foreground mask image corresponding to any other frame of the image to be segmented can be used to extract the foreground area image of the reference frame image. Figure 5 The foreground mask image M is shown 1 With the reference frame image I 0 Multiply them together to get the foreground image of the reference frame, which can be recorded as F 0 .

[0092] Continue to refer Figure 2In step S230, each foreground sub-region is fused according to the maximum pixel value of each foreground sub-region in each foreground region image to form a foreground fused image, wherein the foreground sub-region is a sub-region of the foreground region image.

[0093] The foreground sub-regions may be sub-regions obtained by dividing the foreground image in any manner. For example, every 2*2 or 3*3 pixels in the foreground image may be regarded as a foreground sub-region, or each pixel in the foreground image may be regarded as a foreground sub-region, or the foreground sub-regions may be divided according to the color distribution of the foreground image, etc. It should be noted that the foreground sub-regions may be divided in the same manner for different foreground images, so that each foreground sub-region is located at the same position in different foreground images.

[0094] In this exemplary embodiment, the foreground area image is divided into different foreground sub-areas, and for each foreground sub-area, the maximum pixel value is selected from each foreground area image, and then each foreground sub-area is spliced ​​into a foreground fused image. Step S230 is actually to perform maximum pixel value fusion on the foreground area images extracted in step S220, that is, to fuse each foreground area image into a foreground fused image, and the maximum pixel value of each foreground sub-area is retained during fusion.

[0095] In one implementation, each pixel in the foreground area image can be used as a foreground sub-area, the pixel value of the pixel is the pixel value of the foreground sub-area, and the maximum pixel value of the pixel is selected from each foreground area image as the pixel value of the pixel in the foreground fused image. In this way, pixel-level foreground fusion can be achieved, which is conducive to improving the precision of foreground fusion.

[0096] In one implementation, step S230 may include the following steps:

[0097] For each foreground sub-region, determining pixel statistics of the foreground sub-region in each foreground region image respectively;

[0098] Determine the foreground region image corresponding to the maximum pixel statistics value of the foreground sub-region as the target foreground region image corresponding to the foreground sub-region;

[0099] The pixel value of the foreground sub-region in the target foreground region image is used as the pixel value of the foreground sub-region in the foreground fused image.

[0100] refer to Figure 6 An example of the above process is given below: In the foreground area image F 0 , F 1 , F 2 , …, F k-1Extract the foreground sub-region A from the image, and record the image block corresponding to A in each foreground region image as A_F 0 , A_F 1 , A_F 2 ,…,A_F k-1 , A_F 0 , these image blocks have the same position, shape, and size in their respective images, such as A_F 0 In F 0 The position, shape, size and A_F 1 In F 1 The position, shape and size of the A_F 0 , A_F 1 , A_F 2 ,…,A_F k-1 Perform pixel statistics to obtain pixel statistics values. The pixel statistics values ​​may be pixel average values, pixel maximum values, brightness average values, etc., which are not limited in the present disclosure. It should be noted that when the foreground sub-region is only one pixel, the average values ​​of the pixel values ​​of each channel of the pixel can be counted, or the pixel value can be converted into a brightness value (or grayscale value) and the brightness value can be used as the pixel statistics value. The pixel average value provides a way to measure the pixel value of the foreground sub-region, thereby comparing the relative size of the pixel values ​​of the same foreground sub-region in different foreground region images. Taking the pixel average value as an example, the statistics A_F 0 The average pixel value of each pixel in is denoted as P(A_F 0 ), and the same statistics give P(A_F 1 ), P(A_F 2 ), …, P(A_F k-1 ); compare P(A_F 0 ), P(A_F 1 ), P(A_F 2 ), …, P(A_F k-1 ), determine the maximum pixel statistics, assuming it is P(A_F 2 ), then determine the corresponding foreground area image F 2 is the target foreground area image corresponding to the foreground sub-area A; then the foreground sub-area A is selected in F 2 The pixel values ​​in are used as the pixel values ​​of the foreground sub-region A in the foreground fused image. Based on the same method, the pixel values ​​of other foreground sub-regions can be determined and finally spliced ​​into the foreground fused image.

[0101] In the foreground fusion image, the maximum pixel value of each foreground sub-region or each pixel point in the foreground part of the continuous multi-frame image is retained, thereby corresponding to the actual position of the moving light source or moving object to a certain extent. Figure 6 In the foreground sub-region A, the foreground region image F2 The pixel statistics value in the foreground area is the largest, indicating that the image F 2 (or the original image I 2 ) is the actual position of the moving light source, while in other frame images, the foreground sub-region A may be the smear of the moving light source. When fusing the foreground area images, selecting the maximum pixel value of each foreground sub-region in different foreground area images is equivalent to fusing the actual image of the moving light source or moving object, which is conducive to achieving the long-exposure motion trajectory effect. In addition, the maximum pixel value fusion method is used for the moving part to reduce the influence of noise and improve image quality.

[0102] Considering that there may be a slight offset of the shooting area between different frames of original images, in order to ensure the accuracy of image fusion, the different frames of original images can be registered. For example, after obtaining multiple consecutive frames of images in step S210, the multiple frames of images are first registered, and then the subsequent steps S220 to S240 are performed based on the registered images.

[0103] In one implementation, the registration may be performed after step S220, specifically including:

[0104] The at least two frames of images are registered according to the background area images of the at least two frames of images.

[0105] The background area image is the part of the original image other than the foreground area image. Figure 5 In the above example, we get the foreground mask image M. 1 , M 2 , …, M k-1 After that, the inverse mask operation is performed, that is, the 0 / 1 value in the foreground mask image is reversed to obtain the background mask image RM 1 , R.M. 2 , …, RM k-1 Then, the background mask image RM 1 , R.M. 2 , …, RM k-1 Respectively with the corresponding original image I 1 , I 2 ,…,I k-1 Multiply them to get the background area image B 1 , B 2 , …, B k-1 . Then according to the background area image B 1 , B 2 , …, B k-1 Perform the registration of the original images. Since the image content of the background area of ​​the original images of different frames is basically the same, the registration using the background area image will not be disturbed by the moving objects in the foreground area, which is conducive to improving the registration accuracy.

[0106] It should be noted that any background mask image can be used to extract the background area image from the reference frame image. For example, the background mask image RM 1 , and the reference frame image I 0 Multiply them to get the background area image B of the reference frame image 0 .

[0107] The purpose of registration is to ensure the accuracy of subsequent foreground area image fusion, so only the images to be segmented in the original image can be registered. When all the original images are selected as the images to be segmented, all the original images are registered.

[0108] In one embodiment, the at least two frames of images may include a reference frame image, and during registration, all images other than the reference frame image may be registered to the reference frame image. 0 , I 1 , I 2 ,…,I k-1 Select the first frame I 0 is the reference frame image, and I 1 , I 2 ,…,I k-1 All towards I 0 Registration, thus unifying the registration standards, and finally achieving I 0 The registration of all original images used as reference actually realizes the registration between any two frames of original images.

[0109] In one embodiment, reference Figure 7 As shown, according to the background area images of the at least two frames of images, the at least two frames of images are registered, which can be achieved by the following steps:

[0110] Step S710, extracting feature points from the background area images of the two frames of images respectively;

[0111] Step S720, performing feature point matching on the background area images of the two frames of images to obtain matching point pairs;

[0112] Step S730, determining a homography matrix according to the matching point pairs;

[0113] Step S740: align one of the two image frames with the other image frame by using a homography matrix.

[0114] Feature points are representative and highly recognizable points or areas in an image, such as corners and boundaries in an image. In the background area image of the original image, gradients at different positions can be detected, and feature points can be extracted at positions with larger gradients. Generally, after extracting feature points, they need to be described, for example, by using an array to describe the pixel distribution characteristics around the feature points, which is called the description information (or descriptor, descriptor) of the feature points. The description information of the feature points can be regarded as local description information of the background area image. The present disclosure does not specifically limit the types of feature points and their description information. For example, the Harris corner algorithm can be used to extract and describe feature points, or FAST (Features From Accelerated Segment Test, features based on accelerated segmentation detection), BRIEF (Binary Robust Independent Elementary Features, binary robust independent basic features), ORB (Oriented FAST and Rotated BRIEF, oriented FAST and rotated BRIEF), SIFT (Scale-Invariant Feature Transform, scale-invariant feature transform and other algorithms can be used.

[0115] A matching point pair refers to a pair of relatively similar feature points in two background area images, which can be considered as the projection of the same object point in the real world onto the two background area images. By calculating the similarity of the description information of the two feature points, it can be determined whether the two feature points are a matching point pair. In general, the description information of the feature points can be represented as a vector, and the L1 distance or L2 distance between the description information vectors of the two feature points is calculated. By traversing all the point pairs between the two background area images and sorting them from small to large according to the L1 distance or L2 distance, the top point pairs (such as the top 15% of the point pairs, or the top n point pairs, where the value of n can be set according to actual needs) are selected as matching point pairs.

[0116] The homography matrix is ​​a matrix that describes the position transformation relationship between two background area images. Generally, the homography matrix can be solved by 4 matching point pairs. If the number of matching point pairs obtained in step S720 is greater than 4, 4 point pairs can be randomly selected, or 4 matching point pairs with the smallest L1 distance or L2 distance can be selected. When solving the homography matrix, it is also possible to iteratively optimize based on the geometric constraint relationship between the background area images, such as epipolar constraints, using algorithms such as RANSAC (Random Sample Consensus, random sampling consistency) to obtain the optimal solution for the homography matrix, so as to further improve the accuracy of image registration.

[0117] The homography matrix can be used to perform an affine transformation on one of the two image frames to align it with the other image frame. For example, if the two image frames include a reference frame image, the non-reference frame image is aligned with the reference frame image, thereby completing the alignment of the two image frames.

[0118] Refer to the above Figure 5 , foreground mask image M 1 , M 2 , …, M k-1 There may be differences between them, that is, the segmentation of the foreground and background of different frame images is not exactly the same, so the range corresponding to the background area of ​​different frame original images may be different.

[0119] In one embodiment, in order to make the background area of ​​the reference frame image consistent with the background area range of other original images, the background mask images of different frames can be used to extract the background area image of the reference frame image, so as to align the original images of different frames. 1 , I 2 ,…,I k-1 To the reference frame image I 0 Registration can be done by comparing the k-1 frame images with the reference frame image I 0 Pairing, get the image pair (I 0 , I 1 ), (I 0 , I 2 ),…,(I 0 , I k-1 ). For the image pair (I 0 , I 1 ), using I 1 The background mask image RM 1 , respectively with I 0 ,I 1 Multiply them together to get two background area images, denoted as B 0_1 , B 1 ; Then calculate B 0_1 With B 1 The homography matrix between them, so that I 1 To I 0 Registration; for an image pair (I 0 , I 2 ), using I 2 The background mask image RM 2 , respectively with I 0 ,I 2 Multiply them together to get two background area images, denoted as B 0_2 , B 2 ; Then calculate B 0_2 With B 2 The homography matrix between them, so that I2 To I 0 Registration... This completes the registration of each frame of the original image to the reference frame image I 0 This can maximize the use of the consistency of the background area for registration, which is beneficial to improving the registration accuracy.

[0120] In another implementation, the background areas of the original images of different frames may also be unified. Specifically, before the registration is performed, the following steps may be performed:

[0121] When the background areas corresponding to the background area images do not completely overlap, the background areas corresponding to the background area images are intersected to obtain a common background area;

[0122] The portion of each background area image that is outside the common background area is deleted.

[0123] For example, the background mask image RM 1 , R.M. 2 , …, RM k-1 Take the intersection, that is, traverse each pixel in the background mask image. If the value of the pixel in all background mask images is 1, it is determined that the pixel is in the common background area, otherwise it is in the foreground area, thus obtaining the common background mask image, recorded as RM inter Then use RM inter Respectively with the above background area image B 0 , B 1 , B 2 , …, B k-1 (or respectively with the original image I 0 , I 1 , I 2 ,…,I k-1 ) is equivalent to deleting the part outside the common background area (actually setting the pixels outside the common background area to 0), thereby optimizing the background area image. Subsequently, registration can be performed based on the optimized background area image, which can further improve the accuracy of registration.

[0124] In one implementation, the foreground region image may be extracted again from the at least two registered frames of images to perform step S230. The extracted foreground region image is registered and has position consistency, which is conducive to accurate fusion.

[0125] Since the segmentation of foreground and background in different frames is not exactly the same, the area ranges corresponding to the foreground area images in different frames may be different, for example Figure 5 Middle foreground image F 0 , F 1 , F 2, …, F k-1 There are slight differences in the shape of the foreground part. In order to achieve more accurate foreground fusion, in one embodiment, before step S230, the following steps may be performed:

[0126] When the foreground areas corresponding to the foreground area images do not completely overlap, the foreground areas corresponding to the foreground area images are intersected to obtain a common foreground area;

[0127] The portion of each foreground area image that is outside the common foreground area is deleted.

[0128] For example, the foreground mask image M 1 , M 2 , …, M k-1 Take the intersection, that is, traverse each pixel in the foreground mask image. If the value of the pixel in all foreground mask images is 1, it is determined that the pixel is in the common foreground area, otherwise it is in the background area. Thus, the common foreground mask image is obtained, which is recorded as M inter Then use M inter Respectively with the foreground area image F 0 , F 1 , F 2 , …, F k-1 (or respectively with the original image I 0 , I 1 , I 2 ,…,I k-1 ) is equivalent to deleting the part outside the common foreground area (actually, setting the pixels outside the common foreground area to 0), thereby optimizing the foreground area image. Subsequently, step S230 can be performed based on the optimized foreground area image, which can further improve the accuracy of fusion, especially improve the error caused by inconsistent segmentation of the foreground and background boundary.

[0129] Continue to refer Figure 2 In step S240, a target image is generated based on the foreground fusion image and the background area image of at least one frame of image.

[0130] In the foreground fused image, the pixel value of the background part is 0, which can be spliced ​​with the background area image of any frame of the original image to obtain a complete image, that is, the target image.

[0131] In one embodiment, reference Figure 8 As shown, step S240 may include:

[0132] Step S810, fusing background area images of at least two frames of images to obtain a background fused image;

[0133] Step S820: generating a target image based on the foreground fused image and the background fused image.

[0134] Among them, at least two frames of images in step S810 are used for background fusion, which can be the same as or different from the at least two frames of images used for foreground fusion in step S220, and the present disclosure does not limit this. In one embodiment, all original images can be selected to perform background fusion in step S810. The background fused image obtained after fusion of multiple frames of background area images has richer detail information than the original background area image, and can reduce defects such as blur and ghosting that may exist in a single background area image, thereby further splicing the foreground fused image and the background fused image to obtain the target image, which is beneficial to improving the quality of the target image.

[0135] The present disclosure does not limit the specific method of background fusion. In one embodiment, step S810 can be implemented by the following steps:

[0136] The pixel values ​​of each pixel in the background area in each background area image are averaged to obtain a background fused image.

[0137] That is, the pixel points at the same position in each background area image are averaged as the pixel value of the pixel point in the background fusion image, thereby realizing the fusion of the pixel average values ​​of each background area image. Therefore, the maximum pixel value fusion is used for the foreground part, and the average pixel value fusion is used for the background part, which can effectively improve the noise problem of the target image and ensure the synthesis of a continuous and clear motion trajectory effect.

[0138] Fig. 9 The flowchart of an image processing method in this exemplary embodiment is shown, and the entire image processing process is divided into three parts:

[0139] Image registration. Get k consecutive frames of original images I 0 , I 1 , I 2 ,…,I k-1 , k is a positive integer greater than 1; select a reference frame image, for example, the first frame I 0 As the reference frame image, subtract the original images of other frames from the reference frame image to obtain the difference image D 1 , D 2 , …, D k-1 ; Binarize the difference image to obtain the foreground mask image M 1 , M 2 , …, M k-1 , and the background mask image RM 1 , R.M. 2 , …, RM k-1, the foreground mask image and the background mask image are complementary, that is, the foreground mask image is the inverse mask image of the background mask image; for RM 1 , R.M. 2 , …, RM k-1 Find the intersection and get the common background mask image RM inter ; The common background mask image RM inter Respectively with the original image I 0 , I 1 , I 2 ,…,I k-1 Multiply them to get the background area image B before registration 0 , B 1 , B 2 , …, B k-1 ; B 1 , B 2 , …, B k-1 With B 0 Perform feature point matching and calculate the homography matrix H 1 , H 2 , …, H k-1 ; Use the homography matrix to transform I 1 , I 2 ,…,I k-1 To I 0 Registration, thus obtaining k consecutive frames of registered images I 0 , I 1 ',I 2 ', ..., I k-1 '.

[0140] Image segmentation. For the foreground mask image M 1 , M 2 , …, M k-1 , find the intersection and get the common foreground mask image M inter ; The common foreground mask image M inter Respectively with the registration image I 0 , I 1 ',I 2 ', ..., I k-1 'Multiply to get the registered foreground area image F 0 ', F 1 ', F 2 ', ..., F k-1 '; The public foreground mask image M inter The inverse mask image (1-M inter )(Note, 1-M inter With RM inter are usually not equal) and the registered image I 0 , I 1 ',I 2', ..., I k-1 'Multiply to get the registered background area image B 0 ', B 1 ', B 2 ', ..., B k-1 '.

[0141] Image fusion. The registered foreground image F 0 ', F 1 ', F 2 ', ..., F k-1 'Perform pixel maximum fusion to obtain the foreground fusion image F fu ; The registered background area image B 0 ', B 1 ', B 2 ', ..., B k-1 'Perform pixel average fusion to obtain background fusion image B fu ; Splice foreground fusion image F fu Fusion image B with background fu , when stitching, the pixel values ​​of the two images can be added to obtain the target image I out Target image I out It has the effect of simulating long exposure images.

[0142] Fig.10 A flow chart of another image processing method in this exemplary embodiment is shown. Fig. 9 Similar to the process architecture diagram, the entire image processing process is divided into three parts:

[0143] Image registration. Get k consecutive frames of original images I 0 , I 1 , I 2 ,…,I k-1 , k is a positive integer greater than 1; select a reference frame image, for example, the first frame I 0 As the reference frame image, subtract the original images of other frames from the reference frame image to obtain the difference image D 1 , D 2 , …, D k-1 ; Binarize the difference image, set the pixel points with pixel values ​​higher than the preset threshold to 0, and set the pixel points with pixel values ​​lower than the preset threshold to 1, thereby obtaining the background mask image RM 1 , R.M. 2 , …, RM k-1 ; for RM 1 , R.M. 2 , …, RM k-1 Find the intersection and get the common background mask image RM inter ; The common background mask image RMinter Respectively with the original image I 0 , I 1 , I 2 ,…,I k-1 Multiply them to get the background area image B before registration 0 , B 1 , B 2 , …, B k-1 ; B 1 , B 2 , …, B k-1 With B 0 Perform feature point matching and calculate the homography matrix H 1 , H 2 , …, H k-1 ; Use the homography matrix to transform I 1 , I 2 ,…,I k-1 To I 0 Registration, thus obtaining k consecutive frames of registered images I 0 , I 1 ',I 2 ', ..., I k-1 '.

[0144] Image segmentation. 0 Other frames outside the registration image and I 0 Subtract and get the registered difference image D 1 ', D 2 ', ..., D k-1 '; For the difference image D 1 ', D 2 ', ..., D k-1 'Perform binarization processing, set the pixel points with pixel values ​​higher than the preset threshold in the difference image to 1, and the pixel points with pixel values ​​lower than the preset threshold to 0, thereby obtaining the registered foreground mask image M 1 ',M 2 ', ..., M k-1 '; for M 1 ',M 2 ', ..., M k-1 'Find the intersection and get the registered common foreground mask image M inter '; The registered common foreground mask image M inter 'Respectively with the registration image I 0 , I 1 ',I 2 ', ..., I k-1 'Multiply to get the registered foreground area image F 0 ', F 1 ', F 2 ', ..., Fk-1 ';Write M inter 'The inverse mask image (1-M inter ') respectively with the registration image I 0 , I 1 ',I 2 ', ..., I k-1 'Multiply to get the registered background area image B 0 ', B 1 ', B 2 ', ..., B k-1 '.

[0145] Image fusion. The registered foreground image F 0 ', F 1 ', F 2 ', ..., F k-1 'Perform pixel maximum fusion to obtain the foreground fusion image F fu ; The registered background area image B 0 ', B 1 ', B 2 ', ..., B k-1 'Perform pixel average fusion to obtain background fusion image B fu ; Splice foreground fusion image F fu Fusion image B with background fu , when stitching, the pixel values ​​of the two images can be added to obtain the target image I out Target image I out It has the effect of simulating long exposure images.

[0146] The exemplary embodiment of the present disclosure also provides another image processing method. Fig.11 As shown, the image processing method includes the following steps S1110 to S1140:

[0147] Step S1110 , determining a reference frame image from a plurality of continuous frame images, and determining at least one frame image other than the reference frame image as an image to be fused.

[0148] Among them, the continuous multi-frame images can be the images obtained by continuously shooting with the lens aimed at a certain area or object. The specific method of obtaining the continuous multi-frame images can refer to the content of the above-mentioned step S210. The reference frame image is an image that provides an image fusion benchmark, and can be any frame image in the continuous multi-frame original image, for example, it can be the first frame image therein. The image to be fused is the image that needs to be fused with the reference frame image, and can be any frame or multiple frames other than the reference frame image. For example, in the continuous multi-frame images, the first frame image is used as the reference frame image, and each of the other frames is used as the image to be fused. In one embodiment, it is also possible to screen according to image brightness, image clarity, etc. to select the image to be fused with higher quality.

[0149] Step S1120, determining the foreground area of ​​the image to be fused.

[0150] The foreground area can be the moving part in the image to be fused, and the corresponding static part is the background area. The foreground area and the background area are usually complementary. The foreground and background can be segmented by detecting the moving object in the image to be fused, thereby determining the foreground area.

[0151] In one implementation, step S1120 may include the following steps:

[0152] Subtract the image to be fused from the reference frame image to obtain a difference image corresponding to the image to be fused;

[0153] According to the comparison result of each pixel value in the difference image and the preset threshold, the foreground area of ​​the image to be fused is determined.

[0154] The above steps can refer to Figure 4 and Figure 5 The preset threshold is the pixel difference standard for dividing moving objects and stationary objects, which can be determined based on experience or actual scenes. In the difference image, if the pixel value of a certain pixel is greater than the preset threshold, it belongs to the foreground area, otherwise it belongs to the background area, thereby realizing the segmentation of the foreground and background, determining the foreground area of ​​the image to be fused, and also determining the background area.

[0155] Step S1130: extract at least one sub-region from the image to be fused.

[0156] The sub-region may be a sub-region obtained by dividing the image to be fused in any form. For example, every 2*2 or 3*3 pixels in the image to be fused is regarded as a sub-region, or each pixel in the image to be fused is regarded as a sub-region, or the sub-region is divided according to the color distribution of the image to be fused, etc. The sub-region extracted in step S1130 may be located in the foreground region, and be a foreground sub-region, or may be located in the background region, and be a background sub-region.

[0157] In one embodiment, a portion of the sub-region in the image to be fused can be extracted according to actual needs, usually the sub-region that needs to be fused to the reference frame image, for example, the sub-region is extracted only from the foreground region, or the gradient of each sub-region in the image to be fused is calculated, and the sub-region whose gradient is greater than a certain threshold (i.e., the high-frequency sub-region) is extracted.

[0158] In one implementation, the image to be fused may be divided into a plurality of sub-regions, and each sub-region is extracted, that is, all sub-regions in the image to be fused are extracted, which is equivalent to fusing the entire image to be fused to the reference frame image.

[0159] In one implementation, each pixel in the image to be fused can be determined as a sub-region. In this way, each pixel is used as a sub-region and fused to the reference frame image in subsequent processing, thereby achieving pixel-level image fusion, which is conducive to improving the precision of image fusion.

[0160] In one embodiment, the image to be fused may be registered with the reference frame image to improve the accuracy of subsequent image fusion. Figure 2 The content of at least two frames of images to be registered.

[0161] It should be noted that the registration can be performed after the reference frame image and the image to be fused are determined in step S1110, or the registration can be performed based on the background area after the foreground area and background area of ​​the image to be fused are determined in step S1120. After the registration, the foreground area of ​​the image to be fused can be re-determined.

[0162] Step S1140, respectively fuse the sub-regions in the image to be fused into the reference frame image, wherein, when the sub-region is located in the foreground area, the larger pixel value of the sub-region in the image to be fused and the reference frame image is used as the pixel value of the sub-region after fusion.

[0163] It should be noted that a sub-region in the image to be fused is generally an image block, and the sub-region has a corresponding image block in the reference frame image, and the positions, shapes, and sizes of the two image blocks are generally the same. 1 Extract sub-region A from the image and record it as image block A_I 1 , sub-region A in the reference frame image I 0 The corresponding image block is A_I 0 , A_I 1 in I 1 Position, shape, size and A_I 0 in I 0 The positions, shapes, and sizes are the same. Select A_I 0 with A_I 1The larger pixel value among them is used as the pixel value of sub-region A in the fused image. Specifically, let A_I 0 with A_I 1 The pixel values ​​are P(A_I 0 ) and P(A_I 1 ), if P(A_I 0 )≥P(A_I 1 ), then keep the reference frame image I 0 The pixel value of the sub-region A, if P(A_I 0 ) <P(A_I 1 ), then the image to be fused I 1 The pixel value of the sub-region A replaces the reference frame image I 0 to complete the fusion of sub-region A.

[0164] In one implementation, step S1140 may include the following steps:

[0165] Determine the pixel statistics of the sub-region in the image to be fused and the pixel statistics in the reference frame image;

[0166] The pixel value of the sub-region in the image corresponding to the larger pixel statistical value is used as the pixel value of the sub-region after fusion.

[0167] The pixel statistics may be a pixel average, a pixel maximum, a brightness average, etc., which is not limited in the present disclosure. It should be noted that when a sub-region is only one pixel point, the average value of the pixel values ​​of each channel of the pixel point can be counted, or the pixel value can be converted into a brightness value (or grayscale value) and the brightness value can be used as the pixel statistics. Thus, the relative size of the pixel values ​​of the same sub-region in different images (i.e., the reference frame image and the image to be fused) can be compared through the pixel statistics to select the pixel value of the sub-region in the image with the larger pixel value as the pixel value after fusion, and complete the fusion of the sub-region.

[0168] In one implementation, when the sub-regions in the image to be fused are fused to the reference frame image respectively, the following steps may also be performed:

[0169] When the sub-region is located in the background region of the image to be fused, the pixel weighted value of the sub-region in the image to be fused and the reference frame image is used as the pixel value of the sub-region after fusion.

[0170] For example, sub-region A in the reference frame image I 0 A_I 0 , in the image to be fused I 1 A_I 1 , the pixel values ​​are P(A_I 0 ) and P(A_I1 ), the pixel weight value can be P(A_I 0 ) and P(A_I 1 ) is the value after weighted averaging. The weights required for weighting can be determined based on experience or actual needs. For example, P(A_I 0 ) and P(A_I 1 ) is always 0.5, that is, P(A_I 0 ) and P(A_I 1 ) is taken as the pixel value of the fused sub-region A.

[0171] It should be supplemented that there may be one or more sub-regions located on the boundary between the foreground region and the background region. For example, part of sub-region A is located in the foreground region and the other part is located in the background region. Sub-region A can usually be regarded as a sub-region of the background region.

[0172] In one implementation, step S1110 may include the following steps:

[0173] Determine an initial reference frame image among multiple consecutive frame images, and sequentially determine each of the other frame images as an image to be fused;

[0174] After the sub-region of the image to be fused is fused to the reference frame image, the reference frame image is updated with the fused image.

[0175] For example, obtain k consecutive frames of image I 0 , I 1 , I 2 ,…,I k-1 ; Select the first frame I 0 is the initial reference frame image I ref , that is, let I ref =I 0 , and take each other frame image as the image to be fused in turn; first extract the sub-region from I1 and fuse them to the reference frame image I ref In the example, the fusion is equivalent to updating the reference frame image I ref ; and then from I 2 Extract sub-regions and fuse them into the reference frame image I ref To update the reference frame image I ref ; ...; Iterate the above process, each fusion is equivalent to updating the reference frame image I ref Until I complete k-1 The fusion of the final reference frame image I ref .

[0176] In one embodiment, after the sub-region of the last frame of the image to be fused is fused with the reference frame image, the target image is obtained. 1, I 2 , …, I k-1 Fused as the images to be fused, and finally the reference frame image I ref is the target image I for output out .

[0177] In one embodiment, during the process of successively fusing each other frame image as the image to be fused, when the sub-region is located in the background region, the weighted pixel values of the sub-region in the image to be fused and the reference frame image can be used as the pixel values of the fused sub-region based on the first weight of the reference frame image and the second weight of the image to be fused. Among them, the first weight is The second weight is i is the ordinal number of the current image to be fused. For example, in the above process of successively fusing I 1 , I 2 , …, I k-1 , record the current image to be fused as I i (0 < i < k), i represents the ordinal number of the current image to be fused, that is, which image to be fused the current image is; if i = 3, the first weight is 3 / 4 and the second weight is 1 / 4. And the current reference frame image is the result of fusing the initial reference frame image and two images to be fused, that is, the result of fusing three images, and the current image to be fused is one image. It can be seen that the first weight and the second weight are the weights of the number of fused images, and the obtained weighted pixel value is actually the pixel average value of the reference frame image and all images to be fused.

[0178] Based on Fig.11 's method, another technical solution for simulating the long exposure effect through multi-frame image fusion is provided, which can be applied to electronic devices such as smart phones, avoiding problems such as ghosting and overexposure caused by inappropriate exposure time, and being less affected by the jitter of the device body, thereby improving the image blurring problem; and being able to retain clear and continuous motion trajectories in the fused images, while reducing the influence of noise and improving the quality of image fusion.

[0179] In one embodiment, Fig.11 's image processing method can be specifically implemented as follows:

[0180] Among the continuous k-frame images I 0 , I 1 , I 2 , …, I k-1 , select the first frame I 0 as the initial reference frame image I ref , that is, let I ref = I 0 ; and I 1 , I 2 , …, Ik-1 All point to I ref Perform registration to obtain the registered image I 1 ', I 2 ', …, I k-1 ' and use them as the images to be fused in sequence;

[0181] Respectively determine the foreground regions in I 1 ', I 2 ', …, I k-1 ' to obtain the foreground mask images M 1 , M 2 , …, M k-1 ;

[0182] Regard each pixel point in each image to be fused as a sub-region. For example, in any i-th (0 < i < k) image to be fused I i ', the pixel point with coordinates (x, y) is denoted as I i '(x, y);

[0183] During fusion, input I 0 , I 1 ', I 2 ', …, I k-1 ', and the k - 1 foreground mask images M 1 , M 2 , …, M k-1 ;

[0184] Sequentially fuse I 1 ', I 2 ', …, I k-1 ' into I ref for updating I ref ; For any i-th image to be fused I i ', judge whether the pixel point is in the foreground region or the background region according to the foreground mask image M i . The pixel points in the foreground region are fused into I ref according to the maximum pixel value, and the pixel points in the background region are fused into I ref according to the pixel weighted value, which is expressed as follows:

[0185]

[0186] Until I 1 ', I 2 ', …, I k-1 ' are all fused, and output the final target image I out = I ref .

[0187] Fig.12The process architecture diagram of the above image processing method is shown, and the entire image processing process is divided into two parts:

[0188] Image segmentation and registration. Get k consecutive frames of original image I 0 , I 1 , I 2 ,…,I k-1 , k is a positive integer greater than 1; select a reference frame image, for example, the first frame I 0 As the initial reference frame image I ref , determine the other frame original images as the images to be fused; respectively ref Subtract and get the difference image D 1 , D 2 , …, D k-1 ; Binarize the difference image to obtain the foreground mask image M 1 , M 2 , …, M k-1 , and the background mask image RM 1 , R.M. 2 , …, RM k-1 , the foreground mask image and the background mask image are complementary, that is, the foreground mask image is the inverse mask image of the background mask image; using RM 1 to I ref with I 1 Extract the feature points of the background area and match them. Calculate I by matching point pairs ref with I 1 The homography matrix H between 1 , using RM 2 to I ref with I 2 Extract feature points from the background area for matching, and calculate I ref with I 2 The homography matrix H between 2 ...Thus, the homography matrix H of all images to be fused is obtained 1 , H 2 , …, H k-1 ; Use the homography matrix to transform I 1 , I 2 ,…,I k-1 To I 0 Registration, thus obtaining k consecutive frames of registered images I 0 , I 1 ',I 2 ', ..., I k-1 '.

[0189] Image fusion. Input I 0 , I 1 ',I 2', ..., I k-1 ', and M 1 , M 2 , …, M k-1 ; Let I ref =I 0 , i = 1, 2, ..., k-1, and continue to increase, execute the above formula (3) to iteratively update I ref ; After traversing all the images to be fused, output the target image I out =I ref .

[0190] Fig.13 These are continuous multi-frame images taken for night scenes of urban roads. Fig.14 To adopt Fig.12 The method flow is Fig.13 It can be seen that the image processing method of this exemplary embodiment can obtain a streamer image with clear images and continuous light tracks in night scene shooting with moving light sources, enrich the night scene shooting style and picture expression, and improve the user experience.

[0191] The exemplary embodiment of the present disclosure also provides an image processing device. Fig.15 As shown, the image processing device 1500 may include:

[0192] The image acquisition module 1510 is configured to acquire a plurality of consecutive frames of images;

[0193] The image segmentation module 1520 is configured to extract foreground area images of at least two frames of images;

[0194] A foreground fusion module 1530 is configured to fuse each foreground sub-region according to the maximum pixel value of each foreground sub-region in each foreground region image to form a foreground fused image, wherein the foreground sub-region is a sub-region of the foreground region image;

[0195] The target fusion module 1540 is configured to generate a target image based on the foreground fusion image and the background area image of at least one frame of image.

[0196] In one implementation, each pixel in the foreground area image is a foreground sub-area.

[0197] In one implementation, the foreground fusion module 1530 is configured to:

[0198] For each foreground sub-region, determining pixel statistics of the foreground sub-region in each foreground region image respectively;

[0199] Determine the foreground region image corresponding to the maximum pixel statistics value of the foreground sub-region as the target foreground region image corresponding to the foreground sub-region;

[0200] The pixel value of the foreground sub-region in the target foreground region image is used as the pixel value of the foreground sub-region in the foreground fused image.

[0201] In one implementation, the target fusion module 1540 is configured to:

[0202] Fusing background area images of at least two frames of images to obtain a background fused image;

[0203] A target image is generated based on the foreground fusion image and the background fusion image.

[0204] In one implementation, the target fusion module 1540 is configured to:

[0205] The pixel values ​​of each pixel in the background area in each background area image are averaged to obtain a background fused image.

[0206] In one implementation, the image acquisition module 1510 is configured to:

[0207] Determine exposure conditions based on ambient lighting parameters;

[0208] Acquire multiple frames of images captured continuously under the exposure conditions.

[0209] In one implementation, the image acquisition module 1510 is configured to:

[0210] Determining a corresponding first exposure duration according to an ambient light parameter;

[0211] Obtain the second exposure time corresponding to the minimum sensitivity;

[0212] The exposure condition is determined according to the maximum value of the first exposure time and the second exposure time, and the minimum sensitivity.

[0213] In one embodiment, the image segmentation module 1520 is configured to:

[0214] Subtracting at least two frames of images to be segmented in the continuous multiple frames of images from the reference frame images in the continuous multiple frames of images to obtain a difference image corresponding to each frame of the images to be segmented;

[0215] Generate a foreground mask image corresponding to each frame of the image to be segmented according to the comparison result of each pixel value in the difference image and the preset threshold;

[0216] The foreground mask image is used to extract the foreground area image from the image to be segmented.

[0217] In one implementation, the image processing apparatus 1500 may further include an image registration module configured to:

[0218] The at least two frames of images are registered according to the background area images of the at least two frames of images.

[0219] In one embodiment, the image segmentation module 1520 is configured to:

[0220] The foreground area image is extracted again from at least two registered image frames.

[0221] In one embodiment, the image registration module is configured to:

[0222] Extract feature points from the background area images of the two frames of images respectively;

[0223] Perform feature point matching on the background area images of the two frames of images to obtain matching point pairs;

[0224] Determine the homography matrix based on the matching point pairs;

[0225] One of the two images is registered to the other through the homography matrix.

[0226] In one embodiment, the image segmentation module 1520 is configured to:

[0227] When the background areas corresponding to the background area images do not completely overlap, the background areas corresponding to the background area images are intersected to obtain a common background area;

[0228] The portion of each background area image that is outside the common background area is deleted.

[0229] In one implementation, the at least two frames of images include a reference frame image.

[0230] The image registration module is configured as follows:

[0231] All images other than the reference frame image in at least two frames of images are registered with the reference frame image.

[0232] In one implementation, the reference frame image may be the first frame image among the above-mentioned continuous multiple frame images.

[0233] In one embodiment, the image segmentation module 1520 is configured to:

[0234] When the foreground areas corresponding to the foreground area images do not completely overlap, the foreground areas corresponding to the foreground area images are intersected to obtain a common foreground area;

[0235] The portion of each foreground area image that is outside the common foreground area is deleted.

[0236] In one embodiment, the image segmentation module 1520 is configured to:

[0237] Extract the foreground area image of each frame.

[0238] The exemplary embodiment of the present disclosure also provides another image processing device. Fig.16 As shown, the image processing device 1600 may include:

[0239] The image determination module 1610 is configured to determine a reference frame image from a plurality of consecutive frame images, and determine at least one frame image other than the reference frame image as an image to be fused;

[0240] An image segmentation module 1620 is configured to determine the foreground area of ​​the image to be fused as a foreground area;

[0241] The sub-region extraction module 1630 is configured to extract at least one sub-region from the image to be fused;

[0242] The sub-region fusion module 1640 is configured to fuse the sub-regions in the image to be fused into the reference frame image respectively, wherein, when the sub-region is located in the foreground area, the larger pixel value of the sub-region in the image to be fused and the reference frame image is used as the pixel value of the fused sub-region.

[0243] In one implementation, the sub-region fusion module 1640 is configured to:

[0244] Determine the pixel statistics of the sub-region in the image to be fused and the pixel statistics in the reference frame image;

[0245] The pixel value of the sub-region in the image corresponding to the larger pixel statistical value is used as the pixel value of the fused sub-region.

[0246] In one implementation, the sub-region fusion module 1640 is configured to:

[0247] In the process of fusing the sub-regions in the image to be fused into the reference frame image respectively, when the sub-region is located in the background area of ​​the image to be fused, the pixel weighted value of the sub-region in the image to be fused and the reference frame image is used as the pixel value of the fused sub-region.

[0248] In one implementation, the image determination module 1610 is configured to:

[0249] Determine an initial reference frame image among multiple consecutive frame images, and sequentially determine each of the other frame images as an image to be fused;

[0250] After the sub-region of the image to be fused is fused to the reference frame image, the reference frame image is updated with the fused image.

[0251] In one implementation, the sub-region fusion module 1640 is configured to:

[0252] In the process of fusing the sub-regions in the image to be fused into the reference frame image respectively, when the sub-region is located in the background region, based on the first weight of the reference frame image and the second weight of the image to be fused, the pixel weighted value of the sub-region in the image to be fused and the reference frame image is used as the pixel value of the fused sub-region;

[0253] The first weight is The second weight is i is the ordinal number of the current image to be fused.

[0254] In one implementation, the sub-region fusion module 1640 is configured to:

[0255] After the sub-region of the last frame of the image to be fused is fused with the reference frame image, the target image is obtained.

[0256] In one embodiment, the image segmentation module 1620 is configured to:

[0257] Subtract the image to be fused from the reference frame image to obtain a difference image corresponding to the image to be fused;

[0258] According to the comparison result of each pixel value in the difference image and the preset threshold, the foreground area of ​​the image to be fused is determined.

[0259] In one implementation, the sub-region extraction module 1630 is configured to:

[0260] The image to be fused is divided into multiple sub-regions, and each sub-region is extracted.

[0261] In one implementation, the sub-region extraction module 1630 is configured to:

[0262] Each pixel in the image to be fused is determined as a sub-region.

[0263] The details of each part of the above device have been described in detail in the method part implementation method, so they will not be repeated here.

[0264] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0265] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In one embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0266] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0267] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0268] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0269] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0270] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0271] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire multiple consecutive frames of images; Extracting foreground area images of at least two frames of images; fusing the foreground sub-regions according to the maximum pixel value of each foreground sub-region in each foreground region image to form a foreground fused image, wherein the foreground sub-region is a sub-region of the foreground region image; Generate a target image based on the foreground fused image and a background area image of at least one frame of image; The step of fusing the foreground sub-regions according to the maximum pixel value of each foreground sub-region in each foreground region image to form a foreground fused image includes: For each foreground sub-region, determining pixel statistics of the foreground sub-region in each of the foreground region images; Determine the foreground area image corresponding to the maximum pixel statistics value of the foreground sub-area as the target foreground area image corresponding to the foreground sub-area; The pixel value of the foreground sub-region in the target foreground region image is used as the pixel value of the foreground sub-region in the foreground fused image.

2. The method according to claim 1, characterized in that Each pixel point in the foreground area image is a foreground sub-area.

3. The method according to claim 1, characterized in that If the foreground sub-region is a pixel point, the pixel statistics of the foreground sub-region include an average value or a brightness value of pixel values ​​of each channel of the pixel point.

4. The method according to claim 1, characterized in that: The step of generating a target image based on the foreground fused image and a background area image of at least one frame of image comprises: Fusing background area images of at least two frames of images to obtain a background fused image; The target image is generated based on the foreground fused image and the background fused image.

5. The method according to claim 4, characterized in that The step of fusing the background area images of at least two frames of images to obtain a background fused image includes: The pixel values ​​of each pixel in the background area in each of the background area images are averaged to obtain the background fused image.

6. The method according to claim 1, characterized in that The step of acquiring a plurality of continuous image frames comprises: Determine exposure conditions based on ambient lighting parameters; A plurality of frames of images continuously shot under the exposure condition are acquired.

7. The method according to claim 6, characterized in that The step of determining the exposure condition according to the ambient light parameter comprises: Determining a corresponding first exposure duration according to an ambient light parameter; Obtain the second exposure time corresponding to the minimum sensitivity; The exposure condition is determined according to a maximum value between the first exposure time and the second exposure time, and the minimum sensitivity.

8. The method according to claim 1, characterized in that The step of extracting the foreground area images of at least two frames of images comprises: Subtracting at least two frames of the to-be-segmented images in the continuous multiple frames of images from the reference frame images in the continuous multiple frames of images to obtain a difference image corresponding to each frame of the to-be-segmented images; Generate a foreground mask image corresponding to each frame of the image to be segmented according to a comparison result between each pixel value in the difference image and a preset threshold; The foreground area image is extracted from the image to be segmented using the foreground mask image.

9. The method according to claim 1, characterized in that: After extracting the foreground area images of at least two frames of images, the method further includes: The at least two frames of images are registered according to the background area images of the at least two frames of images.

10. The method according to claim 9, characterized in that After registering the at least two frames of images, the method further includes: The foreground area image is extracted again from the at least two registered image frames.

11. The method according to claim 9, characterized in that The registering the at least two frames of images according to the background area images of the at least two frames of images comprises: Extract feature points from the background area images of the two frames of images respectively; Performing feature point matching on the background area images of the two frames of images to obtain matching point pairs; Determine a homography matrix according to the matching point pairs; One of the two image frames is registered to the other image frame by using the homography matrix.

12. The method according to claim 9, characterized in that Before registering the at least two frames of images according to the background area images of the at least two frames of images, the method further includes: When the background areas corresponding to the background area images do not completely overlap, taking the intersection of the background areas corresponding to the background area images to obtain a common background area; The portion of each background area image that is outside the common background area is deleted.

13. The method according to claim 9, characterized in that The at least two frames of images include a reference frame image; The registering the at least two frames of images comprises: All images in the at least two frames of images except the reference frame image are registered with the reference frame image.

14. The method according to claim 8 or 13, characterized in that: The reference frame image is the first frame image in the continuous multiple frame images.

15. The method according to claim 1, characterized in that Before fusing the foreground sub-regions according to the maximum pixel value of each foreground sub-region in each foreground region image to form a foreground fused image, the method further includes: When the foreground areas corresponding to the foreground area images do not completely overlap, taking the intersection of the foreground areas corresponding to the foreground area images to obtain a common foreground area; The portion of each foreground area image that is outside the common foreground area is deleted.

16. The method according to claim 1, characterized in that The step of extracting the foreground area images of at least two frames of images comprises: Extract the foreground area image of each frame.

17. An image processing method, characterized in that: include: Determine a reference frame image from a plurality of continuous frame images, and determine at least one frame image other than the reference frame image as an image to be fused; Determining the foreground area of ​​the image to be fused; Extracting at least one sub-region from the image to be fused; The sub-regions in the image to be fused are respectively fused to the reference frame image, wherein, when the sub-region is located in the foreground area, the maximum pixel value of the sub-region in the image to be fused and the reference frame image is used as the pixel value of the sub-region after fusion.

18. The method according to claim 17, characterized in that The taking the maximum pixel value of the sub-region in the image to be fused and the reference frame image as the pixel value of the sub-region after fusion includes: Determine a pixel statistic value of the sub-region in the image to be fused and a pixel statistic value in the reference frame image; The pixel value of the sub-region in the image corresponding to the maximum pixel statistical value is used as the pixel value of the sub-region after fusion.

19. The method according to claim 17, characterized in that When the sub-regions in the image to be fused are respectively fused to the reference frame image, the method further includes: When the sub-region is located in the background region of the image to be fused, the pixel weighted value of the sub-region in the image to be fused and the reference frame image is used as the pixel value of the sub-region after fusion.

20. The method according to claim 17, characterized in that The step of determining a reference frame image from the continuous multiple frame images, and determining at least one frame image other than the reference frame image as the image to be fused, comprises: Determine an initial reference frame image in the continuous multiple frame images, and sequentially determine each other frame image as the image to be fused; After the sub-region of the image to be fused is fused to the reference frame image, the reference frame image is updated with the fused image.

21. The method according to claim 20, characterized in that When the sub-regions in the image to be fused are respectively fused to the reference frame image, the method further includes: When the sub-region is located in the background region, based on the first weight of the reference frame image and the second weight of the image to be fused, taking the pixel weighted value of the sub-region in the image to be fused and the reference frame image as the pixel value of the sub-region after fusion; Among them, the first weight is The second weight is i is the ordinal number of the current image to be fused.

22. The method according to claim 20, characterized in that The method further comprises: After the sub-region of the last frame of the image to be fused is fused with the reference frame image, a target image is obtained.

23. The method according to claim 17, characterized in that The step of determining the foreground area of ​​the image to be fused includes: Subtracting the image to be fused from the reference frame image to obtain a difference image corresponding to the image to be fused; According to the comparison result of each pixel value in the difference image and a preset threshold, the foreground area of ​​the image to be fused is determined.

24. The method according to claim 17, characterized in that The step of extracting at least one sub-region from the image to be fused comprises: The image to be fused is divided into a plurality of sub-regions, and each sub-region is extracted.

25. The method according to claim 24, characterized in that The step of dividing the image to be fused into a plurality of sub-regions comprises: Each pixel in the image to be fused is determined as a sub-region.

26. An image processing device, characterized in that: include: An image acquisition module is configured to acquire multiple consecutive frames of images; An image segmentation module is configured to extract foreground area images of at least two frames of images; a foreground fusion module configured to fuse the foreground sub-regions according to the maximum pixel value of each foreground sub-region in each foreground region image to form a foreground fused image, wherein the foreground sub-region is a sub-region of the foreground region image; A target fusion module is configured to generate a target image based on the foreground fusion image and a background area image of at least one frame of image; Among them, the foreground fusion module is configured as follows: for each foreground sub-region, determining the pixel statistics of the foreground sub-region in each of the foreground region images respectively; determining the foreground region image corresponding to the maximum pixel statistics of the foreground sub-region as the target foreground region image corresponding to the foreground sub-region; and using the pixel value of the foreground sub-region in the target foreground region image as the pixel value of the foreground sub-region in the foreground fusion image.

27. An image processing device, characterized in that: include: An image determination module is configured to determine a reference frame image from a plurality of consecutive frame images, and determine at least one frame image other than the reference frame image as an image to be fused; An image segmentation module is configured to determine the foreground area of ​​the image to be fused as a foreground area; A sub-region extraction module, configured to extract at least one sub-region from the image to be fused; The sub-region fusion module is configured to fuse the sub-regions in the image to be fused into the reference frame image respectively, wherein, when the sub-region is located in the foreground area, the maximum pixel value of the sub-region in the image to be fused and the reference frame image is used as the pixel value of the sub-region after fusion.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 25 is implemented.

29. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 25 by executing the executable instructions.

Citation Information

Patent Citations

  • Infrared-image foreground detection method and infrared-image foreground detection equipment

    CN108133488A

  • Multi-frame image fusion method and device

    CN110189285A

  • Image processing method and device and electronic equipment

    CN111369469A

  • Content adaptive background-foreground segmentation for video coding

    WO2015175201A1

  • Method and device for image filtering

    WO2017101489A1