Foreground removal method, electronic device, computer storage medium and computer program product

Abstract

Embodiments of the present application disclose a foreground removal method, electronic device, computer storage medium, and computer program product. The electronic device iteratively processes multiple frames of images using multiple stages, gradually expanding from a local area to capture foreground changes in these images. This processing utilizes both short frames containing a small number of images and long frames containing a larger number of images, resulting in a more effective foreground removal method.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a foreground removal method, an electronic device, a computer storage medium, and a computer program product. Background Art

[0002] When shooting, users often want to capture clean, distraction-free footage. However, due to factors such as heavy traffic and complex scenery, the scene may contain a constant flow of cars and people, resulting in users only being able to capture images with people, cars, or other distracting objects in the foreground. To achieve the desired shot, users need to wait for the right moment, and not all shooting scenes offer the right timing. Summary of the Invention

[0003] The embodiments of the present application provide a foreground removal method, an electronic device, a computer storage medium, and a computer program product, which can capture foreground changes in an image by gradually expanding from a local area, thereby providing a good foreground removal effect.

[0004] In a first aspect, a foreground removal method is provided, which may include: obtaining a first frame sequence, the first frame sequence including multiple frames of images; iteratively processing the first frame sequence through n stages to obtain a second frame sequence, the second frame sequence including multiple frames of images, where n is an integer greater than or equal to 2; synthesizing the multiple frames of the second frame sequence into one frame of image; wherein, in the process of iteratively processing the first frame sequence in n stages: the input of the first stage is the first frame sequence, the input of each subsequent stage is the output of the adjacent previous stage, and the output of the nth stage is the second frame sequence; for each stage, the image input to the stage is divided into one or more segments, and for each segment, background enhancement processing is performed on each frame of the segment based on the segment, and multiple frames of images corresponding to the segment are output, wherein the image output corresponding to the segment contained in the stage is the output of the stage; the number of image frames of each segment in the subsequent stage is greater than the number of image frames of each segment in the adjacent previous stage.

[0005] Through the method of the first aspect, the electronic device uses multiple stages to iteratively process multiple frame images, gradually expanding from the local to capture the foreground changes in these images. In this processing process, both short frames containing a small number of frame images and long frames containing more frame images are used. Therefore, the method of the present application has a better foreground removal effect.

[0006] In combination with the first aspect, in some implementations, the number of image frames in each segment of the same stage is the same.

[0007] In combination with the previous embodiment, in some embodiments, the number of image frames of the segments in the first stage is a preset value, and the number of image frames of the segments in each subsequent stage is increased by the preset value over the number of image frames of the segments in the adjacent previous stage.

[0008] In some embodiments, during the n-stage iterative processing of the first frame sequence, images that are insufficient to be divided into a segment are discarded in each stage.

[0009] In conjunction with the first aspect, in some embodiments, during the n-stage iterative processing of the first frame sequence, background enhancement processing is performed on different image segments in the same stage in parallel, which can improve processing efficiency.

[0010] In combination with the first aspect, in some embodiments, for a segment, background enhancement processing is performed on each frame image in the segment based on the segment, and the step of outputting multiple frames of images corresponding to the segment may include: calculating the average frame of the multiple frames of images in the segment; calculating the difference frame between each frame image in the segment and the average frame; converting the multiple difference frames into grayscale images; performing threshold segmentation on each grayscale image to obtain a corresponding binary image; converting each binary image into a foreground image and a background image with color information; and performing weighted summation on each pair of foreground images and background images to obtain a corresponding fused image.

[0011] In combination with the previous embodiment, in some embodiments, when there are more foreground objects in the multiple frames of images included in the first frame sequence, the weight corresponding to the foreground image is greater when the foreground image and the background image are weighted summed.

[0012] In combination with the first aspect, in some embodiments, multiple frame images of the second frame sequence are synthesized into one frame image, specifically including: for each pixel point representing the same position, the pixel values ​​of the pixel points in the multiple frame images of the second frame sequence are traversed, and the pixel value with the largest number of occurrences is taken as the pixel value of the corresponding pixel point in the synthesized one frame image.

[0013] In conjunction with the first aspect, in some embodiments, synthesizing multiple frames of the second frame sequence into a single frame of image specifically includes: for each pixel representing the same location, first traversing the pixel values ​​of the pixel in the multiple frames of the second frame sequence, dividing the pixel values ​​of the pixel in the multiple frames into different pixel value ranges, and taking the pixel value with the most occurrences within the pixel value range with the most pixel values ​​as the pixel value of the corresponding pixel in the synthesized single frame of image. This approach can effectively remove foreground objects that appear briefly in the image, achieving excellent foreground removal results.

[0014] In conjunction with the first aspect, in some embodiments, during the n-stage iterative processing of the first frame sequence, the input and output of each stage are stored in a read-only memory (ROM). ROM is a non-volatile memory, meaning that stored data is not lost after a power outage. Storing data in ROM ensures that the method of the present embodiment is not affected by actions such as restoring the electronic device to factory settings, upgrading the system, or restarting the electronic device, and can be successfully executed.

[0015] In combination with the first aspect, in some embodiments, after acquiring the first frame sequence and before iteratively processing the first frame sequence in n stages, the method may further include: performing anti-shake processing on each frame image in the first frame sequence.

[0016] In combination with the first aspect or any of the above embodiments, in some embodiments, after obtaining the first frame sequence and before iteratively processing the first frame sequence through n stages, the method may also include: detecting that the camera application is closed; or detecting that the camera that captured the first frame sequence is not occupied.

[0017] In a second aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method provided in the first aspect or any one of the embodiments of the first aspect.

[0018] In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect or any embodiment of the first aspect is implemented.

[0019] In a fourth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, it implements the method provided in the first aspect or any embodiment of the first aspect.

[0020] In a fifth aspect, a chip system is provided, which includes a processing circuit and an interface circuit. The interface circuit is used to receive computer instructions and transmit them to the processing circuit. The processing circuit is used to run the computer instructions to implement the method provided in the first aspect or any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A The method flow of removing foreground from cloud is shown;

[0022] Figure 1B The flow of the method for removing foreground on the end side is shown;

[0023] Figure 2 A flowchart of a foreground removal method provided in an embodiment of the present application;

[0024] Figure 3 Schematic diagram of n-stage iterative processing of the first frame sequence;

[0025] Figure 4 The process of background enhancement processing of a clip is shown as an example;

[0026] Figure 5 A schematic diagram of synthesizing multiple frame images in a second frame sequence into one frame image;

[0027] Figure 6A and Figure 6B The foreground removal method provided by the embodiment of the present application is respectively compared with the foreground removal method before and after the ...;

[0028] Figure 7 A software structure diagram of an electronic device provided in an embodiment of the present application;

[0029] Figure 8 This is a hardware structure diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] An image may include a foreground and a background.

[0031] The foreground usually refers to the scenery in the front of the image. It is closer to the camera and can play a role in guiding the line of sight and transitioning between scenes in the picture, making it more conducive to highlighting the subject. The foreground can be people, animals, objects, etc.

[0032] The background refers to all areas of the image beyond the foreground. It's the scenery behind the camera, farther from the lens, and often occupies a larger area within the frame than the foreground. The background can be the sky, the ground, buildings, or other elements.

[0033] Figure 1A The method flow of removing foreground from the cloud is shown. Figure 1A As shown, after the user triggers the electronic device to take a photo, the electronic device stores the taken photos in the gallery. Then, the electronic device uploads the photos in the gallery that need foreground removal to the cloud server. Afterwards, the cloud server calls the foreground removal algorithm to remove the foreground in the photo, and then returns the processed photo to the gallery of the electronic device. This method has the following problems: 1. Most users pay attention to privacy, and uploading photos to the cloud will cause some users to give up using this function. 2. The training of the foreground removal algorithm is based on the training data set, which is difficult to cover all shooting scenes. Therefore, this method is not a universal foreground removal method applicable to all shooting scenes, and it may be less effective in some shooting scenes. 3. The cost of accessing the foreground removal algorithm on the cloud server side is too high, and the computing power resources required of the cloud server are also high.

[0034] Figure 1BFIG. 4 shows the process of the method for removing the foreground on the end side. Figure 1B As shown in the figure, after an electronic device takes a photo and generates an image, it first performs preprocessing, such as grayscale conversion and Gaussian blurring. It then detects the foreground through edge detection and threshold segmentation. Morphological operations, such as dilation and erosion, are then used to remove noise. Contour detection is then used to generate a foreground mask. Finally, an image inpainting algorithm is used to fill in the foreground area to make it consistent with the background. This method has very poor foreground removal performance and is only suitable for images with regular foreground patterns, such as raindrops.

[0035] The following embodiments of the present application provide a foreground removal method, which adopts reliable algorithm logic, has high-quality foreground removal effect, and has no special requirements for the shooting scene. In addition, the method is executed on the electronic device side, avoiding the high cost of foreground removal by the cloud server, which can effectively reduce the execution cost. This method does not require uploading user photos to the cloud, which can ensure data security and user privacy, and improve user experience and satisfaction. In addition, the method adopts end-side processing, which avoids the abnormal situations that may occur during cloud server processing, and can make the foreground removal effect of the image more controllable.

[0036] Figure 2 This is a flow chart of the foreground removal method provided in the embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0037] First, obtain the frame sequence.

[0038] S101 : The electronic device generates a first frame sequence including multiple frames of images in response to a shooting operation of a user.

[0039] The method of the embodiment of the present application does not use a single frame image as input to remove the foreground. Instead, it uses a first frame sequence containing multiple frames as input to remove the foreground. The multiple frames in the first frame sequence can be arranged in chronological order according to their generation time. With multiple frames as input, the foreground removal effect can be more accurate.

[0040] S101 may be performed in a scene with foreground interference such as traffic or pedestrian flow, so that part or all of the frames in the first frame sequence may contain foreground interference that needs to be removed.

[0041] The user's shooting operation may be a photo operation. The first frame sequence may be a plurality of frames of images output by a sensor in a currently activated camera of the electronic device over a period of time. The period of time includes the moment when the electronic device receives the shooting operation. The first frame sequence may include the image output by the sensor at the time the electronic device receives the shooting operation, the image output by the sensor before the shooting operation, or the image output by the sensor after the shooting operation.

[0042] The number of the multiple frame images included in the first frame sequence can be expressed as Y. The value of Y can be preset as needed and is not specifically limited here.

[0043] After the photosensitive element in the camera captures an image, the raw image data is usually first stored in a buffer of the electronic device. The buffer may be a part of the photosensitive element or an external memory area connected to the photosensitive element. In some embodiments of the present application, the electronic device can dump the first frame sequence into the read-only memory (ROM) of the electronic device. The format of the image dumped into the ROM can be RAW format or RGB format. After the data is written to the ROM, the data therein cannot be modified or deleted at will, can only be read, and can be quickly accessed and processed. Dumping the first frame sequence into the ROM can free up buffer space, and since the foreground removal method of the present application takes a certain amount of time, it can also facilitate the electronic device to read the first frame sequence from the ROM when executing the method of the present application. In other embodiments, the electronic device can also store the first frame sequence directly into the ROM without having to dump it from the buffer.

[0044] In some embodiments, the user may use a tripod or other stable device to shoot, which can avoid severe shaking and make the image generated by the electronic device more stable, thereby achieving better subsequent foreground removal effect.

[0045] In other embodiments, the user can enable the anti-shake mode of the electronic device. In this way, after the photosensitive element outputs the image, the electronic device can use an anti-shake algorithm to perform anti-shake processing on the image, eliminating the problem of abnormal foreground movement caused by shaking, and then dump the first frame sequence obtained after anti-shake to ROM. The anti-shake algorithm may include one or more processes such as cropping, scaling, pixel prediction, and pixel compensation. The embodiments of the present application do not specifically limit the anti-shake algorithm.

[0046] Second, iteratively remove the foreground.

[0047] In this process, the first frame sequence is iteratively processed through multiple stages, gradually expanding from a local area to capture the foreground changes in the first frame sequence, and generating a second frame sequence that contains a large amount of background information after removing the foreground.

[0048] S102 , iteratively process the first frame sequence in n stages to obtain a second frame sequence, where n is an integer greater than or equal to 2.

[0049] The input of the first stage is the first frame sequence generated in S101, the input of each subsequent stage is the output of the adjacent previous stage, and the output of the last stage is the second frame sequence.

[0050] In each stage, the input image in that stage is divided into one or more segments, and each segment in the same stage contains the same number of multi-frame images. In each stage, after the input image is divided into one or more segments in turn, the last image that is not large enough to be divided into a segment can also be discarded; then, background enhancement processing can be performed on each frame image in the segment based on each segment to obtain the multi-frame image corresponding to the segment, and the images output corresponding to the segments contained in this stage are used as inputs for the next adjacent stage. The images output by each stage can be stored in ROM to facilitate the electronic device to read in the next stage. The image output by the last stage can also be stored in ROM to facilitate the electronic device to execute the subsequent steps of S102.

[0051] When the first frame sequence and the images output in each of the above stages are stored in ROM, since ROM is a non-volatile memory, that is, the stored data will not be lost after power failure, therefore, even if the electronic device is restored to factory settings, the system is upgraded, or restarted during the process of executing S102-S103, the data stored in ROM can be restored, thereby not affecting the results of the embodiment of the present application. Of course, in some other embodiments, the above data can also be stored in a random access memory (RAM) or other types of storage areas.

[0052] The number of image frames contained in each segment in the first stage can be represented as k, where the value of k can be preset. The number of image frames per segment in each subsequent stage is increased by k over the number of image frames per segment in the previous stage. This gradually increases the number of image frames per segment in each stage, capturing foreground changes in the first frame sequence from a local perspective and gradually expanding the image. The resulting second frame sequence has a good foreground removal effect.

[0053] The total number of stages, n, can be determined based on the number of image frames, Y, in the first frame sequence and the number of image frames, k, contained in the segments of the first stage, where n is greater than or equal to 2. In some embodiments, based on the description in the previous two paragraphs, the Y image frames in the first frame sequence can be segmented according to the logic of the previous two paragraphs until only one segment can be segmented in the last stage, thereby determining the total number of stages, n. In other embodiments, the total number of stages, n, can also be less than the total number of stages determined in the previous embodiment, such that the last stage may be segmented into two or more segments.

[0054] Figure 3 Schematic diagram of n-stage iterative processing of the first frame sequence.

[0055] like Figure 3As shown in the figure, the camera sensor outputs Y frames of images, the first frame in a sequence. These Y frames are labeled in chronological order: image (frame) 1, image (frame) 2, image (frame) 3, and so on, image (frame) Y. The electronic device first applies an anti-shake algorithm to the Y frames output by the sensor and writes the processed Y frames to ROM.

[0056] Phase 1

[0057] In the first stage, the electronic device reads Y frames from the ROM in chronological order and, starting with the first frame, segments them. Each segment contains k frames, and this continues until there are not enough frames left to form a segment. Assuming the number of image frames contained in a segment in the first stage is k, then Y = a1 * k + b1, where a1 is the number of segments in the first stage, b1 is the number of frames discarded in the first stage, and b1 < k. If Y is an integer multiple of k, then b1 is 0.

[0058] For example, refer to Figure 3 , Phase 1 includes the following segments:

[0059] Stage 1_1: frame 1, frame 2, ... frame k.

[0060] Segment 2 (stage 1_2): image (frame) k+1, image (frame) k+2…image (frame) 2k.

[0061] Stage 1_3: Image (frame) 2k+1, image (frame) 2k+2…image (frame) 3k.

[0062] …

[0063] Fragment a1 (stage 1_a1): image (frame) (a1-1)k+1, image (frame) (a1-1)k+2…image (frame) a1*k.

[0064] The frames of images discarded in the first stage are: image (frame) a1*k+1, image (frame) a1*k+2…image (frame) Y.

[0065] After segmenting, the electronic device performs background enhancement processing on each segment in stage 1, generating multiple image frames corresponding to each segment. The final number of image frames output from stage 1 is a1*k. The electronic device can store these a1*k frames in ROM as input for stage 2. The specific implementation of background enhancement processing performed by the electronic device on each segment can be found in the following embodiments.

[0066] Phase 2

[0067] In the second stage, the electronic device reads the a1*k frames output from the ROM in chronological order of generation. Starting with frame 1, these images are divided into segments, each containing 2k frames, until the remaining frames are insufficient to form a segment. That is, a1*k = a2*2k + b2, where a2 is the number of segments in the second stage, b2 is the number of frames discarded in the second stage, and b2 < 2k. If a1*k is an integer multiple of 2k, b2 is 0.

[0068] For example, refer to Figure 3 , Phase 2 includes the following segments:

[0069] Segment 1 (stage 2_1): image (frame) 1, image (frame) 2…image (frame) 2k.

[0070] Stage 2_2: Image (frame) 2k+1, image (frame) 2k+2…image (frame) 4k.

[0071] Fragment 3 (stage 2_3): Image (frame) 4k+1, image (frame) 4k+2…image (frame) 6k.

[0072] …

[0073] Fragment a2 (stage 2_a2): image (frame) (a2-1)*2k+1, image (frame) (a2-1)*2k +2…image (frame) a2*2k.

[0074] The frames of images discarded in the second stage are: image (frame) a2*2k +1, image (frame) a2*2k +2…image (frame) a1*k.

[0075] After segmenting, the electronic device performs background enhancement processing on each segment in stage 2, generating multiple image frames corresponding to each segment. The final number of image frames output from stage 2 is a2*2k. The electronic device can store these a2*2k frames in ROM as input for stage 3. The specific implementation of background enhancement processing performed by the electronic device on each segment can be found in the following embodiments.

[0076] …

[0077] Stage n

[0078] In the nth stage, the electronic device reads the a output of the n-1th stage from the ROM according to the order of generation time. n-1 *(n-1)k frames of images, and divide these images into segments starting from the first frame, each segment contains nk frames of images, until the number of frames of the remaining images is not enough to form a segment, where a n-1 is the number of fragments in the n-1th stage. n-1 * (n-1) k = a n *nk+b n , where a n is the number of fragments in stage n, b n is the number of frames of discarded images in stage n, and b n <nk. If a n-1 * (n-1) k is an integer multiple of nk, then b n is 0. Figure 3 In, a n The value of is 1.

[0079] For example, refer to Figure 3 , the nth stage includes the following fragments:

[0080] Stage 1 (stage n_1): image (frame) 1, image (frame) n…image (frame) nk.

[0081] The frames of images discarded in the nth stage are: image (frame) nk +1, image (frame) nk +n…image (frame) a n-1 * (n-1)k.

[0082] After the segments are divided, the electronic device performs background enhancement processing on each segment in the nth stage, obtaining multiple frames of images corresponding to each segment. The number of image frames output in the final nth stage is nk. The specific implementation of the electronic device performing background enhancement processing on each segment can be referred to in the subsequent embodiments.

[0083] In some embodiments, in each stage, each fragment in the stage will be placed in the thread pool of the electronic device as a task for background enhancement processing. Different fragments in the same stage can be processed in parallel as different tasks, and the electronic device can use the thread pool mechanism to perform parallel background enhancement processing on multiple fragments in the same stage. In some embodiments, the thread pool can maintain the starting frame information (such as the starting frame identifier startIndex), the ending frame information (such as the ending frame identifier endIndex), and the identifier of the fragment (stage x_j) of each fragment in the same stage, where x represents the stage, j represents the jth fragment, and stage x_j represents the jth fragment in the xth stage. The thread pool can also register or define a callback function for each fragment in the same stage to notify the main thread of the image processing in the electronic device that the background enhancement processing of the fragment in the stage has been completed. Figure 3 The diagram exemplarily shows the information maintained by the thread pool and the defined callback function corresponding to the fragment in the nth stage (which includes one fragment).

[0084] At this point, the electronic device has completed n-stage iterative processing on Y frames of image data, resulting in nk frames of image data, which constitute the second frame sequence. Because some images are discarded during the n-stage iterative processing, the number of image frames in the second frame sequence, nk, is less than the number of image frames, Y, in the first frame sequence.

[0085] During the n-stage iterative processing of the Y frame image, the number of image segments in each stage gradually increases, allowing the foreground changes in the first frame sequence to be captured from a localized perspective. This multi-stage approach, with the number of image segments gradually increasing, utilizes both short frames containing a small number of frames and long frames containing a larger number of frames, resulting in a better foreground removal effect than processing the image in only one stage.

[0086] For example, if only one stage is set and the segments are divided according to short frames in this stage, then in a scene where a foreground object appears for a short time, some segments may not be able to capture the changes of the foreground object, and eventually the foreground object will be regarded as the background, resulting in poor foreground removal effect.

[0087] For another example, if only one stage is set and the segments are divided according to long frames in this stage, assuming that a cat constantly appears and disappears in the shooting scene with unchanged position and posture, and its appearance time \(t>T / 2\), where \(T\) is the duration of the first frame sequence, then if only long frames are used, this cat will be recognized as the background, which does not conform to the actual situation. If the \(n\)-stage iterative processing algorithm that gradually expands from short frames to long frames in this application is used, it is very likely that during the process of intercepting short frames, it will be found that in some segment \(p\), the appearance time of the cat \(t_p < T_p / 2\), where \(T_p\) is the duration of segment \(p\), so the cat will be recognized as the foreground. And during the continuous calculation process, due to the cat's certain posture changes, the information that "the cat is the foreground" in this segment will be continuously amplified, and finally the cat in the entire first frame sequence will be recognized as the foreground, thus achieving the removal effect. In other words, the judgment of the foreground is relatively loose. Whether an object is found to be the foreground in short frames or long frames, using the stage iterative foreground removal in this application can remove it; while the judgment of the background is relatively strict. An object will be finally determined as the background only if it remains unchanged (position, posture, speed, etc.) in both short frames and long frames.

[0088] The following introduces how to perform background enhancement processing on the segments in each stage. In the embodiments of this application, the logic of performing background enhancement processing on each segment in each stage is the same.

[0089] Background enhancement processing

[0090] In the embodiments of this application, the principle of performing background enhancement processing on a segment is to perform background enhancement processing on each frame image in the segment according to multiple frame images in the segment, and output images with the same number of frames as the images in the segment. Since background enhancement processing is performed on each frame image according to multiple frame images instead of directly performing background enhancement processing on this frame image, more image information can be referred to during background enhancement processing, and the foreground and background in the frame image can be better distinguished.

[0091] The following gives a specific example of an electronic device performing background enhancement processing on a segment. The segment in this example contains \(m\) frame images.

[0092] Figure 4 Exemplarily shows the processing flow in this example. As Figure 4 shown, this process may include the following steps:

[0093] 1. Calculate the average frame of the \(m\) frame images in the segment.

[0094] First, the electronic device reads the \(m\) frame images and ensures that these \(m\) frame images have the same size. If the image sizes are different, the \(m\) frame images need to be adjusted to the same size first.

[0095] Then, the electronic device can calculate the average frame of the m frames by accumulating the values ​​frame by frame. For example, the electronic device can use Formula 1 to calculate the average value of the m frames.

[0096] Formula 1

[0097] in, Indicates the average frame of m frames, startIndex is the start frame identifier in the current segment, and endIndex is the end frame identifier in the current segment. Indicates the i-th frame image in the segment. Figure 4 The starting frame of the current segment is shown as , the ending frame in the current segment is shown as .

[0098] In an embodiment of the present application, the electronic device can calculate the average value for each pixel position, and after traversing all pixel positions, the average frame of the m frames of images in the current segment can be obtained.

[0099] 2. Calculate the difference between each frame in the m-frame image and the average frame.

[0100] Exemplarily, the electronic device may use Formula 2 to calculate the difference frame between each frame of the m frames and the average frame.

[0101] Formula 2

[0102] in, Indicates the difference frame between the i-th frame image and the average frame in the current segment. In the embodiment of the present application, the electronic device can calculate the difference value for each pixel position, and after traversing all pixel positions, the difference frame between the i-th frame image and the average frame can be obtained.

[0103] 3. Convert the difference frames between each frame in the m frames and the average frame into grayscale images.

[0104] This step is equivalent to grayscale conversion of the output of step 2, converting the difference frame from RGB color space to grayscale color space. Grayscale images only contain brightness information, not color information.

[0105] Exemplarily, the electronic device may use Formula 3 to convert the difference frame into a grayscale image.

[0106] Formula 3

[0107] in, Represents the grayscale image converted from the difference frame between the i-th frame image and the average frame in the current segment, R is the red value in the difference frame, G is the green value in the difference frame, and B is the blue value in the difference frame.

[0108] 4. Perform threshold segmentation on each grayscale image to obtain the corresponding binary image, and then convert the binary image into foreground image and background image in RGB format.

[0109] For example, the electronic device may use Formula 4 to perform threshold segmentation on each grayscale image to obtain a corresponding binary image.

[0110] Formula 4

[0111] in, represents a binary image, and threshold represents a threshold. The threshold can be set as needed and is not limited here. In an embodiment of the present application, the electronic device can calculate the corresponding binary value for each pixel position of the i-th frame grayscale image, and after traversing all pixel positions, the i-th frame binary image can be obtained.

[0112] Then, the electronic device can convert the binary image into a foreground image and a background image with color information. Specifically, for each frame of the binary image, the pixel point with a pixel value of 255 is regarded as a pixel in the foreground image, and the pixel point with a pixel value of 0 is regarded as a pixel in the background image. Thus, the corresponding pixels in the foreground image and the corresponding pixels in the background image can be known from the binary image. Find the color values ​​(i.e. RGB values) of the above pixels, fill these values ​​into the foreground image and background image, and you will get the foreground image and background image with color information.

[0113] In step 4, m frames of binary images can be obtained, and each frame of the binary image can be converted into a corresponding frame of foreground image and a frame of background image with color information.

[0114] 5. Perform weighted summation on each pair of foreground and background images to obtain the corresponding fused image.

[0115] For example, the electronic device may use Formula 5 to perform weighted summation on a pair of foreground images and background images to obtain a corresponding fused image.

[0116] Formula 5

[0117] in, represents the fused image of the i-th frame, represents the foreground image converted from the binary image of frame i in step 4, represents the background image converted from the binary image of frame i in step 4, and α is the weighting coefficient.

[0118] α can be preset, and its specific value is not limited herein. For example, it can be 0.1 or another value. In some embodiments, α can also be adaptively adjusted based on the current shooting scene. For example, the value of α increases when there are more foreground objects (e.g., more people). The electronic device can analyze the number of foreground objects based on the image output by the sensor to determine the value of α.

[0119] This step can obtain m frames of fused images.

[0120] pass Figure 4 In the processing flow shown, the m-frame fused image obtained by the electronic device is the result of removing the foreground and enhancing the background of each frame image in the current segment.

[0121] Figure 4 This is merely an example. The embodiments of the present application may also employ other methods to perform background enhancement processing on multiple frames within each segment, as long as the effect of performing background enhancement processing on each frame within the multiple frames within the reference segment is achieved. The embodiments of the present application do not impose any specific restrictions on the method for performing background enhancement processing on each segment.

[0122] Finally, the multiple frame images of the second frame sequence are synthesized into one frame image.

[0123] S103 , the electronic device combines the multiple frame images of the second frame sequence into one frame image.

[0124] There are two ways to combine multiple frames of images in the second frame sequence into one frame of image:

[0125] Method 1 iterates over the pixel values ​​of the same pixel in multiple frames and takes the pixel value with the highest number of occurrences as the pixel value of the corresponding pixel in the final synthesized frame. After performing the above steps for each pixel representing the same position in the multiple frames of the second frame sequence, the pixel value of each pixel in the final synthesized frame can be obtained, thus obtaining the final synthesized frame.

[0126] Method 2 first traverses the pixel values ​​of the same pixel point in multiple frames of images, divides these pixel values ​​into different pixel value ranges (also called sets), and then takes the pixel value with the most occurrences within the pixel value range with the most pixel values ​​as the pixel value of the corresponding pixel point in the final synthesized frame of image. The sizes of the various pixel value ranges may be the same or different. In some embodiments, a k-nearest neighbor (KNN) algorithm can be used to classify the multiple pixel values, thereby obtaining multiple different pixel value ranges. After performing the above steps for each pixel point in the multiple frames of the second frame sequence, the pixel value of each pixel point in the final synthesized frame of image can be obtained, thus obtaining the final synthesized frame of image.

[0127] Figure 5 This is a schematic diagram of method 2. Figure 5 As shown, for the first pixel (pixel 1), the pixel value of the pixel in the multi-frame image is divided into different pixel value ranges, and then the pixel value range with the most pixel values ​​(such as Figure 5 The pixel value with the largest number of occurrences is taken as the pixel value of the first pixel point. The processing of the remaining pixels is similar.

[0128] Using method 2, you can effectively remove foreground objects that appear briefly in the image, and achieve excellent foreground removal results. Compared to method 1, method 2 can avoid the ghosting caused by foreground objects that are temporarily motionless. For example Figure 5 The last two pixel value ranges of the first pixel in the image are as follows: the pixel values ​​in these two pixel value ranges may represent a foreground object that appears briefly, and the foreground object can be effectively removed by using method 2.

[0129] The format of the frame of image synthesized by the electronic device in S103 may be a JPEG (joint photographic experts group) format or other formats, which is not limited in this embodiment of the present application.

[0130] At this point, the electronic device has completed the foreground removal during the shooting process. The frame of image outputted in S103 is the result of the shooting by the electronic device, which can be displayed on a display screen for the user to view.

[0131] Figure 6A and Figure 6B The figures show before and after comparisons of the foreground removal method provided in the embodiments of the present application.

[0132] refer to Figure 6A , Figure 6A The left side is an image that does not use the foreground removal method of the embodiment of the present application, which can be regarded as an original image. The original image can be an image in the first frame sequence in S101. The original image contains foreground interference, such as a vehicle on the middle road. Figure 6A The right side is an image after using the foreground removal method of an embodiment of the present application. Most of the foreground interference is removed from the image, and the vehicles on the road in the middle of the image are removed.

[0133] refer to Figure 6B , Figure 6B The left side is an image that does not use the foreground removal method of the embodiment of the present application, which can be regarded as an original image. The original image can be an image in the first frame sequence in S101. The original image contains foreground interference, such as a person. Figure 6BThe right side is an image after using the foreground removal method of an embodiment of the present application. Most of the foreground interference is removed from the image, and the person in the image is removed.

[0134] In the embodiment of the present application, S102-S103 may be performed in the following situations:

[0135] 1. After the electronic device obtains the first frame sequence in S101, it can execute S102-S103, giving the user real-time feedback so that the user can immediately see the foreground removal effect.

[0136] 2. After the electronic device obtains the first frame sequence in S101, it first stores the first frame sequence and then executes S102-S103 when the computing power is idle (for example, after the camera application is closed, or when the camera in the electronic device that shoots the first frame sequence is not occupied). This does not occupy the computing power of the electronic device during shooting, and can provide a smoother shooting experience.

[0137] 3. The user triggers the electronic device to execute S102-S103. For example, after the electronic device obtains the first frame sequence in S101, it first stores the first frame sequence and displays the original image to the user. Then, after receiving the user input to remove the foreground, it executes S102-S103 again. In this way, the foreground of the image can be removed according to the actual needs of the user.

[0138] 4. After the electronic device recognizes that the captured image has a relatively cluttered or large foreground, it can automatically execute S102-S103, so that the foreground can be removed for the required image.

[0139] Moreover, the foreground removal method provided in the embodiment of the present application can be applied not only to photo shooting scenes, but also to video recording scenes. For video recording scenes, the method provided in the present application can be used to remove the foreground of multiple first frame sequences to obtain multiple frames of video.

[0140] Other implementations of the present invention

[0141] When the first frame sequence is iteratively processed in n stages in S102, when the input multi-frame image is segmented in each stage, it is not necessary to start from the first frame of the multi-frame image and segment from the front to the back, thereby potentially discarding the last few frames. In other embodiments, the multi-frame image can be segmented from the last frame of the multi-frame image and segmented from the back to the front, and the first few frames can be discarded. In other embodiments, the multi-frame images in each segment may not be continuous frames, but may skip frames, and frames may also be skipped between segments.

[0142] When the first frame sequence is iteratively processed in n stages in S102, the number of image frames of the fragments in each subsequent stage is not limited to being increased by k above the number of image frames of the fragments in the adjacent previous stage. In other embodiments, as long as the number of image frames of the fragments in the subsequent stage is greater than the number of image frames of the fragments in the adjacent previous stage, it is sufficient. In this way, the foreground changes in the first frame sequence can be captured by gradually expanding from a local area.

[0143] When performing n-stage iterative processing on the first frame sequence in S102, images that are insufficient to form a segment in each stage may not be discarded, or images that are insufficient to form a segment may be treated as separate segments. In this way, the number of image frames in the second frame sequence is the same as the number of image frames in the first frame sequence.

[0144] When performing n-stage iterative processing on the first frame sequence in S102 , the number of frames included in each segment in each stage may also be different.

[0145] Other application scenarios of the embodiments of the present application

[0146] In addition to being applied to scenes where the foreground contains interferences such as traffic or pedestrians, the method of the embodiment of the present application can also be applied to scenes where rain or fog is captured, and can be used to remove the rain or fog in the foreground.

[0147] The logic of the embodiment of the present application can also be applied to the cutout function.

[0148] For example, Figure 4 In the background enhancement process performed on the fragment shown in FIG, after obtaining m frames of colored foreground images in step 4, the foreground images can be Figure 2 In S103, the m frames of colored foreground images are synthesized into one frame of image, which is the result output after the cutout.

[0149] Or, for example, Figure 4 In the background enhancement process performed on the fragment shown in FIG, after obtaining m frames of colored background images in step 4, the background images can be Figure 2 In S103, the background images of the m frames are synthesized into a single frame image, i.e., the background image. The original image and the frame image are then subtracted, and the subtracted image is the result of the cutout output. Because the n-stage iterative processing method provided in the embodiment of the present application is more effective for identifying the background in a frame sequence, the method of the embodiment of the present application is first used to generate a background image. Then, the background image is subtracted from the original image (or the original frame sequence, i.e., the first frame sequence) to obtain a foreground image (or a foreground frame sequence). The foreground frame sequence can also be synthesized into a single frame image.

[0150] Figure 7 A software structure diagram of the electronic device 100 provided in an embodiment of the present application.

[0151] The electronic device 100 may be used to execute the foreground removal method described above.

[0152] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, a mobile operating system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0153] Figure 7 It is a software structure diagram of the electronic device 100 according to an embodiment of the present application.

[0154] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, a mobile operating system is divided into four layers: application layer, application framework layer / core services layer, system libraries and runtime layer, and kernel layer.

[0155] The application layer can include a series of application packages.

[0156] like Figure 7 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0157] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0158] like Figure 7 As shown, the application framework layer may include a foreground removal module, an image synthesis module, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0159] The foreground removal module can be used to perform the above Figure 2 The foreground removal module may be integrated with a background enhancement processing algorithm, which may be referenced in step S102 of the method shown. Figure 4 processing flow.

[0160] The image synthesis module can be used to perform the above Figure 2 Step S103 in the method shown.

[0161] Of course, in some other implementations, the foreground removal module and the image synthesis module may also be located in other layers of the software framework, which is not limited in the embodiment of the present application.

[0162] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0163] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0164] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0165] The phone manager is used to provide communication functions for electronic devices, such as call status management (including answering, hanging up, etc.).

[0166] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0167] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0168] The runtime can refer to all code libraries, frameworks, and other components required for a program to run. For example, for the C language, the runtime includes a series of function libraries required for C programs to run. For the Java language, in addition to the core libraries, the runtime also includes the virtual machine required for Java programs to run. These core libraries include the functional functions required by the Java language.

[0169] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0170] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0171] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0172] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0173] A 2D graphics engine is a drawing engine for 2D drawings.

[0174] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0175] Figure 8 This is a hardware structure diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 is used to execute the foreground removal method provided in the above method embodiment.

[0176] Electronic device 100 may include a processor 101, memory 102, a wireless communication module 103, a mobile communication module 104, an antenna 103A, an antenna 104A, a power switch 105, a sensor module 106, a focus motor 107, a camera 108, a display screen 109, and the like. Sensor module 106 may include a gyroscope sensor 106A, an acceleration sensor 106B, an ambient light sensor 106C, an image sensor 106D, a distance sensor 106E, and the like. Wireless communication module 103 may include a WLAN communication module, a Bluetooth communication module, and the like. These multiple components may transmit data via a bus.

[0177] The processor 101 may include one or more processing units. For example, the processor 101 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0178] The memory 102 can be used to store computer executable program code, which can include instructions. The processor 101 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 102. The memory 102 can include a program storage area and a data storage area. In a specific implementation, the memory 102 can include a high-speed random access memory and can also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0179] The wireless communication function of the electronic device 100 can be implemented through the antenna 103A, the antenna 104A, the mobile communication module 104, the wireless communication module 103, the modem processor and the baseband processor.

[0180] Antenna 103A and antenna 104A can be used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0181] The mobile communication module 104 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for the electronic device 100. The mobile communication module 104 can include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), and the like. The mobile communication module 104 can receive electromagnetic waves through the antenna 104A, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 104 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves for radiation via the antenna 104A.

[0182] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium- or high-frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs audio signals through an audio device or displays images or videos on the display screen 109.

[0183] The wireless communication module 103 can provide wireless communication solutions including wireless local area networks (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied to the electronic device 100. The wireless communication module 103 can be one or more devices integrating at least one communication processing module. The wireless communication module 103 receives electromagnetic waves via the antenna 103A, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 101. The wireless communication module 103 can also receive the signal to be sent from the processor 101, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 103A.

[0184] The power switch 105 may be used to control the supply of power to the electronic device 100 .

[0185] The gyroscope sensor 106A can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 106A. The gyroscope sensor 106A can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 106A detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 106A can also be used for navigation and somatosensory game scenes.

[0186] Accelerometer 106B can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the electronic device's posture. For example, accelerometer 106B can be used in applications such as landscape and portrait screen switching and pedometers.

[0187] The ambient light sensor 106C is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 109 based on the sensed ambient light brightness. The ambient light sensor 106C can also be used to automatically adjust the white balance when taking pictures.

[0188] Image sensor 106D, also known as a photosensitive element, utilizes the photoelectric conversion function of a photoelectric device to convert the light image on the photosensitive surface into an electrical signal proportional to the light image. The image sensor can be a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0189] The distance sensor 106E can be used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some shooting scenarios, the electronic device 100 can use the distance sensor 106E to measure distance to achieve fast focusing.

[0190] The focus motor 107 can be used for fast focusing. The electronic device 100 can control the movement of the lens through the focus motor 107 to achieve automatic focusing.

[0191] The electronic device 100 can implement a shooting function through an ISP, a camera 108, a video codec, a GPU, a display screen 109, and an application processor.

[0192] The ISP processes data fed back by the camera 108. For example, when taking a photo, the shutter is opened, and light passes through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization for image noise and brightness. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within the camera 108.

[0193] The camera 108 can be used to capture still images or videos. An object's optical image is projected onto the image sensor through the lens. The image sensor converts the optical signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB or YUV. In some embodiments, the electronic device 100 may include one or N cameras 108, where N is a positive integer greater than one.

[0194] Video codecs are used to compress or decompress digital images. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can open or save pictures or videos in multiple encoding formats.

[0195] Electronic device 100 can implement display functions using a GPU, display screen 109, and an application processor. A GPU is a microprocessor for image processing that connects display screen 109 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 101 may include one or more GPUs that execute program instructions to generate or modify display information.

[0196] Display screen 109 is used to display images, videos, etc. Display screen 109 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 109, where N is a positive integer greater than 1.

[0197] In the embodiment of the present application, the memory 102 stores a computer program for implementing the foreground removal method provided in the embodiment of the present application, and the processor 101 can be used to execute the computer program to implement the method provided in the embodiment of the present application on an electronic device. Specifically, the GPU in the processor 101 can be used to execute Figure 2 S102-S103 in the method shown. The operations performed by each component in the electronic device 100 can be specifically referred to the relevant description of the above method embodiment, which will not be elaborated here.

[0198] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0199] It should be understood that each step in the above method embodiment can be completed by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0200] The present application also provides an electronic device, which may include a memory and a processor. The memory may be used to store a computer program, and the processor may be used to call the computer program in the memory to enable the electronic device to execute the method executed on the electronic device side in any of the above embodiments.

[0201] The present application also provides a chip system, which includes a processing circuit and an interface circuit. The interface circuit is used to receive computer instructions and transmit them to the processing circuit. The processing circuit is used to run the computer instructions to implement the method executed by the electronic device in any of the above embodiments.

[0202] The present application also provides a chip system, which includes at least one processor for implementing the method executed by the electronic device in any of the above embodiments. In one possible design, the chip system also includes a memory for storing program instructions and data, and the memory is located within or outside the processor.

[0203] The chip system can be composed of chips, or can include chips and other discrete devices.

[0204] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0205] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.

[0206] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0207] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the electronic device in any of the above embodiments.

[0208] The present application also provides a computer-readable storage medium storing a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0209] The various implementation modes of this application can be combined arbitrarily to achieve different technical effects.

[0210] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product comprises one or more computer instructions. When loaded and executed on a computer, the computer program instructions fully or partially perform the processes or functions described herein. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (SSD)).

[0211] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0212] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0213] The terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0214] In short, the above description is only an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of this application should be included in the scope of protection of this application.

Claims

1. A foreground removal method, characterized in that: The method comprises: Acquire a first frame sequence, where the first frame sequence includes multiple frames of images; Iteratively processing the first frame sequence in n stages to obtain a second frame sequence, where the second frame sequence includes a plurality of frames of images, and n is an integer greater than or equal to 2; For each pixel point representing the same position, pixel values ​​of the pixel point in multiple frames of the second frame sequence are traversed, and the pixel value with the largest number of occurrences is taken as the pixel value of the corresponding pixel point in the synthesized frame image. Alternatively, for each pixel point representing the same position, pixel values ​​of the pixel point in multiple frames of the second frame sequence are first traversed, the pixel values ​​of the pixel point in the multiple frames of the second frame sequence are divided into different pixel value ranges, and the pixel value with the largest number of occurrences within the pixel value range with the largest number of pixel values ​​is taken as the pixel value of the corresponding pixel point in the synthesized frame image, and the synthesized frame image is output; Wherein, during the n-stage iterative processing of the first frame sequence: The input of the first stage is the first frame sequence, the input of each subsequent stage is the output of the adjacent previous stage, and the output of the nth stage is the second frame sequence; For each stage, the input image of the stage is divided into one or more segments, and for each segment, background enhancement processing is performed on each frame image in the segment based on the segment, and multiple frames of images corresponding to the segment are output, wherein the images output corresponding to the segments included in the stage are the outputs of the stage; The number of image frames of each segment in the latter stage is greater than the number of image frames of each segment in the adjacent previous stage.

2. The method according to claim 1, characterized in that The number of image frames in each segment of the same stage is the same.

3. The method according to claim 2, characterized in that The number of image frames of the segments in the first stage is a preset value, and the number of image frames of the segments in each subsequent stage is increased by the preset value over the number of image frames of the segments in the adjacent previous stage.

4. The method according to claim 2, characterized in that During the n-stage iterative processing of the first frame sequence, images that are insufficient to be divided into a segment are discarded in each stage.

5. The method according to claim 1, wherein During the n-stage iterative processing of the first frame sequence, background enhancement processing is performed on images of different segments in the same stage in parallel.

6. The method according to claim 1, wherein For each segment, background enhancement processing is performed on each frame image in the segment based on the segment, and the steps of outputting multiple frames of images corresponding to the segment include: For each segment, calculating an average frame of multiple frames of images in the segment; Calculating the difference frame between each frame image in the segment and the average frame; Converting the plurality of difference frames into grayscale images; Performing threshold segmentation on each of the grayscale images to obtain a corresponding binary image; Based on each of the binary images, respectively obtain a foreground image and a background image with color information corresponding to each frame of the image; A weighted sum is performed on each pair of foreground image and background image to obtain a fused image corresponding to each frame of image.

7. The method according to claim 6, characterized in that When there are more foreground objects in the multiple frames of images included in the first frame sequence, the weight corresponding to the foreground image is greater when weighted summation is performed on the foreground image and the background image.

8. The method according to claim 1, characterized in that During the n-stage iterative processing of the first frame sequence, the input and output of each stage are stored in a read-only memory ROM.

9. The method according to claim 1, characterized in that After acquiring the first frame sequence and before iteratively processing the first frame sequence through n stages, the method further includes: Perform anti-shake processing on each frame image in the first frame sequence.

10. The method according to any one of claims 1 to 9, characterized in that After acquiring the first frame sequence and before iteratively processing the first frame sequence through n stages, the method further includes: Detect that the camera app is closed; or, It is detected that the camera shooting the first frame sequence is not occupied.

11. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A chip system, characterized in that: The chip system includes a processing circuit and an interface circuit, wherein the interface circuit is used to receive computer instructions and transmit them to the processing circuit, and the processing circuit is used to execute the computer instructions to implement the method according to any one of claims 1 to 10.

Images