Foreground removal method, electronic equipment, computer storage medium and computer program product

Abstract

The embodiment of the invention discloses a foreground removal method, electronic equipment, a computer storage medium and a computer program product. The electronic equipment adopts a multi-stage iteration mode to process multiple frames of images, and foreground changes in the images are captured from local gradual enlargement. In the processing process, short frames containing a small number of frame images and long frames containing more frame images are used, so that the method has a better foreground removal effect.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a foreground removal method, an electronic device, a computer storage medium and a computer program product. Background Art

[0002] When shooting, users usually want to record a clean picture without interference, but due to factors such as excessive flow of people and complex scenery, there may be a constant flow of cars and people in the shooting scene, resulting in users only being able to shoot pictures with people, cars or other interferences in the foreground. If users want to shoot the desired effect, they need to wait for the right time, and not all shooting scenes can wait for the right time. Summary of the invention

[0003] The embodiments of the present application provide a foreground removal method, an electronic device, a computer storage medium, and a computer program product, which can capture foreground changes in an image by gradually expanding from a local area, and provide a good foreground removal effect.

[0004] In a first aspect, a foreground removal method is provided, which may include: acquiring a first frame sequence, the first frame sequence including multiple frames of images; iteratively processing the first frame sequence through n stages to obtain a second frame sequence, the second frame sequence including multiple frames of images, where n is an integer greater than or equal to 2; synthesizing the multiple frames of images in the second frame sequence into one frame of image; wherein, in the process of iteratively processing the first frame sequence in n stages: the input of the first stage is the first frame sequence, the input of each subsequent stage is the output of the adjacent previous stage, and the output of the nth stage is the second frame sequence; for each stage, the image input to the stage is divided into one or more fragments, and for each fragment, background enhancement processing is performed on each frame of the image in the fragment based on the fragment, and multiple frames of images corresponding to the fragment are output, wherein the image output corresponding to the fragment contained in the stage is the output of the stage; the number of image frames of each fragment in the subsequent stage is greater than the number of image frames of each fragment in the adjacent previous stage.

[0005] Through the method of the first aspect, the electronic device uses multiple stages to iteratively process multiple frame images, gradually expanding from the local to capture the foreground changes in these images. In this processing process, both short frames containing a small number of frame images and long frames containing more frame images are used. Therefore, the method of the present application has a better foreground removal effect.

[0006] In combination with the first aspect, in some implementations, the number of image frames in each segment of the same stage is the same.

[0007] In combination with the previous embodiment, in some embodiments, the number of image frames of the fragments in the first stage is a preset value, and the number of image frames of the fragments in each subsequent stage is increased by the preset value over the number of image frames of the fragments in the adjacent previous stage.

[0008] In some implementations, during the n-stage iterative processing of the first frame sequence, images that are not large enough to be divided into a segment are discarded in each stage.

[0009] In conjunction with the first aspect, in some implementations, during the n-stage iterative processing of the first frame sequence, background enhancement processing is performed on different fragments of images in the same stage in parallel, which can improve processing efficiency.

[0010] In combination with the first aspect, in some embodiments, for a segment, background enhancement processing is performed on each frame image in the segment based on the segment, and the step of outputting multiple frames of images corresponding to the segment may include: calculating the average frame of the multiple frames of images in the segment; calculating the difference frame between each frame image in the segment and the average frame; converting the multiple difference frames into grayscale images; performing threshold segmentation on each grayscale image to obtain a corresponding binary image; converting each binary image into a foreground image and a background image with color information; and performing weighted summation on each pair of foreground images and background images to obtain a corresponding fused image.

[0011] In combination with the previous embodiment, in some embodiments, when there are more foreground objects in the multiple frame images included in the first frame sequence, when the foreground image and the background image are weighted summed, the weight corresponding to the foreground image is larger.

[0012] In combination with the first aspect, in some embodiments, multiple frame images of the second frame sequence are synthesized into one frame image, specifically including: for each pixel point representing the same position, the pixel values ​​of the pixel point in the multiple frame images of the second frame sequence are traversed, and the pixel value with the largest number of occurrences is taken as the pixel value of the corresponding pixel point in the synthesized frame image.

[0013] In combination with the first aspect, in some embodiments, the multiple frames of the second frame sequence are synthesized into one frame of image, specifically including: for each pixel representing the same position, the pixel values ​​of the pixel points in the multiple frames of the second frame sequence are first traversed, the pixel values ​​of the pixel points in the multiple frames of image are divided into different pixel value ranges, and the pixel value with the most occurrences in the pixel value range with the most pixel values ​​is taken as the pixel value of the corresponding pixel point in the synthesized one frame of image. In this way, foreground objects that appear briefly in the image can be effectively removed, and an excellent foreground removal effect can be obtained.

[0014] In conjunction with the first aspect, in some embodiments, during the n-stage iterative processing of the first frame sequence, the input and output of each stage are stored in a read-only memory ROM. ROM is a non-volatile memory, that is, the stored data will not be lost after power failure. Storing data in ROM can ensure that the method of the embodiment of the present application is not affected by actions such as restoring the factory settings of the electronic device or upgrading or restarting the system, and can be executed smoothly.

[0015] In combination with the first aspect, in some embodiments, after acquiring the first frame sequence and before iteratively processing the first frame sequence in n stages, the method may further include: performing anti-shake processing on each frame image in the first frame sequence respectively.

[0016] In combination with the first aspect or any one of the above embodiments, in some embodiments, after acquiring the first frame sequence and before iteratively processing the first frame sequence through n stages, the method may also include: detecting that the camera application is closed; or, detecting that the camera that captured the first frame sequence is not occupied.

[0017] In a second aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method provided in the first aspect or any one of the embodiments of the first aspect.

[0018] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect or any one of the embodiments of the first aspect is implemented.

[0019] According to a fourth aspect, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the method provided in the first aspect or any one of the embodiments of the first aspect is implemented.

[0020] In a fifth aspect, a chip system is provided, which includes a processing circuit and an interface circuit, the interface circuit is used to receive computer instructions and transmit them to the processing circuit, and the processing circuit is used to execute the computer instructions to implement the method provided in the first aspect or any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A The method flow of removing foreground from the cloud is shown; Figure 1B The process of the method for removing foreground on the end side is shown; Figure 2 A flowchart of a foreground removal method provided in an embodiment of the present application; Figure 3 It is a schematic diagram of iteratively processing the first frame sequence in n stages; Figure 4 The process of background enhancement processing of a clip is exemplified; Figure 5 A schematic diagram of synthesizing multiple frame images in a second frame sequence into one frame image; Fig. 6A and Figure 6B The foreground removal method provided by the embodiment of the present application is respectively compared with the foreground removal method before and after the foreground removal method provided by the embodiment of the present application is used; Figure 7 A software structure diagram of an electronic device provided in an embodiment of the present application; Figure 8 A hardware structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The image may include a foreground and a background.

[0023] The foreground usually refers to the scenery in the front of the image. It is closer to the lens and can guide the line of sight and the transition of scenery in the picture, which is more conducive to highlighting the subject. The foreground may be people, animals, objects, etc.

[0024] The background refers to all areas of the image except the foreground. The background is the scenery behind the image, which is farther away from the lens and often occupies a larger area in the picture than the foreground. The background may be the sky, the ground, buildings or other elements.

[0025] Figure 1A The method flow of removing the foreground from the cloud is shown. Figure 1A As shown, after the user triggers the electronic device to take a photo, the electronic device stores the taken photos in the gallery. Then, the electronic device uploads the photos in the gallery that need foreground removal to the cloud server. Afterwards, the cloud server calls the foreground removal algorithm to remove the foreground in the photo, and then returns the processed photo to the gallery of the electronic device. This method has the following problems: 1. Most users pay attention to privacy, and uploading photos to the cloud will cause some users to give up using this function. 2. The training of the foreground removal algorithm is based on the training data set, which is difficult to cover all shooting scenes. Therefore, this method is not a universal foreground removal method applicable to all shooting scenes, and it may be less effective in some shooting scenes. 3. The cost of accessing the foreground removal algorithm on the cloud server side is too high, and the computing power resources required of the cloud server are also high.

[0026] Figure 1B FIG. 4 shows the process of the method for removing the foreground on the end side. Figure 1BAs shown in the figure, after the electronic device takes a photo and generates an image, it first preprocesses the image, such as grayscale and Gaussian blur smoothing. Then it detects the foreground through edge detection and threshold segmentation. Then it uses morphological operations (such as dilation and erosion) to remove noise. Then it generates a foreground mask through contour detection. Finally, it uses an image restoration algorithm to fill the foreground area to make it consistent with the background. The foreground removal effect of this method is very poor, and it is only applicable to some images with regular foregrounds (such as raindrops).

[0027] The following embodiments of the present application provide a foreground removal method, which adopts reliable algorithm logic, has high-quality foreground removal effect, and has no special requirements for the shooting scene. In addition, the method is executed on the electronic device side, avoiding the high cost of foreground removal by the cloud server, and can effectively reduce the execution cost. This method does not need to upload user photos to the cloud, which can ensure data security and user privacy, and improve user experience and satisfaction. In addition, the method adopts end-side processing to avoid abnormal situations that may occur during cloud server processing, which can make the foreground removal effect of the image more controllable.

[0028] Figure 2 Flow chart of the foreground removal method provided in the embodiment of the present application. Figure 2 As shown, the method may include the following steps: First, obtain the frame sequence.

[0029] S101, the electronic device generates a first frame sequence including multiple frame images in response to a shooting operation of a user.

[0030] The method of the embodiment of the present application does not use a single frame image as input to remove the foreground, but uses a first frame sequence containing multiple frame images as input to remove the foreground. The multiple frame images in the first frame sequence can be arranged in order according to the generation time. With multiple frame images as input, the foreground removal effect can be more accurate.

[0031] S101 may be performed in a scene with foreground interference such as traffic or pedestrian flow, so that part or all of the frames in the first frame sequence may contain foreground interference with the shooting effect that needs to be removed.

[0032] The user's shooting operation may be a photo operation. The first frame sequence may be a plurality of frames of images output by a photosensitive element (sensor) in a camera currently activated by the electronic device within a period of time. The period of time includes the moment when the electronic device receives the shooting operation, and the first frame sequence may include the image output by the photosensitive element when the electronic device receives the shooting operation, and may also include the image output by the photosensitive element before the shooting operation, and may also include the image output by the photosensitive element after the shooting operation.

[0033] The number of the multiple frame images included in the first frame sequence can be represented as Y. The value of Y can be preset as needed and is not specifically limited here.

[0034] After the photosensitive element in the camera captures the image, the raw image data is usually first stored in the buffer of the electronic device. The buffer may be a part of the photosensitive element, or it may be an external memory area connected to the photosensitive element. In some embodiments of the present application, the electronic device can dump the first frame sequence into the read-only memory (ROM) of the electronic device. The format of the image dumped into the ROM can be RAW format, or it can also be RGB format. After the data is written to the ROM, the data therein cannot be modified or deleted at will, it can only be read, and can be quickly accessed and processed. Dumping the first frame sequence into the ROM can free up buffer space, and since the foreground removal method of the present application will take a certain amount of time, it can also facilitate the electronic device to read the first frame sequence from the ROM when executing the method of the present application. In other embodiments, the electronic device can also store the first frame sequence directly into the ROM without having to dump it from the buffer.

[0035] In some implementations, the user may use a tripod or other stable equipment to shoot, which can avoid severe shaking and make the image generated by the electronic device more stable, thereby achieving a better effect of subsequent foreground removal.

[0036] In other implementations, the user can turn on the anti-shake mode of the electronic device, so that after the photosensitive element outputs the image, the electronic device can use the anti-shake algorithm to perform anti-shake processing on the image, remove the abnormal foreground movement problem caused by shaking, and then dump the first frame sequence obtained after anti-shake to the ROM. The anti-shake algorithm may include one or more processes such as cropping, scaling, pixel prediction, pixel compensation, etc., and the embodiment of the present application does not specifically limit the anti-shake algorithm.

[0037] Second, the foreground is removed iteratively.

[0038] In this process, the first frame sequence is iteratively processed through multiple stages, gradually expanding from a local area to capture the foreground changes in the first frame sequence, and generating a second frame sequence containing a large amount of background information after removing the foreground.

[0039] S102, iteratively process the first frame sequence in n stages to obtain a second frame sequence, where n is an integer greater than or equal to 2.

[0040] The input of the first stage is the first frame sequence generated in S101, the input of each subsequent stage is the output of the adjacent previous stage, and the output of the last stage is the second frame sequence.

[0041] In each stage, the input image in the stage is divided into one or more segments, and each segment in the same stage contains the same number of multi-frame images. In each stage, after the input image is divided into one or more segments in turn, the last image that is not enough to be divided into a segment can also be discarded; then, the background enhancement processing can be performed on each frame image in the segment based on each segment to obtain the multi-frame image corresponding to the segment, and the images corresponding to the output of the segments contained in the stage are used as the input of the next adjacent stage. The images output by each stage can be stored in ROM, which is convenient for the electronic device to read in the next stage. The image output by the last stage can also be stored in ROM, which is convenient for the electronic device to execute the subsequent steps of S102.

[0042] When the first frame sequence and the images outputted in each of the above stages are stored in ROM, since ROM is a non-volatile memory, that is, the stored data will not be lost after power failure, even if the electronic device is restored to factory settings or the system is upgraded or restarted during the process of executing S102-S103, the data stored in ROM can be restored, thereby not affecting the results of the embodiment of the present application. Of course, in some other implementations, the above data can also be stored in a random access memory (RAM) or other types of storage areas.

[0043] The number of image frames contained in each segment in the first stage can be expressed as k, and the value of k can be preset. The number of image frames of each segment in each subsequent stage is increased by k on top of the number of image frames of each segment in the adjacent previous stage. In this way, the number of image frames of the segments in each stage gradually increases, and the foreground changes in the first frame sequence can be captured from a local perspective, and the obtained second frame sequence has a good foreground removal effect.

[0044] The total number of stages n can be determined according to the number of image frames Y in the first frame sequence and the number of image frames k included in the fragments in the first stage, where n is greater than or equal to 2. In some embodiments, based on the introduction in the first two paragraphs, the Y frame images in the first frame sequence can be segmented according to the logic of the first two paragraphs until only one segment can be divided in the last stage, so that the total number of stages n can be determined. In some other embodiments, the total number of stages n can also be less than the total number of stages determined in the previous embodiment, so that the last stage may be divided into 2 or more segments.

[0045] Figure 3 Schematic diagram of n-stage iterative processing of the first frame sequence.

[0046] like Figure 3As shown in the figure, the sensor in the camera outputs Y frames of images, i.e., the first frame sequence. The Y frames of images are marked as image (frame) 1, image (frame) 2, image (frame) 3…image (frame) Y in the order of generation time. The electronic device first uses the anti-shake algorithm to perform anti-shake processing on the Y frames of images output by the sensor, and writes the anti-shake processed Y frames of images into the ROM.

[0047] Phase 1 In the first stage, the electronic device reads Y frames of images from the ROM according to the order of generation time, and divides the Y frames of images into segments starting from the first frame of image, each segment contains k frames of images, until the number of frames of the remaining images is insufficient to constitute a segment. Assuming that the number of image frames contained in the segments in the first stage is k, then Y=a1*k+b1, where a1 is the number of segments in the first stage, b1 is the number of frames of images discarded in the first stage, and b1<k. If Y is an integer multiple of k, then b1 is 0.

[0048] For example, reference Figure 3 , Phase 1 includes the following segments: Fragment 1 (stage 1_1): image (frame) 1, image (frame) 2…image (frame) k.

[0049] Fragment 2 (stage 1_2): image (frame) k+1, image (frame) k+2…image (frame) 2k.

[0050] Fragment 3 (stage 1_3): image (frame) 2k+1, image (frame) 2k+2…image (frame) 3k.

[0051] … Fragment a1 (stage 1_a1): image (frame) (a1-1)k+1, image (frame) (a1-1)k+2…image (frame) a1*k.

[0052] The several frames of images discarded in the first stage are: image (frame) a1*k+1, image (frame) a1*k+2...image (frame) Y.

[0053] After the segments are divided, the electronic device performs background enhancement processing on each segment in the first stage to obtain multiple frames of images corresponding to each segment. The number of image frames output in the final first stage is a1*k, and the electronic device can store these a1*k frames of images in ROM as input for the second stage. The specific implementation of the electronic device performing background enhancement processing on each segment can refer to the subsequent embodiments.

[0054] Phase 2 In the second stage, the electronic device reads the a1*k frames of images output in the first stage from the ROM according to the order of generation time, and divides these images into segments starting from the first frame, each segment contains 2k frames of images, until the number of frames of the remaining images is insufficient to form a segment. That is, a1*k=a2*2k+b2, where a2 is the number of segments in the second stage, b2 is the number of frames of the discarded images in the second stage, and b2<2k. If a1*k is an integer multiple of 2k, then b2 is 0.

[0055] For example, reference Figure 3 , Phase 2 includes the following segments: Fragment 1 (stage 2_1): image (frame) 1, image (frame) 2…image (frame) 2k.

[0056] Fragment 2 (stage 2_2): image (frame) 2k+1, image (frame) 2k+2…image (frame) 4k.

[0057] Fragment 3 (stage 2_3): image (frame) 4k+1, image (frame) 4k+2…image (frame) 6k.

[0058] … Fragment a2 (stage 2_a2): image (frame) (a2-1)*2k+1, image (frame) (a2-1)*2k +2…image (frame) a2*2k.

[0059] The image frames discarded in the second stage are: image (frame) a2*2k +1, image (frame) a2*2k +2…image (frame) a1*k.

[0060] After the segments are divided, the electronic device performs background enhancement processing on each segment in the second stage to obtain multiple frames of images corresponding to each segment. The number of image frames output by the final second stage is a2*2k, and the electronic device can store these a2*2k frames of images in ROM as input for the third stage. The specific implementation of the electronic device performing background enhancement processing on each segment can refer to the subsequent embodiments.

[0061] … Stage n In the nth stage, the electronic device reads the a output in the n-1th stage from the ROM according to the order of generation time. n-1 *(n-1)k frames of images, and divide these images into segments starting from the first frame, each segment contains nk frames of images, until the number of frames of the remaining images is not enough to form a segment, where a n-1 is the number of fragments in the n-1th stage. n-1 * (n-1) k = a n *nk+b n , where a n is the number of fragments in stage n, b n is the number of frames of the discarded image in the nth stage, and b n <nk. If a n-1 * (n-1) k is an integer multiple of nk, then b n is 0. Figure 3 In, a n The value of is 1.

[0062] For example, reference Figure 3 , the nth stage includes the following fragments: Fragment 1 (stage n_1): image (frame) 1, image (frame) n…image (frame) nk.

[0063] The frames of images discarded in the nth stage are: image (frame) nk +1, image (frame) nk +n…image (frame) a n-1 * (n-1)k.

[0064] After the segments are divided, the electronic device performs background enhancement processing on each segment in the nth stage to obtain multiple frames of images corresponding to each segment, and the number of image frames output in the final nth stage is nk. The specific implementation of the electronic device performing background enhancement processing on each segment can refer to the subsequent embodiments.

[0065] In some embodiments, in each stage, each fragment in the stage will be placed in the thread pool in the electronic device as a task for background enhancement processing. Different fragments in the same stage can be processed in parallel as different tasks, and the electronic device can use the thread pool mechanism to perform parallel background enhancement processing on multiple fragments in the same stage. In some embodiments, the thread pool can maintain the starting frame information (such as the starting frame identifier startIndex), the ending frame information (such as the ending frame identifier endIndex), and the identifier of the fragment (stage x_j) of each fragment in the same stage, where x represents the stage, j represents the jth fragment, and stage x_j represents the jth fragment in the xth stage. The thread pool can also register or define a callback function for each fragment in the same stage to notify the main thread that processes the image in the electronic device that the background enhancement processing of the fragment in the stage has been completed. Figure 3 The diagram exemplarily shows the information maintained by the thread pool corresponding to the fragment (which includes one fragment) in the nth stage and the defined callback function.

[0066] At this point, the electronic device has completed the n-stage iterative processing of the Y frame images, and obtained nk frame images, which are the second frame sequence. Since some images are discarded during the n-stage iterative processing, the number of image frames nk of the second frame sequence is less than the number of image frames Y of the first frame sequence.

[0067] In the above process of performing n-stage iterative processing on the Y frame image, the number of image frames of the fragments in each stage gradually increases, and the foreground changes in the first frame sequence can be captured by gradually expanding from the local. This method of setting multiple stages and gradually increasing the number of image frames of the fragments in each stage is equivalent to using both short frames containing a small number of frame images and long frames containing more frame images in this process, which has a better foreground removal effect than using only one stage to process the image.

[0068] For example, if only one stage is set and the segments are divided according to short frames in the stage, then in a scene where a foreground object appears for a short time, some segments may not be able to capture the changes of the foreground object, and eventually the foreground object will be regarded as the background, resulting in poor foreground removal effect.

[0069] For another example, if only one stage is set and the segments are divided according to long frames in this stage. Suppose a cat constantly appears and disappears in the shooting scene, with its position and posture unchanged, and its appearance time t > T / 2, where T is the duration of the first frame sequence. Then, if only long frames are used, this cat will be recognized as the background, which does not conform to the actual situation. If the n-stage iterative processing algorithm that gradually expands from short frames to long frames in this application is used, it is very likely that during the process of intercepting short frames, it will be found that in some segment p, the appearance time tp of the cat < Tp / 2, where Tp is the duration of segment p. Thus, the cat will be recognized as the foreground. And during the continuous calculation process, due to the certain posture change of the cat, the information that "the cat is the foreground" in this segment will be continuously amplified, and finally, the cat in the entire first frame sequence will be recognized as the foreground, thus achieving the removal effect. In other words, the judgment of the foreground is relatively loose. Whether an object is found to be the foreground in short frames or long frames, using the stage iterative foreground removal in this application can remove it; while the judgment of the background is relatively strict. An object will be finally determined as the background only if it remains unchanged (position, posture, speed, etc.) in both short frames and long frames.

[0070] The following describes how to perform background enhancement processing on the segments in each stage. In the embodiments of this application, the logic of the background enhancement processing performed on each segment in each stage is the same.

[0071] Background enhancement processing In the embodiments of this application, the principle of performing background enhancement processing on a segment is to perform background enhancement processing on each frame image in the segment based on multiple frame images in the segment, and output images with the same number of frames as the images in the segment. Since background enhancement processing is performed on each frame image based on multiple frame images instead of directly performing background enhancement processing on this frame image, more image information can be referred to during background enhancement processing, and the foreground and background in the frame image can be better distinguished.

[0072] The following gives a specific example of an electronic device performing background enhancement processing on a segment. The segment in this example contains m frame images.

[0073] Figure 4 Exemplarily shows the processing flow in this example. As Figure 4 shown, this process may include the following steps: 1. Calculate the average frame of the m frame images in the segment.

[0074] First, the electronic device reads the m frame images and ensures that the m frame images have the same size. If the image sizes are different, the m frame images need to be adjusted to the same size first.

[0075] Then, the electronic device can calculate the average frame of the m frames by accumulating frame by frame through incremental calculation. For example, the electronic device can use Formula 1 to calculate the average value of the m frames.

[0076] Formula 1 in, Represents the average frame of m frames, startIndex is the start frame identifier in the current segment, and endIndex is the end frame identifier in the current segment. Indicates the i-th frame image in the segment. Figure 4 In the example, the starting frame in the current segment is shown as , the ending frame in the current segment is shown as .

[0077] In an embodiment of the present application, the electronic device can calculate the average value for each pixel position, and after traversing all pixel positions, the average frame of the m frames of images in the current segment can be obtained.

[0078] 2. Calculate the difference frame between each frame in the m frame image and the average frame.

[0079] Exemplarily, the electronic device may use Formula 2 to calculate the difference frame between each frame of the m frames and the average frame.

[0080] Formula 2 in, Indicates the difference frame between the i-th frame image and the average frame in the current segment. In the embodiment of the present application, the electronic device can calculate the difference for each pixel position, and after traversing all pixel positions, the difference frame between the i-th frame image and the average frame can be obtained.

[0081] 3. Convert the difference frame between each frame in the m frames and the average frame into a grayscale image.

[0082] This step is equivalent to grayscale conversion of the output of step 2, converting the difference frame from RGB color space to grayscale color space. Grayscale images only contain brightness information, not color information.

[0083] Exemplarily, the electronic device may use Formula 3 to convert the difference frame into a grayscale image.

[0084] Formula 3 in, Represents the grayscale image converted from the difference frame between the i-th frame image and the average frame in the current segment, R is the red value in the difference frame, G is the green value in the difference frame, and B is the blue value in the difference frame.

[0085] 4. Perform threshold segmentation on each grayscale image to obtain the corresponding binary image, and then convert the binary image into foreground image and background image in RGB format.

[0086] Exemplarily, the electronic device may use Formula 4 to perform threshold segmentation on each grayscale image to obtain a corresponding binary image.

[0087] Formula 4 in, represents a binary image, and threshold represents a threshold. The threshold can be set as needed and is not limited here. In an embodiment of the present application, the electronic device can calculate the corresponding binary value for each pixel position of the i-th frame grayscale image, and after traversing all pixel positions, the i-th frame binary image can be obtained.

[0088] Then, the electronic device can convert the binary image into a foreground image and a background image with color information. Specifically, for each frame of the binary image, the pixel point with a pixel value of 255 is regarded as a pixel in the foreground image, and the pixel point with a pixel value of 0 is regarded as a pixel in the background image, so that the corresponding pixels in the foreground image and the corresponding pixels in the background image can be known from the binary image. Find the color values ​​(i.e. RGB values) of the above pixels, fill these values ​​into the foreground image and background image, and you will get the foreground image and background image with color information.

[0089] In step 4, m frames of binary images can be obtained, and each frame of the binary image can be converted into a corresponding frame of foreground image and a frame of background image with color information.

[0090] 5. Perform weighted summation on each pair of foreground and background images to obtain the corresponding fused image.

[0091] Exemplarily, the electronic device may use Formula 5 to perform weighted summation on a pair of foreground images and background images to obtain a corresponding fused image.

[0092] Formula 5 in, represents the fused image of the i-th frame, represents the foreground image converted from the binary image of the i-th frame in step 4, represents the background image converted from the binary image of the i-th frame in step 4, and α is the weighting coefficient.

[0093] α can be preset, and its specific value is not limited here. For example, it can be 0.1 or other values. In some embodiments, α can also be adaptively adjusted according to the current shooting scene. For example, when there are more foreground objects (such as more people), the value of α is higher. The electronic device can analyze the number of foreground objects according to the image output by the sensor, and then determine the value of α.

[0094] This step can obtain m frames of fused images.

[0095] pass Figure 4 In the processing flow shown, the m-frame fused image obtained by the electronic device is the result of removing the foreground and enhancing the background of each frame image in the current segment.

[0096] Figure 4 This is just an example. The embodiment of the present application may also use other methods to perform background enhancement processing on multiple frames in each segment, as long as the effect of performing background enhancement processing on each frame in the reference segment can be achieved. The embodiment of the present application does not specifically limit the method of performing background enhancement processing on each segment.

[0097] Finally, multiple frame images of the second frame sequence are combined into one frame image.

[0098] S103, the electronic device combines the multiple frame images of the second frame sequence into one frame image.

[0099] The method of synthesizing the multiple frame images in the second frame sequence into one frame image may include the following two methods: Method 1, traverse the pixel values ​​of the same pixel point in multiple frames of images, and take the pixel value with the largest number of occurrences as the pixel value of the corresponding pixel point in the final synthesized frame of image. After performing the above steps on each pixel point representing the same position in the multiple frames of images in the second frame sequence, the pixel value of each pixel point in the final synthesized frame of image can be obtained, and the final synthesized frame of image is obtained.

[0100] Method 2, first traverse the pixel values ​​of the same pixel point in multiple frames of images, divide these pixel values ​​into different pixel value ranges (also called sets), and then take the pixel value with the most occurrences in the pixel value range with the most pixel values ​​as the pixel value of the corresponding pixel point in the final synthesized frame of image. Among them, the sizes of each pixel value range may be the same or different. In some embodiments, a neighbor classification (k-nearest neighbor, KNN) algorithm can be used to classify multiple pixel values, and multiple different pixel value ranges can be obtained. After performing the above steps on each pixel point in the multiple frames of images in the second frame sequence, the pixel value of each pixel point in the final synthesized frame of image can be obtained, and the final synthesized frame of image can be obtained.

[0101] Figure 5 This is a schematic diagram of method 2. Figure 5 As shown, for the first pixel (pixel 1), the pixel value of the pixel in the multi-frame image is divided into different pixel value ranges, and then the pixel value range with the most pixel values ​​(such as Figure 5 The pixel value with the largest number of occurrences is taken as the pixel value of the first pixel point. The processing of the remaining pixel points is similar.

[0102] By adopting method 2, foreground objects that appear briefly in the image can be effectively removed, and an excellent foreground removal effect can be obtained. Compared with method 1, method 2 can avoid the ghost image produced when the foreground object is temporarily motionless. Figure 5 The last two pixel value ranges of the first pixel in the image are as follows: the pixel values ​​in these two pixel value ranges may represent a foreground object that appears briefly, and the foreground object can be effectively removed by using method 2.

[0103] The format of the frame of image synthesized by the electronic device in S103 may be a JPEG (joint photographic experts group) format or other formats, which is not limited in this embodiment of the present application.

[0104] At this point, the electronic device has completed the foreground removal during the shooting process. The frame of image outputted in S103 is the result of the shooting by the electronic device, and the result can be displayed on a display screen for the user to view.

[0105] Fig. 6A and Figure 6B The diagrams are respectively shown as before and after comparisons of the foreground removal method provided by the embodiments of the present application.

[0106] refer to Fig. 6A , Fig. 6A The left side is an image without using the foreground removal method of the embodiment of the present application, which can be regarded as an original image. The original image can be an image in the first frame sequence in S101. There is foreground interference in the original image, such as a vehicle on the middle road. Fig. 6A The right side of is an image after using the foreground removal method of an embodiment of the present application. Most of the foreground interference is removed from the image, and the vehicles on the road in the middle of the image are removed.

[0107] refer to Figure 6B , Figure 6B The left side is an image without using the foreground removal method of the embodiment of the present application, which can be regarded as an original image. The original image can be an image frame in the first frame sequence in S101. There is foreground interference in the original image, such as a person; Figure 6BThe right side of is an image after using the foreground removal method of an embodiment of the present application. Most of the foreground interference is removed from the image, and the person in the image is removed.

[0108] In the embodiment of the present application, S102-S103 may be performed in the following situations: 1. After the electronic device obtains the first frame sequence in S101, it can execute S102-S103, giving the user real-time feedback so that the user can immediately see the foreground removal effect.

[0109] 2. After the electronic device obtains the first frame sequence in S101, it first stores the first frame sequence, and then executes S102-S103 when the computing power is idle (for example, after the camera application is closed, or when the camera in the electronic device that shoots the first frame sequence is not occupied). This does not occupy the computing power of the electronic device when shooting, and can provide a smoother shooting experience.

[0110] 3. The user triggers the electronic device to execute S102-S103. For example, after the electronic device obtains the first frame sequence in S101, it first stores the first frame sequence, first displays the original image to the user, and then executes S102-S103 after receiving the user input of the operation of removing the foreground. In this way, the foreground in the image can be removed according to the actual needs of the user.

[0111] 4. After the electronic device recognizes that the captured image has a relatively messy or large foreground, it can automatically execute S102-S103, so that the foreground can be removed for the required image.

[0112] Moreover, the foreground removal method provided in the embodiment of the present application can be applied not only to the photo scene, but also to the video scene. For the video scene, the method provided in the present application can be used to remove the foreground of multiple first frame sequences, thereby obtaining multiple frames of video images.

[0113] Other implementations of the present application When the first frame sequence is subjected to n-stage iterative processing in S102, when the input multi-frame image is segmented in each stage, it is not necessary to segment from the first frame of the multi-frame image from the front to the back, so that the last few frames of the image may be discarded. In other embodiments, the multi-frame image may be segmented from the last frame of the multi-frame image from the back to the front, and the first few frames of the image may be discarded. In other embodiments, the multi-frame image in each segment may not be continuous frames, but may skip frames, and frames may be skipped between segments.

[0114] When the first frame sequence is iteratively processed in n stages in S102, it is not limited to that the number of image frames of the fragments in each subsequent stage is increased by k over the number of image frames of the fragments in the adjacent previous stage. In other implementations, as long as the number of image frames of the fragments in the subsequent stage is greater than the number of image frames of the fragments in the adjacent previous stage, it is sufficient. In this way, the foreground changes in the first frame sequence can be captured by gradually expanding from the local area.

[0115] When performing n-stage iterative processing on the first frame sequence in S102, images that are not sufficient to form a segment in each stage may not be discarded, and images that are not sufficient to form a segment may be treated as a separate segment. In this way, the number of image frames in the second frame sequence is the same as the number of image frames in the first frame sequence.

[0116] When performing n-stage iterative processing on the first frame sequence in S102, the number of frames contained in each segment in each stage may also be different.

[0117] Other application scenarios of the embodiments of the present application In addition to being applied to scenes in which the foreground contains distracting objects such as traffic or pedestrians, the method of the embodiment of the present application can also be applied to scenes in which rain or fog is captured, and can be used to remove the rain or fog in the foreground.

[0118] The logic of the embodiment of the present application can also be applied to the cutout function.

[0119] For example, Figure 4 In the background enhancement process performed on the fragment shown in FIG. 1 , after obtaining m frames of colored foreground images in step 4, the foreground images can be Figure 2 In S103, the m frames of colored foreground images are synthesized into one frame of image, which is the result of output after matting.

[0120] Or, for example, Figure 4 In the background enhancement process performed on the fragment shown in FIG. 1 , after obtaining m frames of colored background images in step 4, the background images can be Figure 2 In S103, the background images of the m frames are synthesized into one frame of image, i.e., the background image, and then the original image is subtracted from the frame image, and the subtracted image is the result of the output after the cutout. Since the n-stage iterative processing method provided in the embodiment of the present application is more effective for the recognition of the background in the frame sequence, the background image is first generated using the method of the embodiment of the present application, and then the background image is subtracted from the original image (or the original frame sequence, i.e., the first frame sequence) to obtain the foreground image (or the foreground frame sequence), and the foreground frame sequence can also be synthesized into one frame of image.

[0121] Figure 7 A software structure diagram of the electronic device 100 provided in an embodiment of the present application.

[0122] The electronic device 100 may be used to execute the foreground removal method described above.

[0123] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes a mobile operating system with a layered architecture as an example to exemplify the software structure of the electronic device 100.

[0124] Figure 7 It is a software structure diagram of the electronic device 100 according to an embodiment of the present application.

[0125] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the mobile operating system is divided into four layers, from top to bottom: application layer, application framework layer / core service layer, system library and runtime, and kernel layer.

[0126] The application layer can include a series of application packages.

[0127] like Figure 7 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0128] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0129] like Figure 7 As shown, the application framework layer may include a foreground removal module, an image synthesis module, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0130] The foreground removal module can be used to perform the above Figure 2 Step S102 in the method shown. The foreground removal module may be integrated with a background enhancement processing algorithm, which may be referred to as Figure 4 The processing flow.

[0131] The image synthesis module can be used to perform the above Figure 2 Step S103 in the method shown.

[0132] Of course, in some other implementations, the foreground removal module and the image synthesis module may also be located in other layers of the software framework, which is not limited in the embodiments of the present application.

[0133] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0134] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0135] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0136] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0137] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0138] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0139] The runtime can refer to all code libraries, frameworks, etc. required for the program to run. For example, for the C language, the runtime includes a series of function libraries required for the C program to run. For the Java language, in addition to the core library, the runtime also includes the virtual machine required for the Java program to run. The above core library may include the function functions that the Java language needs to call.

[0140] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0141] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0142] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0143] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0144] A 2D graphics engine is a drawing engine for 2D drawings.

[0145] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0146] Figure 8 The hardware structure diagram of the electronic device 100 provided in the embodiment of the present application is shown in FIG. The electronic device 100 is used to execute the foreground removal method provided in the above method embodiment.

[0147] The electronic device 100 may include a processor 101, a memory 102, a wireless communication module 103, a mobile communication module 104, an antenna 103A, an antenna 104A, a power switch 105, a sensor module 106, a focus motor 107, a camera 108, a display screen 109, etc. Among them, the sensor module 106 may include a gyroscope sensor 106A, an acceleration sensor 106B, an ambient light sensor 106C, an image sensor 106D, a distance sensor 106E, etc. Among them, the wireless communication module 103 may include a WLAN communication module, a Bluetooth communication module, etc. The above multiple parts can transmit data through a bus.

[0148] The processor 101 may include one or more processing units, for example, the processor 101 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0149] The memory 102 may be used to store computer executable program codes, and the executable program codes may include instructions. The processor 101 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 102. The memory 102 may include a program storage area and a data storage area. In a specific implementation, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0150] The wireless communication function of the electronic device 100 can be implemented through the antenna 103A, the antenna 104A, the mobile communication module 104, the wireless communication module 103, the modem processor and the baseband processor.

[0151] Antenna 103A and antenna 104A can be used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas.

[0152] The mobile communication module 104 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 104 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 104 can receive electromagnetic waves through the antenna 104A, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 104 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 104A.

[0153] The modulation and demodulation processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is transmitted to the application processor. The application processor outputs a sound signal through an audio device, or displays an image or video through the display screen 109.

[0154] The wireless communication module 103 can provide wireless communication solutions including wireless local area networks (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 103 can be one or more devices integrating at least one communication processing module. The wireless communication module 103 receives electromagnetic waves via the antenna 103A, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 101. The wireless communication module 103 can also receive the signal to be sent from the processor 101, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 103A.

[0155] The power switch 105 may be used to control the supply of power to the electronic device 100 .

[0156] The gyro sensor 106A can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 106A. The gyro sensor 106A can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 106A detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 106A can also be used for navigation and somatosensory game scenes.

[0157] The acceleration sensor 106B can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device. For example, the acceleration sensor 106B can be applied to applications such as horizontal and vertical screen switching, pedometers, etc.

[0158] The ambient light sensor 106C is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 109 according to the sensed brightness of the ambient light. The ambient light sensor 106C can also be used to automatically adjust the white balance when taking pictures.

[0159] The image sensor 106D, also known as a photosensitive element, can utilize the photoelectric conversion function of the photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image. The image sensor can be a charge coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor.

[0160] The distance sensor 106E can be used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some shooting scenarios, the electronic device 100 can use the distance sensor 106E to measure the distance to achieve fast focusing.

[0161] The focus motor 107 can be used for fast focusing. The electronic device 100 can control the movement of the lens through the focus motor 107 to achieve automatic focusing.

[0162] The electronic device 100 can realize the shooting function through the ISP, the camera 108, the video codec, the GPU, the display screen 109 and the application processor.

[0163] The ISP is used to process the data fed back by the camera 108. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 108.

[0164] The camera 108 can be used to capture still images or videos. The object generates an optical image through the lens and projects it onto the image sensor. The image sensor can convert the optical signal into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP can output the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 108, where N is a positive integer greater than 1.

[0165] The video codec is used to compress or decompress digital images. The electronic device 100 may support one or more image codecs. In this way, the electronic device 100 can open or save pictures or videos in multiple coding formats.

[0166] The electronic device 100 can realize the display function through a GPU, a display screen 109, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 109 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 101 may include one or more GPUs, which execute program instructions to generate or change display information.

[0167] The display screen 109 is used to display images, videos, etc. The display screen 109 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 109, where N is a positive integer greater than 1.

[0168] In the embodiment of the present application, the memory 102 stores a computer program for implementing the foreground removal method provided in the embodiment of the present application, and the processor 101 can be used to execute the computer program to implement the method provided in the embodiment of the present application on the electronic device. Specifically, the GPU in the processor 101 can be used to execute Figure 2 S102-S103 in the method shown. The operations performed by the components in the electronic device 100 may be specifically referred to the relevant description of the above method embodiment, which will not be elaborated in detail here.

[0169] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0170] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0171] The present application also provides an electronic device, which may include: a memory and a processor. The memory may be used to store a computer program; the processor may be used to call the computer program in the memory so that the electronic device executes the method executed by the electronic device side in any of the above embodiments.

[0172] The present application also provides a chip system, which includes a processing circuit and an interface circuit. The interface circuit is used to receive computer instructions and transmit them to the processing circuit. The processing circuit is used to run the computer instructions to implement the method executed by the electronic device in any of the above embodiments.

[0173] The present application also provides a chip system, which includes at least one processor, for implementing the method executed by the electronic device in any of the above embodiments. In a possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0174] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0175] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0176] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0177] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0178] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the electronic device in any of the above embodiments.

[0179] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0180] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0182] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0183] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0184] The terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0185] In short, the above is only an embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of this application shall be included in the protection scope of this application.

Claims

1. A foreground removal method, characterized in that: The method comprises: Acquire a first frame sequence, where the first frame sequence includes multiple frame images; Iteratively processing the first frame sequence in n stages to obtain a second frame sequence, wherein the second frame sequence includes a plurality of frames of images, and n is an integer greater than or equal to 2; Combining multiple frames of the second frame sequence into one frame of image; Wherein, in the process of iteratively processing the first frame sequence in the n stages: The input of the first stage is the first frame sequence, the input of each subsequent stage is the output of the adjacent previous stage, and the output of the nth stage is the second frame sequence; For each stage, the input image of the stage is divided into one or more segments, and for each segment, background enhancement processing is performed on each frame image in the segment based on the segment, and multiple frames of images corresponding to the segment are output, wherein the output images corresponding to the segments included in the stage are the outputs of the stage; The number of image frames of each segment in the latter stage is greater than the number of image frames of each segment in the adjacent previous stage.

2. The method according to claim 1, characterized in that The number of image frames in each segment of the same stage is the same.

3. The method according to claim 2, characterized in that The number of image frames of the fragments in the first stage is a preset value, and the number of image frames of the fragments in each subsequent stage is increased by the preset value over the number of image frames of the fragments in the adjacent previous stage.

4. The method according to claim 2, characterized in that: During the n-stage iterative processing of the first frame sequence, images that are not large enough to be divided into a segment are discarded in each stage.

5. The method according to claim 1, characterized in that: During the n-stage iterative processing of the first frame sequence, background enhancement processing is performed on images of different segments in the same stage in parallel.

6. The method according to claim 1, characterized in that For a segment, the steps of performing background enhancement processing on each frame image in the segment based on the segment and outputting multiple frames of images corresponding to the segment include: Calculate the average frame of multiple frames of images in the segment; Calculate the difference frame between each frame image in the segment and the average frame; Converting the plurality of difference frames into grayscale images; Performing threshold segmentation on each of the grayscale images to obtain a corresponding binary image; Convert each of the binary images into a foreground image and a background image with color information; Perform weighted summation on each pair of foreground image and background image to obtain the corresponding fused image.

7. The method according to claim 6, characterized in that When there are more foreground objects in the multiple frame images included in the first frame sequence, when weighted summing is performed on the foreground image and the background image, the weight corresponding to the foreground image is larger.

8. The method according to claim 1, characterized in that Combining the multiple frames of the second frame sequence into one frame of image specifically includes: For each pixel representing the same position, the pixel values ​​of the pixel in multiple frames of the second frame sequence are traversed, and the pixel value with the largest number of occurrences is taken as the pixel value of the corresponding pixel in the synthesized frame of the image.

9. The method according to claim 1, characterized in that: Combining the multiple frames of the second frame sequence into one frame of image specifically includes: For each pixel point representing the same position, the pixel values ​​of the pixel point in the multiple frame images of the second frame sequence are first traversed, the pixel values ​​of the pixel point in the multiple frame images are divided into different pixel value ranges, and the pixel value with the largest number of occurrences in the pixel value range with the largest number of pixel values ​​is taken as the pixel value of the corresponding pixel point in the synthesized frame image.

10. The method according to claim 1, characterized in that During the n-stage iterative processing of the first frame sequence, the input and output of each stage are stored in a read-only memory ROM.

11. The method according to claim 1, characterized in that: After acquiring the first frame sequence and before iteratively processing the first frame sequence in n stages, the method further includes: Anti-shake processing is performed on each frame image in the first frame sequence.

12. The method according to any one of claims 1 to 11, characterized in that: After acquiring the first frame sequence and before iteratively processing the first frame sequence in n stages, the method further includes: The camera application is detected to be closed; or, It is detected that the camera shooting the first frame sequence is not occupied.

13. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A chip system, characterized in that: The chip system includes a processing circuit and an interface circuit, wherein the interface circuit is used to receive computer instructions and transmit them to the processing circuit, and the processing circuit is used to execute the computer instructions to implement the method as described in any one of claims 1-12.

Images