Background replacement method and apparatus, electronic device, and storage medium

Abstract

The application relates to the technical field of image processing, and provides a background replacement method and device, electronic equipment and a storage medium, which comprises the following steps: obtaining a reference image containing a target background and an image to be processed containing a pure color background, and determining the pixel value of each pixel point in the image to be processed; then determining a threshold value according to the pixel value of each pixel point in the image to be processed, wherein the threshold value represents a critical value for distinguishing the foreground and the background in the image to be processed; then obtaining a mask image according to the pixel value of each pixel point in the image to be processed and the threshold value; and finally performing background replacement on the image to be processed according to the mask image and the reference image to obtain a target image, wherein the target image contains the foreground in the image to be processed and the target background. The threshold value, i.e. the parameter, for distinguishing the background and the foreground is automatically determined, the parameter is adapted to the application scene, the operation is simplified, the background replacement effect is optimized, and the background replacement processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a background replacement method, apparatus, electronic device, and storage medium. Background Technology

[0002] Background replacement refers to replacing the background of a video frame in an image or video. It is used in many scenarios, such as replacing the background of a photograph of a person or replacing the background of a live stream. Current technologies typically involve manually determining parameters to achieve background replacement, which suffers from operational complexity and poor replacement results. Summary of the Invention

[0003] In view of this, the object of the present invention is to provide a background replacement method, apparatus, electronic device and storage medium.

[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0005] In a first aspect, the present invention provides a background replacement method, the method comprising:

[0006] Obtain a reference image containing the target background and an image to be processed containing a solid color background;

[0007] Determine the pixel value of each pixel in the image to be processed;

[0008] A threshold value is determined based on the pixel value of each pixel in the image to be processed; the threshold value represents the critical value that distinguishes the foreground and background in the image to be processed.

[0009] A mask image is obtained based on the pixel value of each pixel in the image to be processed and the threshold value;

[0010] The background of the image to be processed is replaced according to the mask image and the reference image to obtain a target image, wherein the target image contains the foreground and the target background of the image to be processed.

[0011] In an optional implementation, the step of determining the threshold value based on the pixel value of each pixel in the image to be processed includes:

[0012] The pixel values ​​of each pixel in the image to be processed are normalized and smoothed to obtain the smoothed pixel values ​​of each pixel in the image to be processed.

[0013] According to preset parameters, histogram statistics are performed on the smoothed pixel values ​​of all pixels in the image to be processed to obtain multiple reference values ​​and the frequency corresponding to each reference value.

[0014] Obtain the reference values ​​corresponding to the two maximum frequencies to obtain two first reference values;

[0015] Obtain the second reference value corresponding to the minimum frequency from the reference value interval formed by the two first reference values;

[0016] The second reference value is denormalized to obtain the threshold value.

[0017] In an optional implementation, the step of obtaining the mask image based on the pixel value of each pixel in the image to be processed and the threshold value includes:

[0018] Each pixel in the image to be processed whose pixel value is less than the threshold value is taken as a first pixel, and a first region containing all first pixels is obtained; the first region represents the foreground in the image to be processed.

[0019] Each pixel in the image to be processed whose pixel value is equal to or greater than the threshold value is taken as a second pixel, resulting in a second region containing all second pixels; the second region represents the background in the image to be processed.

[0020] Obtain a mask image that includes the first region and the second region.

[0021] In an optional implementation, the step of performing background replacement on the image to be processed based on the mask image and the reference image to obtain the target image includes:

[0022] The mask image is optimized to obtain an optimized mask image;

[0023] Based on the optimized mask image, the reference image and the image to be processed are fused to obtain the target image.

[0024] In an optional implementation, the mask image includes a first region representing the foreground in the image to be processed and a second region representing the background in the image to be processed;

[0025] The step of optimizing the mask image to obtain an optimized mask image includes:

[0026] The mask image is eroded to obtain the boundary region;

[0027] Based on each pixel in the boundary region and the HSV color model, the HSV parameters corresponding to each pixel in the boundary region are obtained;

[0028] The pixels in the boundary region whose HSV parameters meet the preset conditions are assigned to the first region to obtain the undetermined mask image;

[0029] The undetermined mask image is subjected to median filtering and guided filtering to obtain the optimized mask image.

[0030] In an optional implementation, after the step of determining the pixel value of each pixel in the image to be processed, the method further includes:

[0031] The image to be processed is reduced to a thumbnail image of a preset resolution, and the pixel value of each pixel in the thumbnail image is obtained;

[0032] The threshold value is determined based on the pixel value of each pixel in the thumbnail image.

[0033] In an optional implementation, the step of acquiring a reference image containing a target background and an image to be processed containing a solid color background includes:

[0034] In response to the user's selection, obtain a reference image containing the target background;

[0035] In response to the user's live stream start operation, the system captures each frame to be broadcast in real time and treats each frame as an image to be processed.

[0036] Secondly, the present invention provides a background replacement device, the device comprising:

[0037] The acquisition module is used to acquire a reference image containing the target background and an image to be processed containing a solid color background;

[0038] The processing module is used to determine the pixel value of each pixel in the image to be processed;

[0039] A threshold value is determined based on the pixel value of each pixel in the image to be processed; the threshold value represents the critical value that distinguishes the foreground and background in the image to be processed.

[0040] A mask image is obtained based on the pixel value of each pixel in the image to be processed and the threshold value;

[0041] The replacement module is used to perform background replacement on the image to be processed based on the mask image and the reference image to obtain a target image, wherein the target image includes the foreground and the target background in the image to be processed.

[0042] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing a computer program, wherein when the processor executes the computer program, it implements the method described in any of the foregoing embodiments.

[0043] Fourthly, the present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the method described in any one of the foregoing embodiments.

[0044] The background replacement method, apparatus, electronic device, and storage medium provided in this invention acquire a reference image containing a target background and an image to be processed containing a solid-color background, and determine the pixel value of each pixel in the image to be processed. Then, a threshold value is determined based on the pixel value of each pixel in the image to be processed; the threshold value represents the critical value for distinguishing the foreground and background in the image to be processed. Next, a mask image is obtained based on the pixel value of each pixel in the image to be processed and the threshold value. Finally, the background of the image to be processed is replaced based on the mask image and the reference image to obtain a target image, which contains both the foreground and the target background in the image to be processed. By automatically determining the threshold value, i.e., the parameter, for distinguishing the background and foreground, the parameters are adapted to the application scenario, thereby simplifying the operation, optimizing the background replacement effect, and improving the processing efficiency of background replacement.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A block diagram of an electronic device provided in an embodiment of the present invention is shown;

[0048] Figure 2 This invention illustrates a flowchart of a background replacement method provided in an embodiment of the present invention.

[0049] Figure 3 This illustration shows another flowchart of the background replacement method provided in an embodiment of the present invention;

[0050] Figure 4 An example diagram of the background replacement method provided in an embodiment of the present invention is shown;

[0051] Figure 5 This illustration shows another flowchart of the background replacement method provided in an embodiment of the present invention;

[0052] Figure 6This illustration shows another flowchart of the background replacement method provided in an embodiment of the present invention;

[0053] Figure 7 Another example diagram of the background replacement method provided in an embodiment of the present invention is shown;

[0054] Figure 8 Another example diagram of the background replacement method provided in an embodiment of the present invention is shown;

[0055] Figure 9 Another example diagram of the background replacement method provided in an embodiment of the present invention is shown;

[0056] Figure 10 Another example diagram of the background replacement method provided in an embodiment of the present invention is shown;

[0057] Figure 11 A functional block diagram of a background replacement device provided in an embodiment of the present invention is shown.

[0058] Icons: 110 - Bus; 120 - Processor; 130 - Memory; 150 - I / O Module; 170 - Communication Interface; 300 - Background Replacement Device; 310 - Acquisition Module; 330 - Processing Module; 350 - Replacement Module. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0060] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0061] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0062] Background replacement refers to replacing the background of a video frame in an image or video. It is used in many scenarios, such as replacing the background of a photographed person, replacing the background of a live stream, or replacing the background of a video frame. Since background replacement typically requires distinguishing between the foreground and background in an image based on parameters, existing technologies generally determine these parameters manually. However, for different application scenarios, manually determined parameters may not be suitable for the scene, leading to complex operation and poor replacement results. Therefore, this invention provides a background replacement method to solve the above problems.

[0063] Please refer to Figure 1 This is a block diagram of an electronic device provided in an embodiment of the present invention. The electronic device includes a bus 110, a processor 120, a memory 130, an I / O module 150, and a communication interface 170.

[0064] Bus 110 may be a circuit that interconnects the aforementioned components and transmits communication (e.g., control messages) between them.

[0065] The processor 120 can receive commands from the other components (such as memory 130, I / O module 150, communication interface 170, etc.) via bus 110, interpret the received commands, and perform calculations or data processing according to the interpreted commands.

[0066] The processor 120 can be an integrated circuit chip with signal processing capabilities. The processor 120 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0067] The memory 130 may store commands or data received from the processor 120 or other components (such as I / O module 150, communication interface 170, etc.) or commands or data generated by the processor 120 or other components.

[0068] The memory 130 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM).

[0069] The I / O module 150 can receive commands or data input by the user via input-output means (e.g., sensors, keyboard, touch screen, etc.), and can transmit the received commands or data to the processor 120 or memory 130 via the bus 110. It is also used to display various information received, stored, and processed from the aforementioned components (e.g., multimedia data, text data), and can display videos, images, data, etc., to the user.

[0070] Communication interface 170 can be used to communicate with other node devices for signaling or data.

[0071] Understandable Figure 1 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0072] The electronic devices provided in this embodiment of the invention can be smartphones, personal computers, tablets, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This embodiment of the invention does not impose any limitations on these.

[0073] The above-mentioned electronic device will be used as the execution subject to execute the various steps of the methods provided in the embodiments of the present invention and achieve the corresponding technical effects.

[0074] Please see Figure 2 , Figure 2 This is a schematic flowchart of a background replacement method provided in an embodiment of the present invention.

[0075] Step S202: Obtain a reference image containing the target background and an image to be processed containing a solid color background;

[0076] In this embodiment, the reference image includes the target background, which is the background that the user expects to display in the image after background replacement. The image to be processed is the image whose background is to be replaced, and its background is a solid color, i.e., a single color. For example, the background in the image to be processed can be green or other colors.

[0077] For ease of understanding, this embodiment of the invention uses a green background in the image to be processed as an example. For images to be processed with a non-green background, color conversion can be performed on the image to be processed, i.e., changing the Hue value of the image to be processed, to change the background color to green.

[0078] For different application scenarios, reference images can be obtained based on user interaction, or default reference images can be preset.

[0079] If the user is replacing the background of a captured image of a person, that image will be used as the image to be processed. If the user is replacing the background of a live stream, and the background of the captured image is a solid color (e.g., a green screen can be used), then each captured image will be used as the image to be processed. If the user is replacing the background of a video frame, and the background of each video frame is a solid color, then each video frame will be used as the image to be processed.

[0080] Step S204: Determine the pixel value of each pixel in the image to be processed;

[0081] In this embodiment, the three primary color values ​​of each pixel in the image to be processed can be obtained first, namely the red component value R, the green component value G, and the blue component value B. The pixel value of each pixel is calculated according to the preset weight value and the three primary color values ​​of each pixel, which can be expressed by the formula Z = -0.299*R + 0.413*G - 0.114*B, where Z represents the pixel value.

[0082] Alternatively, the image to be processed can be converted from RGB format to YUV format, and the green component value G and luminance Y of each pixel in the image to be processed can be obtained. The pixel value of each pixel can be calculated based on the green component value G and luminance Y, which can be expressed by the formula Z = G – Y.

[0083] Step S206: Determine the threshold value based on the pixel value of each pixel in the image to be processed; the threshold value represents the critical value that distinguishes the foreground and background in the image to be processed.

[0084] In this embodiment, since the background in the image to be processed is a solid color, there is a difference in the pixel value distribution between the background and the foreground. Therefore, the pixel value that can distinguish the foreground and the background in the image to be processed can be determined based on the pixel value of each pixel in the image to be processed. This pixel value can be regarded as the critical value.

[0085] Step S208: Obtain a mask image based on the pixel value and threshold value of each pixel in the image to be processed;

[0086] In this embodiment, the threshold value can distinguish between the foreground and the background in the image to be processed. The pixel value of each pixel in the image to be processed can be compared with the threshold value, and the region representing the foreground and the region representing the background can be determined based on the comparison result, thus obtaining the mask image.

[0087] Step S210: Replace the background of the image to be processed according to the mask image and the reference image to obtain the target image. The target image contains the foreground and target background of the image to be processed.

[0088] In this embodiment, based on the obtained mask image, the background of the image to be processed is replaced according to the mask image and the reference image, that is, the background in the image to be processed is replaced with the target background while retaining its foreground, thus obtaining a target image containing the foreground and target background of the image to be processed.

[0089] As can be seen from the above steps, by acquiring a reference image containing the target background and an image to be processed containing a solid-color background, and determining the pixel value of each pixel in the image to be processed; then determining a threshold value based on the pixel value of each pixel in the image to be processed, the threshold value represents the critical value for distinguishing the foreground and background in the image to be processed; next, obtaining a mask image based on the pixel value of each pixel in the image to be processed and the threshold value; finally, performing background replacement on the image to be processed based on the mask image and the reference image to obtain the target image, which contains the foreground and target background of the image to be processed. By automatically determining the threshold value, i.e., the parameter, for distinguishing the background and foreground, the parameters are adapted to the application scenario, thereby simplifying the operation, optimizing the effect of background replacement, and improving the processing efficiency of background replacement.

[0090] Optionally, for step S206 above, this embodiment of the invention provides a possible implementation method, please refer to [link to relevant documentation]. Figure 3 Step S206 includes the following steps:

[0091] Step S206-1: Normalize and smooth the pixel value of each pixel in the image to be processed to obtain the smoothed pixel value of each pixel in the image to be processed.

[0092] In this embodiment, the maximum and minimum pixel values ​​can be obtained from the pixel values ​​of all pixels in the image to be processed. Based on the maximum and minimum pixel values, the pixel value of each pixel in the image to be processed is normalized. For example, Zmax represents the maximum pixel value, Zmin represents the minimum pixel value, and Zn = (Z – Zmin) / (Zmax – Zmin), where Zn represents the normalized pixel value of the pixel.

[0093] Then, the normalized pixel value of each pixel in the image to be processed is smoothed. That is, the normalized pixel value of each pixel in the image to be processed is subjected to mean filtering with radius r=8, and then median filtering with radius r=2 to obtain the smoothed pixel value of each pixel in the image to be processed.

[0094] Step S206-3: According to preset parameters, perform histogram statistics on the smoothed pixel values ​​of all pixels in the image to be processed to obtain multiple reference values ​​and the frequency corresponding to each reference value;

[0095] In this embodiment, the preset parameter represents the total number of preset intervals, and each interval has a corresponding reference value. It should be understood that the preset parameter can be set according to actual application, and the implementation of this invention is not limited.

[0096] For example, with a preset parameter of 256, a histogram is performed on the smoothed pixel values ​​of all pixels in the image to be processed. This involves dividing the image into 256 even intervals and counting the number of pixels in each interval. The parameter value corresponding to each interval is obtained, and the number of pixels in each interval is used as the frequency of each reference value. This yields 256 reference values ​​and the frequency of each reference value.

[0097] Step S206-5: Obtain the reference values ​​corresponding to the two maximum frequencies to obtain two first reference values;

[0098] Step S206-7: Obtain the second reference value corresponding to the minimum frequency from the reference value interval composed of the two first reference values;

[0099] Step S206-9: Perform inverse normalization on the second reference value to obtain the threshold value.

[0100] In this embodiment, based on the frequency of each obtained reference value, two reference values ​​corresponding to the two maximum frequencies can be obtained from all reference values, that is, two first reference values ​​are obtained; wherein, the maximum frequency can be understood as the largest frequency in a local interval, that is, the frequency is greater than the frequency of the interval adjacent to its own interval; then the reference value interval formed by the two first reference values ​​is determined, and the reference value corresponding to the minimum frequency is obtained from this reference value interval, that is, the second reference value is obtained; then the second reference value is subjected to inverse normalization processing to obtain the threshold value.

[0101] For ease of understanding, this embodiment of the invention provides an example diagram using a preset parameter of 256 as an example. Please refer to... Figure 4 , Figure 4 (a) is the histogram obtained by performing histogram statistics on the normalized pixel values ​​of each pixel in the image to be processed, i.e. the histogram without smoothing, which can be called the histogram before smoothing. Figure 4 (b) The histogram obtained by performing histogram statistics on the smoothed pixel values ​​of all pixels in the image to be processed, i.e., the histogram after smoothing, can be called the smoothed histogram. For these two histograms, the horizontal and vertical axes represent 256 intervals, and the vertical axis represents the frequency.

[0102] according to Figure 4 As can be seen, the curves in the histogram after smoothing are much gentler than those in the histogram before smoothing, which allows for a more accurate determination of the critical point and the threshold value for distinguishing the foreground and background in the image to be processed.

[0103] exist Figure 4In (b), L1 and L2 are the points corresponding to the two maximum frequencies, i.e., the peak points. Based on the intervals to which points L1 and L2 belong, their corresponding reference values ​​are obtained, thus yielding the two first parameter values. From the reference value interval formed by these two first reference values, the reference value corresponding to the minimum frequency is obtained, i.e. Figure 4 (b) Point S is the reference value corresponding to the valley point, and the second reference value is obtained; then the second reference value is subjected to inverse normalization to obtain the threshold value.

[0104] As can be seen, by normalizing and smoothing the pixel values ​​of each pixel in the image to be processed, a smoothed pixel value for each pixel in the image to be processed is obtained. Then, histogram statistics are performed on the smoothed pixel values ​​of all pixels in the image to be processed according to preset parameters, resulting in multiple reference values ​​and the frequency corresponding to each reference value. Next, the reference values ​​corresponding to the two maximum frequencies are obtained, resulting in two first reference values. Then, the second reference value corresponding to the minimum frequency is obtained from the reference value interval formed by the two first reference values. Finally, the second reference value is denormalized to obtain the threshold value. By determining the threshold value that distinguishes the foreground and background in the image to be processed through smoothing and histogram statistics, the operation is simplified, the accuracy of obtaining the threshold value (i.e., the parameter) is improved, and the effect of background replacement is enhanced.

[0105] Optionally, for step S208 above, this embodiment of the invention provides a possible implementation method, please refer to [link to relevant documentation]. Figure 5 Step S208 includes the following steps:

[0106] Step S208-1: Each pixel in the image to be processed whose pixel value is less than a threshold value is taken as a first pixel, and a first region containing all first pixels is obtained; the first region represents the foreground in the image to be processed.

[0107] Step S208-3: Each pixel in the image to be processed whose pixel value is equal to or greater than a threshold value is taken as a second pixel, resulting in a second region containing all second pixels; the second region represents the background in the image to be processed.

[0108] Step S208-5: Obtain a mask image including the first region and the second region.

[0109] In this embodiment, based on the obtained threshold value, the pixel value of each pixel in the image to be processed can be compared with the threshold value; if the pixel value of a pixel is less than the threshold value, then the pixel is regarded as the first pixel, that is, the pixel value of the pixel is set to 1; if the pixel value of a pixel is equal to or greater than the threshold value, then the pixel is regarded as the second pixel, that is, the pixel value of the pixel is set to 0.

[0110] Then, the region formed by all the first pixels is obtained as the first region, which represents the foreground region in the image to be processed; the region formed by all the second pixels is obtained as the second region, which represents the background region in the image to be processed.

[0111] Optionally, for step S210 above, this embodiment of the invention provides a possible implementation method, please refer to [link to relevant documentation]. Figure 6 Step S210 includes the following steps:

[0112] Step S210-1: Optimize the mask image to obtain the optimized mask image;

[0113] Step S210-3: Based on the optimized mask image, perform image fusion on the reference image and the image to be processed to obtain the target image.

[0114] In this embodiment, in order to further improve the background replacement effect, the obtained mask image can be optimized to obtain an optimized mask image; then, the reference image and the image to be processed are image fused according to the optimized mask image.

[0115] The mask image includes a first region representing the foreground in the image to be processed and a second region representing the background in the image to be processed. Based on the first region in the optimized mask image and the image to be processed, a first image containing the foreground in the image to be processed can be obtained. Then, based on the second region in the optimized mask image and a reference image, a second image containing the target background in the reference image can be obtained. Finally, the target image is obtained by image fusion of the first and second images.

[0116] Optionally, for step S210-1 above, the present invention provides a possible implementation method.

[0117] Step S210-1-1: Perform image erosion on the mask image to obtain the boundary region;

[0118] In this embodiment, the mask image includes a first region and a second region. The mask image can be eroded according to a preset radius to obtain the boundary region where the first region and the second region meet.

[0119] Step S210-1-3: Based on each pixel in the boundary region and the HSV color model, obtain the HSV parameters corresponding to each pixel in the boundary region.

[0120] Step S210-1-5: Divide the pixels in the boundary region whose HSV parameters meet the preset conditions into the first region to obtain the undetermined mask image;

[0121] In this embodiment, the HSV color model is a color model that includes three attributes of an image: hue, saturation, and value. The three primary color values ​​of each pixel in the boundary region can be obtained, and then converted based on these values ​​and the HSV color model to obtain the corresponding HSV parameters for each pixel in the boundary region, namely, the color parameters H, saturation S, and value V.

[0122] Then, pixels in the boundary region whose HSV parameters satisfy a preset condition are obtained, resulting in each undetermined pixel. This preset condition is H∈[35 / 180,155 / 180], S≥43 / 255, and V≥46 / 255. Pixels satisfying this preset condition can be understood as pixels that should originally represent the foreground in the boundary region, but are actually classified as pixels representing the background. Therefore, these undetermined pixels are assigned to the first region representing the foreground, resulting in an undetermined mask image that includes a new first region and a new second region.

[0123] Step S210-1-7: Perform median filtering and guided filtering on the image to be masked to obtain the optimized mask image.

[0124] In this embodiment, based on the obtained undetermined mask image, median filtering is applied to reduce noise points, and guided filtering is applied to optimize the boundary, thus obtaining the optimized mask image.

[0125] To facilitate understanding, several example figures are provided in the embodiments of the present invention. Please refer to... Figures 7 to 9 ,in Figure 7 The image to be processed has a person in the foreground and a green curtain in the background. Following the steps described above, a threshold value is determined based on this image, and a mask image is obtained based on the threshold value. Figure 8 (c) The white portion of the mask image is the first region, and the black portion is the second region; then, the mask image is optimized according to steps S210-1-1 to S210-1-7 to obtain the optimized mask image as follows. Figure 8 (d)

[0126] according to Figure 8 As can be seen, the boundary lines in the optimized mask image are smoother than those in the unoptimized mask image, and the optimization also filters out the noise points (white parts) in the upper left corner of the unoptimized mask image. Then, based on the optimized mask image, image fusion is performed on the reference image and the image to be processed to obtain the target image, as shown below. Figure 9 .

[0127] Optionally, in order to further improve the processing speed of background replacement, this embodiment of the invention also provides an implementation method for determining a threshold value. After step S204, the image to be processed can be reduced to a thumbnail image of a preset resolution, and the pixel value of each pixel in the thumbnail image can be obtained; the threshold value is determined based on the pixel value of each pixel in the thumbnail image.

[0128] In this embodiment, after determining the pixel value of each pixel in the image to be processed, the image to be processed can be reduced to a thumbnail image of a preset resolution, or the image to be processed can be reduced to obtain a thumbnail image according to a preset comparison; then, based on the pixel value of each pixel in the image to be processed, the pixel value of each pixel in the thumbnail image is obtained; and then a threshold value is determined based on the pixel value of each pixel in the thumbnail image.

[0129] Understandably, determining the threshold value based on the pixel value of each pixel in the thumbnail image is similar to determining the threshold value based on the pixel value of each pixel in the image to be processed. Specifically, the pixel value of each pixel in the thumbnail image is normalized and smoothed to obtain a smoothed pixel value; histogram statistics are performed on the smoothed pixel values ​​of all pixels in the thumbnail image according to preset parameters to obtain multiple reference values ​​and the frequency corresponding to each reference value; the reference values ​​corresponding to the two maximum frequencies are obtained to obtain two first reference values; the second reference value corresponding to the minimum frequency is obtained from the reference value interval formed by the two first reference values; and the threshold value is obtained by inverse normalizing the second reference value.

[0130] Optionally, when the application scenario is to replace the background of the live broadcast screen, the present invention provides a possible implementation method for the above step S202: in response to the user's selection operation, a reference image containing the target background is obtained; in response to the user's live broadcast start operation, each screen to be broadcast is captured in real time and each screen to be broadcast is used as an image to be processed.

[0131] For ease of understanding, an example diagram is provided in this embodiment of the invention. Please refer to... Figure 10 , Figure 10 The interactive interface (e1) displays multiple background template images. The user can select one of the background template images, and the selected background template image becomes the reference image. Alternatively, the user can click the "Try Custom" button to select a local image from the electronic device as the reference image.

[0132] After obtaining the reference image, it is displayed as follows. Figure 10(e2) is an interactive interface where the images to be played are captured by the camera module of the electronic device. Users can perform live streaming start operations, such as clicking the "Start Live Stream" button, which will then use each image to be played captured in real time by the camera module as an image to be processed.

[0133] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of the background replacement device is given below. Please refer to... Figure 11 , Figure 11 This is a functional block diagram of a background replacement device 300 provided in an embodiment of the present invention. It should be noted that the background replacement device 300 provided in this embodiment has the same basic principle and technical effects as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. The background replacement device 300 includes:

[0134] The acquisition module 310 is used to acquire a reference image containing the target background and an image to be processed containing a solid color background;

[0135] Processing module 330 is used to determine the pixel value of each pixel in the image to be processed;

[0136] The threshold value is determined based on the pixel value of each pixel in the image to be processed; the threshold value represents the critical value that distinguishes the foreground and background in the image to be processed.

[0137] The mask image is obtained by using the pixel value and threshold value of each pixel in the image to be processed;

[0138] The replacement module 350 is used to perform background replacement on the image to be processed based on the mask image and the reference image to obtain the target image, which contains the foreground and target background of the image to be processed.

[0139] Optionally, the processing module 330 is further configured to: normalize and smooth the pixel value of each pixel in the image to be processed to obtain the smoothed pixel value of each pixel in the image to be processed; perform histogram statistics on the smoothed pixel values ​​of all pixels in the image to be processed according to preset parameters to obtain multiple reference values ​​and the frequency corresponding to each reference value; obtain the reference values ​​corresponding to the two maximum frequencies to obtain two first reference values; obtain the second reference value corresponding to the minimum frequency from the reference value interval formed by the two first reference values; and perform inverse normalization on the second reference value to obtain a threshold value.

[0140] Optionally, the processing module 330 is further configured to: take each pixel in the image to be processed whose pixel value is less than a threshold value as a first pixel, and obtain a first region containing all the first pixels; the first region represents the foreground in the image to be processed; take each pixel in the image to be processed whose pixel value is equal to or greater than a threshold value as a second pixel, and obtain a second region containing all the second pixels; the second region represents the background in the image to be processed; and obtain a mask image including the first region and the second region.

[0141] Optionally, the replacement module 350 is also used to: optimize the mask image to obtain an optimized mask image; and perform image fusion on the reference image and the image to be processed based on the optimized mask image to obtain a target image.

[0142] Optionally, the replacement module 350 is further configured to: perform image erosion on the mask image to obtain a boundary region; obtain the HSV parameters corresponding to each pixel in the boundary region based on each pixel in the boundary region and the HSV color model; divide the pixels in the boundary region whose HSV parameters meet the preset conditions into a first region to obtain a mask image to be determined; and perform median filtering and guided filtering on the mask image to be determined to obtain an optimized mask image.

[0143] Optionally, the processing module 330 is further configured to: reduce the image to be processed to a thumbnail image of a preset resolution, and obtain the pixel value of each pixel in the thumbnail image; and determine a threshold value based on the pixel value of each pixel in the thumbnail image.

[0144] Optionally, the acquisition module 310 is also used to: in response to the user's selection operation, acquire a reference image containing the target background; in response to the user's live broadcast start operation, capture each screen to be broadcast in real time and treat each screen to be broadcast as an image to be processed.

[0145] This invention also provides an electronic device, including a processor 120 and a memory 130. The memory 130 stores a computer program, and when the processor executes the computer program, it implements the background replacement method disclosed in the above embodiments.

[0146] This invention also provides a storage medium storing a computer program that, when executed by a processor 120, implements the background replacement method disclosed in this invention.

[0147] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0148] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0149] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A background replacement method, characterized in that, The method includes: Obtain a reference image containing the target background and an image to be processed containing a solid color background; Determine the pixel value of each pixel in the image to be processed; The pixel values ​​of each pixel in the image to be processed are normalized and smoothed to obtain the smoothed pixel values ​​of each pixel in the image to be processed. According to preset parameters, histogram statistics are performed on the smoothed pixel values ​​of all pixels in the image to be processed to obtain multiple reference values ​​and the frequency corresponding to each reference value. Obtain the reference values ​​corresponding to the two maximum frequencies to obtain two first reference values; Obtain the second reference value corresponding to the minimum frequency from the reference value interval formed by the two first reference values; The second reference value is denormalized to obtain a threshold value; the threshold value represents the critical value that distinguishes the foreground and background in the image to be processed. A mask image is obtained based on the pixel value of each pixel in the image to be processed and the threshold value; The background of the image to be processed is replaced according to the mask image and the reference image to obtain a target image, wherein the target image contains the foreground and the target background of the image to be processed.

2. The method according to claim 1, characterized in that, The step of obtaining a mask image based on the pixel value of each pixel in the image to be processed and the threshold value includes: Each pixel in the image to be processed whose pixel value is less than the threshold value is taken as a first pixel, and a first region containing all first pixels is obtained; the first region represents the foreground in the image to be processed. Each pixel in the image to be processed whose pixel value is equal to or greater than the threshold value is taken as a second pixel, resulting in a second region containing all second pixels; the second region represents the background in the image to be processed. Obtain a mask image that includes the first region and the second region.

3. The method according to claim 1, characterized in that, The step of performing background replacement on the image to be processed based on the mask image and the reference image to obtain the target image includes: The mask image is optimized to obtain an optimized mask image; Based on the optimized mask image, the reference image and the image to be processed are fused to obtain the target image.

4. The method according to claim 3, characterized in that, The mask image includes a first region representing the foreground in the image to be processed and a second region representing the background in the image to be processed; The step of optimizing the mask image to obtain an optimized mask image includes: The mask image is eroded to obtain the boundary region; Based on each pixel in the boundary region and the HSV color model, the HSV parameters corresponding to each pixel in the boundary region are obtained; The pixels in the boundary region whose HSV parameters meet the preset conditions are assigned to the first region to obtain the undetermined mask image; The undetermined mask image is subjected to median filtering and guided filtering to obtain the optimized mask image.

5. The method according to claim 1, characterized in that, After the step of determining the pixel value of each pixel in the image to be processed, the method further includes: The image to be processed is reduced to a thumbnail image of a preset resolution, and the pixel value of each pixel in the thumbnail image is obtained; The threshold value is determined based on the pixel value of each pixel in the thumbnail image.

6. The method according to claim 1, characterized in that, The step of obtaining a reference image containing the target background and an image to be processed containing a solid color background includes: In response to the user's selection, obtain a reference image containing the target background; In response to the user's live stream start operation, the system captures each frame to be broadcast in real time and treats each frame as an image to be processed.

7. A background replacement device, characterized in that, The device includes: The acquisition module is used to acquire a reference image containing the target background and an image to be processed containing a solid color background; The processing module is used to determine the pixel value of each pixel in the image to be processed; perform normalization and smoothing processing on the pixel value of each pixel in the image to be processed to obtain the smoothed pixel value of each pixel in the image to be processed; perform histogram statistics on the smoothed pixel values ​​of all pixels in the image to be processed according to preset parameters to obtain multiple reference values ​​and the frequency corresponding to each reference value; obtain the reference values ​​corresponding to the two maximum frequencies to obtain two first reference values; obtain the second reference value corresponding to the minimum frequency from the reference value interval formed by the two first reference values; perform inverse normalization processing on the second reference value to obtain a threshold value; the threshold value represents the critical value for distinguishing the foreground and background in the image to be processed; and obtain a mask image based on the pixel value of each pixel in the image to be processed and the threshold value. The replacement module is used to perform background replacement on the image to be processed based on the mask image and the reference image to obtain a target image, wherein the target image includes the foreground and the target background in the image to be processed.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor, when executing the computer program, implements the method of any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 6.

Images