Image background replacement method, system, electronic device and storage medium

Abstract

The present disclosure provides an image background replacement method, which is applied to the field of financial technology or other fields, including: obtaining the foreground prediction probability of each pixel in the first image; selecting points whose foreground prediction probability is close to the characteristic value as uncertain points; fusing the feature maps of different scales of the uncertain points to obtain a foreground image and an alpha mask with clear boundaries; using the background image to be replaced to synthesize with the foreground image and the alpha mask to obtain a second image under a new background. In the foreground and background segmentation, the method of the present disclosure performs edge contour processing on the edge portion between the foreground and the background, retains more details, and obtains a background replacement image with higher segmentation accuracy. The present disclosure also provides an image background replacement system, an electronic device, a computer-readable storage medium, and a program product.

Description

Technical Field

[0001] The present disclosure relates to the technical field of financial technology, and specifically to an image background replacement method, system, electronic device and storage medium. Background Art

[0002] In response to the need for epidemic prevention and control, the bank's remote video customer service has become an important business for the bank. Traditional video customer service is just a simple video call. When users consider their personal privacy and do not want the customer service to see their background, this service is difficult to meet user needs. Therefore, replacing the background of the user's video is an improvement that the bank's video customer service needs to make, and the portrait segmentation algorithm can achieve this function.

[0003] In the current portrait segmentation algorithms, there are often problems such as low segmentation accuracy and poor detail processing. The result of portrait segmentation is just an outline, and the details at the edge of the portrait are easily over-segmented or under-segmented. Therefore, we consider using an improved background-matting algorithm to achieve portrait segmentation, and put the segmented portrait on a new background to achieve the purpose of background replacement. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the above problems, the present disclosure provides an image background replacement method, system, electronic device and storage medium, which are used to at least partially solve the technical problems of low edge segmentation accuracy and poor detail processing in traditional image background replacement.

[0006] (II) Technical solution

[0007] On the one hand, the present disclosure provides an image background replacement method, including: obtaining a foreground prediction probability for each pixel in a first image; selecting points whose foreground prediction probabilities are close to characteristic values ​​as uncertain points; fusing feature maps of uncertain points at different scales to obtain a foreground image and an alpha mask with clear boundaries; and synthesizing a background image to be replaced with the foreground image and the alpha mask to obtain a second image under a new background.

[0008] Furthermore, obtaining the foreground prediction probability of each pixel in the first image includes: obtaining the first image; inputting the first image into a convolutional neural network to obtain a feature map of the image; and inputting the feature map into a pyramid pooling module to obtain the foreground prediction probability of each pixel.

[0009] Furthermore, inputting the feature map into the pyramid pooling module includes: constructing a multi-layer feature pyramid, inputting the feature map into each layer of the pyramid for pooling at different scales; and obtaining the foreground prediction probability of each pixel after upsampling, full connection layer, and activation function processing.

[0010] Furthermore, selecting points whose foreground prediction probabilities are close to the characteristic values ​​as uncertain points also includes: performing a deconvolution process to obtain a roughly segmented feature map.

[0011] Furthermore, selecting points whose foreground prediction probabilities are close to the eigenvalues ​​as uncertain points includes: upsampling the boundaries of the coarsely segmented feature maps by bilinear interpolation; obtaining the foreground prediction probabilities of pixel points at the boundaries; and selecting points whose probabilities are close to the eigenvalues ​​as uncertain points.

[0012] Furthermore, the foreground prediction probability comes from the formula of the cutout algorithm:

[0013] I=αF+(1-α)B

[0014] Where I is the image, F is the foreground of I, α is the foreground prediction probability of the pixel, and B is the background of I.

[0015] Furthermore, the points whose foreground prediction probability is close to the eigenvalue are selected as the uncertain points with the eigenvalue of 0.5.

[0016] Furthermore, the feature maps of different scales of the uncertain points are fused to obtain a foreground map and an alpha mask with clear boundaries, including: inputting the feature maps into each layer of the pyramid for pooling at different scales to obtain feature maps of different scales; and fusing the feature maps of different scales of the uncertain points to obtain a feature map with clear boundaries.

[0017] Furthermore, fusing the feature maps of different scales of the uncertain points also includes: decoding to obtain a foreground map and an alpha mask with clear boundaries.

[0018] Furthermore, decoding to obtain a foreground image with clear boundaries includes: bilinear difference upsampling, convolution, BN, and ReLU processing to obtain a predicted feature map; bilinear difference upsampling, convolution, BN, ReLU, mirror padding, and convolution are performed on the predicted feature map and the feature map with clear boundaries to obtain a foreground image with clear boundaries.

[0019] Furthermore, decoding to obtain an alpha mask with clear boundaries includes: bilinear difference upsampling, convolution, BN, ReLU processing for decoding; mirror padding, convolution, Tanh processing to obtain an alpha mask with clear boundaries.

[0020] Furthermore, synthesizing the background image to be replaced with the foreground image and the alpha mask includes: processing the alpha mask and the foreground image to obtain the foreground part of the second image; processing the alpha mask and the background image to be replaced to obtain the background part of the second image; and superimposing the foreground part and the background part to obtain the second image.

[0021] On the other hand, the present disclosure provides a method for replacing the background of a video image, including: acquiring a first video image in a current video; and replacing the background of the first video image according to the aforementioned image background replacement method to obtain a second video image under a new background.

[0022] Furthermore, obtaining the second video image under the new background includes: using rtmp transmission protocol to transmit real-time video.

[0023] In another aspect, the present disclosure provides an image background replacement system, including: an image acquisition module, used to obtain the foreground prediction probability of each pixel in a first image; a selection module, used to select points whose foreground prediction probabilities are close to characteristic values ​​as uncertain points; a fusion module, used to fuse feature maps of different scales of the uncertain points to obtain a foreground image and an alpha mask with clear boundaries; and a synthesis module, used to synthesize the background image to be replaced with the foreground image and the alpha mask to obtain a second image under a new background.

[0024] In another aspect, the present disclosure provides a video image background replacement system, including: a video acquisition module, used to acquire a first video image in a current video; a background replacement module, used to replace the background of the first video image according to the aforementioned image background replacement method to obtain a second video image under a new background.

[0025] Another aspect of the present disclosure provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor executes the image background replacement method or video image background replacement method as described above.

[0026] Another aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image background replacement method or video image background replacement method as described above is implemented.

[0027] Another aspect of the present disclosure provides a computer program product, including a computer program, which implements the aforementioned image background replacement method or video image background replacement method when executed by a processor.

[0028] (III) Beneficial effects

[0029] The present disclosure provides an image background replacement method, system, electronic device and storage medium. In the process of foreground and background segmentation, the edge part between the foreground and the background is processed by edge contour processing, and the uncertain points of the edge contour are upsampled and feature fused. Thus, the features of the uncertain points include both the features of detecting large targets and the features of detecting details, and more details are retained, thereby obtaining a background replacement image with higher segmentation accuracy. The same method can also be applied to background replacement in videos, and a background replacement video image with higher segmentation accuracy is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] For a more complete understanding of the present disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, in which:

[0031] Figure 1 A schematic diagram of an application scenario of the image background replacement method according to an embodiment of the present disclosure is schematically shown;

[0032] Figure 2 A scene diagram schematically shows a specific application example of the image background replacement method according to an embodiment of the present disclosure;

[0033] Figure 3 A flowchart of an image background replacement method according to an embodiment of the present disclosure is schematically shown;

[0034] Figure 4 A flowchart of a method for obtaining a foreground prediction probability of each pixel in a first image according to an embodiment of the present disclosure is schematically shown;

[0035] Figure 5 A flowchart of a method for inputting a feature map into a pyramid pooling module according to an embodiment of the present disclosure is schematically shown;

[0036] Figure 6 The structure diagram of the pyramid pooling module according to the embodiment of the present disclosure is schematically shown;

[0037] Figure 7 A flowchart of a method for selecting a point whose prospect prediction probability is close to a characteristic value as an uncertain point according to an embodiment of the present disclosure is schematically shown;

[0038] Figure 8 A flowchart of a method for fusing feature maps of different scales of an uncertain point according to an embodiment of the present disclosure is schematically shown;

[0039] Fig. 9 A flowchart of a method for synthesizing a background image to be replaced with a foreground image and an alpha mask according to an embodiment of the present disclosure is schematically shown;

[0040] Fig.10Schematically shows a flow chart of a complete method for image background replacement according to an embodiment of the present disclosure;

[0041] Fig.11 Schematically shows a flow chart of edge contour processing according to an embodiment of the present disclosure;

[0042] Fig.12 A schematic diagram schematically shows an upsampling step according to an embodiment of the present disclosure;

[0043] Fig.13 A schematic diagram schematically shows a video background image replacement according to an embodiment of the present disclosure;

[0044] Fig.14 A schematic diagram schematically shows real-time video transmission according to an embodiment of the present disclosure;

[0045] Fig.15 A flowchart of real-time video transmission according to an embodiment of the present disclosure is schematically shown;

[0046] Fig.16 A block diagram of an image background replacement system according to an embodiment of the present disclosure is schematically shown;

[0047] Fig.17 The block diagram of the video image background replacement system according to the embodiment of the present disclosure is schematically shown;

[0048] Fig.18 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0049] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0050] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0051] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0052] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0053] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some boxes or combinations thereof in the block diagrams and / or flow charts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions can create a device for implementing the functions / operations described in these block diagrams and / or flow charts when executed by the processor. The technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer program product on a computer-readable storage medium storing instructions, which can be used by an instruction execution system or used in conjunction with an instruction execution system.

[0054] The embodiments of the present disclosure provide an image background replacement method, system, electronic device and storage medium. During the image segmentation process, the edge portion between the foreground and the background is processed with edge contours to retain more details, thereby obtaining a background replacement image with higher segmentation accuracy.

[0055] Figure 1 The exemplary system architecture 100 that can be applied to the image background replacement method according to the embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0056] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0057] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as camera application, photo application, web browser application, search application, instant messaging tool, social platform software, etc. (only as examples).

[0058] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0059] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0060] It should be noted that the image background replacement method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the system for the image background replacement method provided in the embodiment of the present disclosure can generally be set in the server 105. The image background replacement method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the system for the image background replacement method provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0061] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0062] Figure 2 A scene diagram schematically shows a specific application example of the image background replacement method according to an embodiment of the present disclosure.

[0063] The terminal device 101, 102 or 103 can transmit the image 201 stored in itself or taken from the outside to the server 105 via the network 104. The server 105 extracts its foreground image 202 and can replace the image background based on the extracted foreground image to obtain an image 203 with the background replaced. The image 203 is transmitted to other network terminals via the network 104 and displayed or played on the other network terminals.

[0064] In order to improve the effect of edge processing in portrait segmentation, the algorithm usually requires the user to manually provide a sketch (Scratch) or a ternary map (Trimap) for auxiliary processing. A similar strategy is used in the background-matting algorithm, except that the background-matting algorithm uses a background without foreground as input. It is very inconvenient in actual use. On the one hand, it is necessary to manually shoot a background image without foreground, which will bring great inconvenience to real-time processing. On the other hand, although the two pictures with and without foreground are taken under exactly the same conditions, this condition is too harsh. Therefore, the present invention uses an edge optimization method applied to background-matting to solve the problem that the traditional method requires additional background pictures to be taken.

[0065] Figure 3 The flowchart of the image background replacement method according to the embodiment of the present disclosure is schematically shown.

[0066] like Figure 3 As shown, the image background replacement method includes:

[0067] In operation S1, a foreground prediction probability of each pixel in a first image is obtained.

[0068] The extraction of the foreground image is an important step in the process of image background replacement. In this extraction method, it is first necessary to obtain the foreground prediction probability of each pixel in the first image. When the probability of a certain pixel is closer to 1, it is considered to be the foreground, and when it is closer to 0, it is considered to be the background, thereby simply distinguishing the pixels of the image into foreground pixels and background pixels. However, this method is prone to leaving a background color band at the edge of the foreground, making the edge contour of the new background image finally fused unnatural, affecting the quality of the image with the background replaced.

[0069] In operation S2, points whose foreground prediction probabilities are close to the characteristic values ​​are selected as uncertain points.

[0070] After obtaining the foreground prediction probability of each pixel point, the pixel points of the foreground edge part are selected to further refine the points on the edge; the probability of selecting pixel points is neither close to 1 nor close to 0, but close to a certain characteristic value as uncertain points.

[0071] In operation S3, the feature maps of different scales of the uncertain points are fused to obtain a foreground map with clear boundaries and an alpha mask.

[0072] Before obtaining the foreground prediction probability of a pixel point, features of different scales have been extracted for each point. After selecting points that can neither be determined as foreground pixels nor as background pixels, that is, after determining the uncertain points, the feature maps of different scales of the uncertain points are further fused. Such features include both features for detecting large targets and features for detecting details. The foreground map and alpha mask are then predicted through the features of the uncertain points. At this time, the boundaries of the obtained foreground map and alpha mask are clearer, ensuring the quality of the image with image background replacement.

[0073] In operation S4, the background image to be replaced is synthesized with the foreground image and the alpha mask to obtain a second image under a new background.

[0074] By using the alpha mask, the pixel value of the pixel point in the foreground image is used as the pixel value of the pixel point at the position corresponding to the pixel point in the foreground image in the image after replacing the background; and the pixel value of the pixel point at the position corresponding to the background pixel point in the background image to be replaced is used as the pixel value of the pixel point at the position corresponding to the background pixel point in the image after replacing the background.

[0075] Figure 4 A flowchart of a method for obtaining a foreground prediction probability of each pixel in a first image according to an embodiment of the present disclosure is schematically shown.

[0076] like Figure 4 As shown, the method for obtaining the foreground prediction probability of each pixel in the first image includes:

[0077] In operation S11, a first image is acquired.

[0078] First, a first image to be replaced with a background is obtained. The first image is not specifically limited here, and can be a stored image or an image captured in real time.

[0079] In operation S12, the first image is input into a convolutional neural network to obtain a feature map of the image.

[0080] The first image is introduced into the CNN network to obtain a preliminary feature map with a large uncertainty area.

[0081] In operation S13, the feature map is input into a pyramid pooling module to obtain a foreground prediction probability of each pixel.

[0082] By utilizing information sets based on different regions, through the pyramid pooling module, the Pyramid Scene Parsing Network (PSPNet) is used to exert the ability to parse contextual information.

[0083] Figure 5 A flowchart of a method for inputting a feature map into a pyramid pooling module according to an embodiment of the present disclosure is schematically shown.

[0084] like Figure 5 As shown, the method of inputting the feature map into the pyramid pooling module includes:

[0085] In operation S131 , a multi-layer feature pyramid is constructed.

[0086] See also Figure 6 ,The pyramid pooling structure integrates features at four different scales and divides the input feature map into 1×1, 2×2, 3×3, 6×6 different sub-regions.

[0087] In operation S132, the feature map is input into each layer of the pyramid for pooling at different scales.

[0088] Each sub-region is pooled, and finally the pooled single feature maps containing position information are combined. Different levels in the pyramid pooling module output feature maps of different scales. In order to maintain the weight of the global feature, a 1x1 convolution kernel is used after each pyramid level. When the dimension of a certain level is n, the dimension of the semantic feature can be reduced to 1 / n of the original feature.

[0089] In operation S133, after upsampling, full connection layer, and activation function processing, the foreground prediction probability of each pixel is obtained.

[0090] Then, the low-dimensional feature map is directly upsampled through bilinear interpolation to make it the same scale as the original feature map. The feature maps of different levels are then spliced ​​into the final pyramid pooling global feature. The size of each level in the pyramid pooling structure can be modified, which is related to the size of the feature map of the input pyramid pooling layer. This structure uses different pooling kernels to extract features of different sub-regions. Finally, after processing with the activation function, the foreground prediction probability of each pixel is obtained.

[0091] On the basis of the above embodiment, selecting a point whose foreground prediction probability is close to a characteristic value as an uncertain point further includes: performing a deconvolution process to obtain a roughly segmented feature map.

[0092] After the activation function is processed, the deconvolution layer is input for deconvolution processing CONV, such as Figure 6As shown, a rough segmentation feature map is obtained.

[0093] Figure 7 A flowchart of a method for selecting points whose prospect prediction probabilities are close to characteristic values ​​as uncertain points according to an embodiment of the present disclosure is schematically shown.

[0094] like Figure 7 As shown in FIG. 1 , the method of selecting points whose prospect prediction probabilities are close to the characteristic values ​​as uncertain points includes:

[0095] In operation S21, the boundary of the roughly segmented feature map is upsampled using bilinear interpolation.

[0096] After obtaining the rough segmentation feature map, the rough edge contour needs to be further processed. First, the boundary of the rough segmentation feature map is upsampled using bilinear interpolation. If the pixel distribution in the image is regarded as a grid, the density of the grid is higher and the resolution is higher after this operation.

[0097] In operation S22, the foreground prediction probability of the pixel points at the boundary is obtained.

[0098] After being processed by the activation function, the foreground prediction probability of each pixel is obtained, so that the foreground prediction probability of the pixel at the boundary can be obtained.

[0099] In operation S22, points whose probabilities are close to the eigenvalues ​​are selected as uncertain points.

[0100] We know that the probability of predicting a pixel point at the boundary of an image segmentation is closer to 1 and is considered to be the foreground, while the probability of predicting a pixel point closer to 0 is considered to be the background; therefore, the pixel point whose probability is neither close to 1 nor close to 0 but close to a certain feature value is selected as the uncertain point.

[0101] Based on the above embodiment, the foreground prediction probability comes from the formula of the cutout algorithm:

[0102] I=αF+(1-α)B

[0103] Where I is the image, F is the foreground of I, α is the foreground prediction probability of the pixel, and B is the background of I.

[0104] This formula embodies the essence of portrait segmentation. When we know α, we can segment the foreground and background from an image.

[0105] On the basis of the above embodiment, a point whose prospect prediction probability is close to the characteristic value is selected as the uncertain point whose characteristic value is 0.5.

[0106] During image processing, most points with foreground prediction probabilities close to 0.5 are ignored, and such uncertain points will directly affect the effect of boundary segmentation; therefore, we select multiple points with probabilities close to 0.5 on a denser grid; a foreground prediction probability close to 0.5 is basically considered to be the edge of the foreground image.

[0107] Figure 8 The flowchart schematically shows a method for fusing feature maps of different scales of uncertain points to obtain a foreground map with clear boundaries and an alpha mask according to an embodiment of the present disclosure.

[0108] like Figure 8 As shown, the method of fusing feature maps of different scales of uncertain points to obtain a foreground map with clear boundaries and an alpha mask includes:

[0109] In operation S1321, the feature map is input into each layer of the pyramid for pooling at different scales to obtain feature maps of different scales.

[0110] In step S132, the feature maps have been input into each layer of the pyramid for pooling at different scales; here, feature maps of different scales of uncertain points are first obtained.

[0111] In operation S1322, feature maps of different scales of the uncertain points are fused to obtain a feature map with clear boundaries.

[0112] The feature maps of uncertain points at different scales are fused as the features of the uncertain points. Such features include both the features for detecting large targets and the features for detecting details, thus retaining more boundary details.

[0113] On the basis of the above embodiment, fusing feature maps of different scales of uncertain points further includes: decoding to obtain a foreground image and an alpha mask with clear boundaries.

[0114] The processes before edge contour processing can be regarded as encoding processes. Here, a decoding process is performed to restore the image and obtain the foreground image and alpha mask respectively.

[0115] On the basis of the above embodiment, decoding to obtain a foreground image with clear boundaries includes: bilinear difference upsampling, convolution, BN, and ReLU processing to obtain a predicted feature map; bilinear difference upsampling, convolution, BN, ReLU, mirror padding, and convolution are performed on the predicted feature map and the feature map with clear boundaries to obtain a foreground image with clear boundaries.

[0116] Foreground prediction branch: In this branch, it is first decoded by a set of 3 residual block decoders to obtain the corresponding feature map. The first part of the decoding uses the previously obtained feature map as input, and obtains the predicted feature map after a set of bilinear difference upsampling, convolution, BN, ReLU operations. The second part of the decoding uses the result of splicing the predicted feature map and the original feature map, and obtains the foreground image after bilinear difference upsampling, convolution, BN, ReLU, mirror padding, and convolution.

[0117] On the basis of the above embodiment, decoding to obtain an alpha mask with clear boundaries includes: bilinear difference upsampling, convolution, BN, ReLU processing for decoding; mirror padding, convolution, Tanh processing to obtain an alpha mask with clear boundaries.

[0118] α prediction branch: Similar to the foreground prediction branch, it is first decoded by a set of 3 residual block decoders, then decoded by two sets of bilinear difference, convolution, BN, ReLU operations, and finally obtained after a set of mirror padding, convolution and Tanh. The reason for using Tanh is that the value of each pixel of the alpha matte needs to be between 0 and 1.

[0119] Fig. 9 The flowchart of the method for synthesizing the background image to be replaced with the foreground image and the alpha mask according to the embodiment of the present disclosure is schematically shown.

[0120] like Fig. 9 As shown, the method of synthesizing the background image to be replaced with the foreground image and the alpha mask includes:

[0121] In operation S41, the alpha mask and the foreground image are processed to obtain a foreground portion of the second image.

[0122] In operation S42, the alpha mask and the background image to be replaced are processed to obtain a background portion of the second image.

[0123] In operation S43, the foreground portion and the background portion are superimposed to obtain a second image.

[0124] The foreground part of the fused image is obtained by multiplying the α mask with the corresponding pixel values ​​of the foreground image, and the background part of the fused image is obtained by multiplying the α mask processed by (1-a) with the corresponding pixel values ​​of the new image. The foreground part and the background part are superimposed to obtain the fused image.

[0125] The present disclosure also provides a video image background replacement method, comprising: obtaining a first video image in a current video; replacing the background of the first video image according to the aforementioned image background replacement method to obtain a second video image under a new background.

[0126] The aforementioned processing method can also be applied to video image background replacement. By acquiring a video frame in a video and performing image replacement, a second video image under a new background can be obtained.

[0127] Based on the above embodiment, obtaining the second video image under the new background includes: using the rtmp transmission protocol to transmit the real-time video.

[0128] In order to achieve real-time video background replacement, the RTMP transmission protocol is used to transmit the video stream.

[0129] The video background replacement method disclosed in the present invention overcomes the shortcomings of traditional portrait segmentation algorithms and retains more portrait details; it allows users to conveniently replace image backgrounds or replace their own backgrounds to protect personal privacy when providing video customer service, while better retaining the portrait part and improving user experience.

[0130] The following further illustrates each step of this method by taking video background image replacement as an example.

[0131] See also Fig.10 .

[0132] Step 1001, first, obtain a video frame in a video;

[0133] Step 1002, inputting the video frame image into a convolutional neural network to obtain a feature map of the image;

[0134] Step 1003, inputting the feature map into a pyramid pooling module for pooling at different scales;

[0135] Step 1004, after upsampling, full connection layer, and activation function processing, the foreground prediction probability of each pixel is obtained;

[0136] The low-dimensional feature map is directly upsampled through bilinear interpolation to make it the same scale as the original feature map. The feature maps of different levels are then spliced ​​into the final pyramid pooling global feature. The size of each level in the pyramid pooling structure can be modified, which is related to the size of the feature map of the input pyramid pooling layer. This structure uses different pooling kernels to extract features of different sub-regions. Finally, after processing with the activation function, the foreground prediction probability of each pixel is obtained.

[0137] Step 1005, inputting the deconvolution layer to obtain a rough segmentation feature map 1006;

[0138] The traditional semantic segmentation network will obtain a feature map of a certain resolution after a series of convolution and pooling. Figure 1The general size is 1 / 8, 1 / 16 or 1 / 32 of the original image. The smaller the feature map, the larger the receptive field, and the more suitable it is for detecting large targets. Conversely, the smaller the feature map, the more suitable it is for detecting small targets. By predicting the points on the feature map, we have the category label and know that a certain pixel belongs to a certain category. Then, we restore it to the original image size through a certain upsampling method, so that we can get the semantic segmentation result of the original image.

[0139] Step 1007, performing edge contour processing on the roughly segmented feature map 1006 to obtain a feature map with clear boundaries;

[0140] In this embodiment, deeplabV3 is first used to extract features from the original image and predict a rough segmentation alpha mask, and a 14*14 feature map is extracted. The edge part of this segmentation result is not very good. The original background-matting algorithm's strategy for edge optimization is to selectively retain the foreground and background in the original image, the background image without foreground, and the alpha mask through a selection unit. Although the purpose of edge optimization can be achieved, the background image without foreground needs to be input. In this disclosure, bilinear interpolation is used to upsample the previously predicted rough boundary. If the pixel distribution in the image is regarded as a grid, the density of the grid is higher and the resolution is higher after this operation. Please refer to Fig.12 . We know that the probability of predicting a certain pixel point in the boundary part of the image segmentation is closer to 1 and is considered as the foreground, and closer to 0 and is considered as the background. A large part of the points with probabilities close to 0.5 are ignored. Such uncertain points will directly affect the effect of boundary segmentation. Therefore, we select multiple points with probabilities close to 0.5 on a denser grid. Because features of different scales have been extracted for each point in deeplabv3, 1 / 4 and 1 / 16 of the original feature map are fused as the features of the uncertain points. Such features include both the features for detecting large targets and the features for detecting details.

[0141] Step 1008, decoding, obtaining a foreground image 1009 with clear boundaries and an alpha mask 1010;

[0142] Next, the foreground and alpha mask are predicted by the features of the uncertain points. The foreground prediction branch is first decoded by a set of 3 residual block decoders to obtain the corresponding feature map. The first part of the decoding uses the previously obtained feature map as input, and obtains the predicted feature map after a set of bilinear difference upsampling, convolution, BN, ReLU operations. The second part of the decoding uses the result of splicing the predicted feature map and the original feature map, and obtains the foreground map after bilinear difference upsampling, convolution, BN, ReLU, mirror padding, and convolution. Similarly, the alpha prediction branch is first decoded by a set of 3 residual block decoders, and then decoded by two sets of bilinear difference, convolution, BN, ReLU operations, and finally obtains the final predicted alpha matte after a set of mirror padding, convolution and Tanh. The reason for using Tanh is that the value of each pixel of the alphamatte needs to be between 0 and 1.

[0143] Step 1011, synthesize the background image to be replaced with the foreground image and the alpha mask to obtain a video image under a new background. For steps 901 to 911, please refer to Fig.13 .

[0144] The edge contour processing 907 specifically includes the following steps, see Fig.11 .

[0145] Step 1101, upsampling the boundary of the roughly segmented feature map by bilinear interpolation;

[0146] Step 1102, predicting the probability of a certain pixel point on the boundary;

[0147] Step 1103, selecting a point with a probability close to 0.5 as an uncertain point;

[0148] Step 1104, fusing feature maps at different scales in the pyramid pooling module of the uncertain point;

[0149] Step 1105, obtaining a feature map with clear boundaries.

[0150] After obtaining the video image under the new background, real-time video transmission is required. Figures 14-15 .

[0151] The general process of real-time video transmission is as follows Fig.14 As shown, the rtmp transmission protocol is used to transmit the video. The source station is the server, which is built using nginx. In order to achieve real-time video background replacement, a processing unit needs to be added to the server side. The entire flow chart is as follows Fig.15As shown, 1. Client: the front-end image acquisition unit, which collects video through the camera, obtains the processed video and displays it on the screen; 2. Server: the server of the video stream, which completes the sending and receiving of the video stream through the specified transmission protocol; 3. Image processing: realizes portrait segmentation and background replacement.

[0152] Fig.16 The block diagram of an image background replacement system according to an embodiment of the present disclosure is schematically shown.

[0153] like Fig.16 As shown, the image background replacement system 1600 includes: an image acquisition module 1610 , a selection module 1620 , a fusion module 1630 , and a synthesis module 1640 .

[0154] The image acquisition module 1610 is used to obtain the foreground prediction probability of each pixel in the first image; according to an embodiment of the present disclosure, the image acquisition module 1610 can be used to perform the above reference Figure 3 The S1 step described is not repeated here.

[0155] The selection module 1620 is used to select a point where the prospect prediction probability is close to the characteristic value as an uncertain point; according to an embodiment of the present disclosure, the selection module 1620 can be used to perform the above reference Figure 3 The described S2 step will not be repeated here.

[0156] The fusion module 1630 is used to fuse the feature maps of different scales of the uncertain points to obtain a foreground map with clear boundaries and an alpha mask; according to an embodiment of the present disclosure, the fusion module 1630 can be used to perform the above reference Figure 3 The S3 step described is not repeated here.

[0157] The synthesis module 1640 is used to synthesize the background image to be replaced with the foreground image and the alpha mask to obtain a second image under a new background; according to an embodiment of the present disclosure, the synthesis module 1640 can be used to perform the above reference Figure 3 The described S4 step will not be repeated here.

[0158] Fig.17 The block diagram of a video image background replacement system according to an embodiment of the present disclosure is schematically shown.

[0159] like Fig.17 As shown, the video image background replacement system 1700 includes: a video acquisition module 1710 and a background replacement module 1720 .

[0160] The video acquisition module 1710 is used to acquire a first video image in the current video.

[0161] The background replacement module 1720 is used to replace the background of the first video image according to the above-mentioned image background replacement method to obtain a second video image under a new background.

[0162] It should be noted that any one or more of the modules, submodules, units, and subunits according to the embodiments of the present disclosure, or at least part of the functions of any one of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any one or more of the modules, submodules, units, and subunits according to the embodiments of the present disclosure can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of the present disclosure can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0163] For example, any of the image acquisition module 1610, the selection module 1620, the fusion module 1630, the synthesis module 1640 or the video acquisition module 1710, and the background replacement module 1720 can be combined in one module for implementation, or any of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the image acquisition module 1610, the selection module 1620, the fusion module 1630, the synthesis module 1640 or the video acquisition module 1710, and the background replacement module 1720 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the image acquisition module 1610, the selection module 1620, the fusion module 1630, the synthesis module 1640 or the video acquisition module 1710, the background replacement module 1720 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0164] The present invention provides an image background replacement method system and method, which can be used in the fields of financial technology, especially the field of remote video agent customer service. It provides a method based on an improved background-matting algorithm to achieve portrait segmentation, and puts the segmented portrait on a new background to achieve the purpose of background replacement.

[0165] Fig.18 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Fig.18 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0166] like Fig.18 As shown, the electronic device 1800 described in this embodiment includes: a processor 1801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1802 or a program loaded from a storage part 1808 to a random access memory (RAM) 1803. The processor 1801 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1801 may also include an onboard memory for caching purposes. The processor 1801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0167] In RAM 1803, various programs and data required for the operation of system 1800 are stored. Processor 1801, ROM 1802, and RAM 1803 are connected to each other via bus 1804. Processor 1801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the programs in ROM 1802 and / or RAM 1803. It should be noted that the program can also be stored in one or more memories other than ROM 1802 and RAM 1803. Processor 1801 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in one or more memories.

[0168] According to an embodiment of the present disclosure, the electronic device 1800 may further include an input / output (I / O) interface 1805, which is also connected to the bus 1804. The system 1800 may further include one or more of the following components connected to the I / O interface 1805: an input portion 1806 including a keyboard, a mouse, etc.; an output portion 1807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1808 including a hard disk, etc.; and a communication portion 1809 including a network interface card such as a LAN card, a modem, etc. The communication portion 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to the I / O interface 1805 as needed. A removable medium 1811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1810 as needed, so that a computer program read therefrom is installed into the storage portion 1808 as needed.

[0169] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1809, and / or installed from the removable medium 1811. When the computer program is executed by the processor 1801, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0170] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the image background replacement method according to the present disclosure is implemented.

[0171] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 1802 and / or RAM 1803 described above and / or one or more memories other than ROM 1802 and RAM 1803.

[0172] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the image background replacement method provided by the embodiment of the present disclosure.

[0173] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0174] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 1809, and / or installed from a removable medium 1811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0175] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1809, and / or installed from the removable medium 1811. When the computer program is executed by the processor 1801, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0176] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0177] It should be noted that the functional modules in the various embodiments of the present disclosure may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present disclosure may essentially or in other words, the part that contributes to the prior art or all or part of the technical solution may be embodied in the form of a software product.

[0178] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0179] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0180] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A method for replacing an image background, characterized in that: include: Obtaining a foreground prediction probability for each pixel in the first image; Select the point where the prospect prediction probability is close to the characteristic value as the uncertain point; The feature maps of different scales of the uncertain points are fused to obtain a foreground image and an alpha mask with clear boundaries; Using the background image to be replaced, synthesizing the foreground image and the alpha mask to obtain a second image under a new background; The method of obtaining the foreground prediction probability of each pixel in the first image includes: obtaining the first image; inputting the first image into a convolutional neural network to obtain a feature map of the image; inputting the feature map into a pyramid pooling module to obtain the foreground prediction probability of each pixel; wherein different levels in the pyramid pooling module output feature maps of different scales, and the feature maps are input into each layer of the pyramid for pooling of different scales; and after upsampling, fully connected layer, and activation function processing, the foreground prediction probability of each pixel is obtained.

2. The image background replacement method according to claim 1, characterized in that: The step of selecting the point where the prospect prediction probability is close to the characteristic value as the uncertain point also includes: Deconvolution is performed to obtain a rough segmentation feature map.

3. The image background replacement method according to claim 2, characterized in that: The step of selecting a point where the prospect prediction probability is close to the characteristic value as an uncertain point comprises: Upsampling the boundary of the roughly segmented feature map by using bilinear interpolation; Obtaining the foreground prediction probability of the pixel points at the boundary; Points whose probabilities are close to the eigenvalues ​​are selected as uncertain points.

4. The image background replacement method according to claim 3, characterized in that: The foreground prediction probability comes from the formula of the cutout algorithm: I=αF+(1-α)B Where I is the image, F is the foreground of I, α is the foreground prediction probability of the pixel, and B is the background of I.

5. The image background replacement method according to claim 4, characterized in that: The point where the prospect prediction probability is close to the characteristic value is selected as the uncertain point, and the characteristic value is 0.

5.

6. The image background replacement method according to claim 1, characterized in that: The step of fusing the feature maps of the uncertain points at different scales to obtain a foreground map and an alpha mask with clear boundaries includes: Inputting the feature map into each layer of the pyramid for pooling at different scales to obtain feature maps of different scales; The feature maps of different scales of the uncertain points are fused to obtain a feature map with clear boundaries.

7. The image background replacement method according to claim 6, characterized in that: The fusing the feature maps of different scales of the uncertain points further comprises: Decode and obtain the foreground image with clear boundaries and alpha mask respectively.

8. The image background replacement method according to claim 7, characterized in that: The decoding to obtain a foreground image with clear boundaries includes: Bilinear difference upsampling, convolution, BN, and ReLU processing are performed to obtain the predicted feature map; The predicted feature map and the feature map with clear boundaries are subjected to bilinear difference upsampling, convolution, BN, ReLU, mirror padding, and convolution to obtain a foreground map with clear boundaries.

9. The image background replacement method according to claim 7, characterized in that: The decoding to obtain an alpha mask with clear boundaries includes: Bilinear difference upsampling, convolution, BN, ReLU processing for decoding; Mirror Padding, convolution, and Tanh processing are used to obtain an alpha mask with clear boundaries.

10. The image background replacement method according to claim 1, characterized in that: The synthesizing the background image to be replaced with the foreground image and the alpha mask comprises: Processing the alpha mask and the foreground image to obtain a foreground portion of the second image; Processing the alpha mask and the background image to be replaced to obtain the background part of the second image; The foreground portion and the background portion are superimposed to obtain a second image.

11. A method for replacing a video image background, characterized in that: include: Get the first video image in the current video; According to the image background replacement method according to any one of claims 1 to 10, the background of the first video image is replaced to obtain a second video image under a new background.

12. The video image background replacement method according to claim 11, characterized in that: The obtaining of the second video image under the new background comprises: Use rtmp transmission protocol to transmit real-time video.

13. An image background replacement system, characterized in that: include: The image acquisition module is used to obtain the foreground prediction probability of each pixel in the first image, including: obtaining the first image; inputting the first image into a convolutional neural network to obtain a feature map of the image; inputting the feature map into a pyramid pooling module to obtain the foreground prediction probability of each pixel; wherein different levels in the pyramid pooling module output feature maps of different scales, and the feature maps are input into each layer of the pyramid for pooling of different scales; after upsampling, full connection layer, and activation function processing, the foreground prediction probability of each pixel is obtained; A selection module, used for selecting points whose prospect prediction probabilities are close to the characteristic values ​​as uncertain points; A fusion module, used for fusing the feature maps of different scales of the uncertain points to obtain a foreground map and an alpha mask with clear boundaries; The synthesis module is used to synthesize the background image to be replaced with the foreground image and the alpha mask to obtain a second image under a new background.

14. A video image background replacement system, characterized in that: include: A video acquisition module, used to acquire a first video image in a current video; The background replacement module is used to replace the background of the first video image with the image background replacement method according to any one of claims 1 to 10 to obtain a second video image under a new background.

15. An electronic device, comprising: processor; A memory storing a computer executable program, which, when executed by the processor, enables the processor to execute the image background replacement method according to any one of claims 1 to 10 or the video image background replacement method according to any one of claims 11 to 12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image background replacement method according to any one of claims 1 to 10 or the video image background replacement method according to any one of claims 11 to 12 is implemented.

17. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for replacing the image background according to any one of claims 1 to 10 or the method for replacing the video image background according to any one of claims 11 to 12 is implemented.

Images