Method, device, storage medium, and program product for changing background in a screen

By capturing and recognizing a base background image to replace the current background in real-time, the method addresses the cost and clarity issues of live streaming, offering flexible and cost-effective background adjustments.

JP7592954B2Active Publication Date: 2024-12-03DOUYIN VISION CO LTD

Patent Information

Application Number
JP2023532118
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-05-12
Filing Date
2022-04-12
Publication Date
2024-12-03
Estimated Expiration
2042-04-12

AI Technical Summary

Technical Problem

The conventional method of using specialized equipment as a live distribution background increases costs and can cause light reflection, leading to unclear live distribution screens.

Method used

A method and device that captures a base background image of a preset scene in advance, recognizes the current background using this image, and replaces it with a preset background image to adjust the screen background without the need for physical equipment.

Benefits of technology

This approach reduces live streaming costs and eliminates light reflection issues, providing flexible and clear background adjustments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007592954000001
    Figure 0007592954000001
  • Figure 0007592954000002
    Figure 0007592954000002
  • Figure 0007592954000003
    Figure 0007592954000003
Patent Text Reader

Abstract

The method, device, storage medium, and program product for changing a background in a scene provided by this embodiment relate to image processing technology. The method includes the steps of acquiring a base background image of a preset scene, where the base background image is an image obtained by previously shooting the preset scene; acquiring a currently shot image obtained by currently shooting the preset scene and determining a current background image of the preset scene based on the base background image and the shot image; and replacing the current background image in the shot image with the preset background image to obtain a final shot image. The solution provided by this embodiment can recognize the current background image in the currently shot image using the previously collected base background image, thereby replacing the current background image in the shot image with the preset background image. This method eliminates the need to place a specific background device in the preset scene, and allows the background in the shot image to be adjusted as needed, improving the flexibility of changing the background in the shot image.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] TECHNICAL FIELD The embodiments of the present disclosure relate to image processing technology, and in particular to a method, an apparatus, a storage medium, and a program product for changing a background in a screen. [Background technology]

[0002] With the development of network technology, currently live streaming via the network has become popular, and the content of live streaming is becoming more and more abundant. For example, live streaming via the network can explain product information to viewers.

[0003] Currently, when live streaming over a network, it is necessary to set up a background, for example, a projection screen or a television set behind the anchor so that information can be displayed on the projection screen or television.

[0004] However, when a projection screen or television is placed, there is a possibility that light reflection may occur on the captured live streaming screen, which may make the live streaming screen unclear. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present disclosure provide a method, device, storage medium, and program product for changing the background in a screen to solve the problem that in the conventional technology, specialized equipment needs to be arranged as a live streaming background, which increases the live streaming cost and may cause the live streaming screen to be unclear due to the specialized equipment. [Means for solving the problem]

[0006] In a first aspect, an embodiment of the present disclosure provides a method for changing a background in a screen, the method comprising: A step of acquiring a base background image of a preset scene, the base background image being an image obtained by photographing the preset scene in advance; acquiring a currently captured image of the preset scene, and determining a current background image of the preset scene based on the base background image and the captured image; replacing a current background image in the shot scene with a preset background image to obtain a final shot scene.

[0007] In a second aspect, an embodiment of the present disclosure provides an apparatus for changing a background in a screen, the apparatus comprising: A base background image of a preset scene is obtained, and the base background image is an image obtained by photographing the preset scene in advance; and an acquisition unit for acquiring a photographed image obtained by currently photographing the preset scene; a recognition unit used to determine a current background image of the preset scene based on the base background image and the shooting screen; a background replacement unit, which is used to replace a current background image in the shooting scene with a preset background image to obtain a final shooting scene.

[0008] In a third aspect, an embodiment of the present disclosure provides an electronic device including at least one processor and a memory; The memory stores computer executable instructions, When the at least one processor executes the computer executable instructions stored in the memory, the at least one processor performs the method for changing a background in a screen described in the first aspect.

[0009] In a fourth aspect, an embodiment of the present disclosure provides a computer readable storage medium having computer executable instructions stored thereon, the method of changing a background in a screen as described in the first aspect being realized when a processor executes the computer executable instructions.

[0010] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processor, realises the method of changing a background in a screen as described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present disclosure provides a computer program, which, when executed by a processor, realises the method of changing a background in a screen as described in the first aspect. Effect of the Invention

[0012] The present disclosure Working Example teeth, Method, device, storage medium and program product for changing background in a screen The method includes the steps of: The method includes the steps of: acquiring a base background image of a preset scene, the base background image being an image obtained by previously shooting the preset scene; acquiring a shooting screen obtained by currently shooting the preset scene, and determining a current background image of the preset scene based on the base background image and the shooting screen; and replacing the current background image in the shooting screen with the preset background image to obtain a final shooting screen. In the solution provided by this embodiment, the previously collected base background image can be used to recognize the current background image in the current shooting screen, so that the current background image in the shooting screen can be replaced with the preset background image. In this way, it is not necessary to place a specific background device in the preset scene, and the background in the shooting screen can be adjusted as needed, improving the flexibility of changing the background in the shooting screen. [Brief description of the drawings]

[0013] In order to more clearly describe the embodiments of the present disclosure or the solutions in the prior art, the drawings that need to be used in the description of the embodiments or the prior art will be briefly described below. Of course, the drawings described below are some embodiments of the present disclosure, and those skilled in the art can think of other drawings based on these drawings without creative efforts. [Figure 1] FIG. 2 is a schematic diagram of a live broadcast screen shown in one exemplary embodiment; [Diagram 2] FIG. 13 is a schematic diagram of a live broadcast screen shown in another exemplary embodiment. [Diagram 3] 4 is a flowchart of a method for changing a background in a screen shown in one exemplary embodiment of the present application. [Figure 4] FIG. 2 is a schematic diagram of a base background image shown in one exemplary embodiment of the present application; [Diagram 5] 4 is a flowchart of a method for changing a background in a screen shown in another exemplary embodiment of the present application; [Figure 6] FIG. 2 is a schematic diagram of a background image shown in one exemplary embodiment of the present application; [Figure 7] FIG. 2 is a schematic diagram of a final shot screen generation shown in one exemplary embodiment of the present application; [Figure 8] FIG. 2 is a structural schematic diagram of an apparatus for changing a background in a screen shown in one exemplary embodiment of the present application; [Figure 9] FIG. 13 is a structural schematic diagram of an apparatus for changing a background in a screen shown in another exemplary embodiment of the present application; [Figure 10] FIG. 2 is a schematic diagram of a hardware structure of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions will be described clearly and completely below with reference to the drawings of the embodiments of the present disclosure. Of course, the described embodiments are only a part of the embodiments of the present disclosure, and are not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative labor, all belong to the protection scope of the present disclosure.

[0015] FIG. 1 is a schematic diagram of a live broadcast screen shown in one exemplary embodiment.

[0016] As shown in FIG. 1, a live distribution screen 11 can be displayed on a client terminal, and a television 12 is present on the live distribution screen.

[0017] In order to display information, a television 12 may be placed at the live distribution site and the live distribution information may be displayed using the television 12. However, a television may cause light reflection, which may make the live distribution screen unclear.

[0018] FIG. 2 is a schematic diagram of a live broadcast screen shown in another exemplary embodiment.

[0019] As shown in FIG. 2, a live distribution screen 21 can be displayed on a client terminal. The live distribution screen has a background 22 (shown with diagonal lines). The background 22 may be, for example, a preset image.

[0020] A green screen 23 (shown as a blank area) may be placed at the live streaming site, specifically, the green screen may be placed behind the presenter as a background, and software may be used to replace the green screen in the image with a pre-defined image.

[0021] However, these methods require the preparation of live streaming equipment, which increases the cost of live streaming.

[0022] In order to solve the above technical problem, the solution provided by the present application captures a base background image of a preset scene in advance, and the background in the shooting screen can be recognized using the base background image, and the background in the shooting screen can be replaced with the preset background image. In this way, there is no need to place a background such as a green screen or a television at the live streaming site, which reduces the cost of live streaming and prevents the screen from becoming unclear due to problems such as light reflection from the television.

[0023] FIG. 3 is a flowchart of a method for changing a background in a screen shown in one exemplary embodiment of the present application.

[0024] As shown in FIG. 3, the method for changing the background in a screen provided by the present application includes the following steps.

[0025] In step 301, a base background image of a preset scene is obtained. The base background image is an image obtained by photographing the preset scene in advance.

[0026] The methods provided herein may be performed by an electronic device having computing capabilities.

[0027] In one embodiment, the electronic device may be, for example, a computer, which may be, for example, a background server providing a live streaming service.

[0028] In another embodiment, the electronic device may be, for example, a mobile terminal, which can function as a live broadcasting terminal for the caster.

[0029] Specifically, by setting the method provided by the present application in an electronic device, the electronic device can be caused to execute the method provided by the present application.

[0030] Furthermore, when changing the screen background, the electronic device can obtain a base background image. The base background image is an image obtained by collecting preset scenes in advance, and for example, a user can obtain the base background image by shooting a scene where live distribution is performed. Specifically, the base background image can be shot from the angle at which the live distribution is shot.

[0031] Also, for example, the method provided herein can perform background modification on a video recorded in a preset scene, or perform background modification on an image captured in a preset scene.

[0032] In practical application, when collecting a base background image, the preset scene may only include an environment where no people are present and live streaming is taking place.

[0033] FIG. 4 is a schematic diagram of a base background image shown in one exemplary embodiment of the present application.

[0034] As shown in FIG. 4, in the base background image, the caster can live stream the scene. Generally, there is no change in the environment during live streaming, and the background of the live streaming screen does not change significantly along with the base background image, so the background part in the live streaming screen can be recognized based on the base background image.

[0035] In step 302, a photographed image obtained by currently photographing a preset scene is obtained, and a current background image of the preset scene is determined based on the base background image and the photographed image.

[0036] The electronic device can obtain a captured image obtained by currently capturing a preset scene, and the captured image may be captured live or in advance. The captured image may be a screen of one frame of a video or a captured image.

[0037] Since the captured image is a screen obtained by capturing a preset scene, the captured image includes the image content of the base background image. For example, when a caster broadcasts live using a preset scene, the captured image includes the caster and a part of the base background image.

[0038] Specifically, since the captured screen is the screen whose background needs to be replaced, the electronic device can acquire the captured screen and determine the current background image of the preset scene based on the base background image and the captured screen.

[0039] Furthermore, the electronic device can compare the base background image with the photographed screen to recognize a portion of the photographed screen that matches the base background image, thereby obtaining the current background image.

[0040] In step 303, the current background image in the shot image is replaced with the preset background image to obtain the final shot image.

[0041] In practical application, the user can also preset a preset background image. When the electronic device is a server, the user can upload the preset background image through the terminal, but when the electronic device is a live streaming user terminal, the user can select an image as the preset background image in the live streaming user terminal.

[0042] The electronic device can use the selected preset background image to replace the current background image in the shooting scene to obtain a final shooting scene. Specifically, the current background image can be used to extract a foreground image from the shooting scene, and the foreground image can be spliced ​​into the preset background image to obtain a final shooting scene.

[0043] Specifically, the user can also change the preset background image, thereby changing the live streaming background as needed during live streaming. For example, when introducing item A, the user can select a photo containing information introducing item A as the preset background image, and when introducing item B, the user can select a photo containing information introducing item B as the preset background image. By using such a means, the user can adjust the live streaming background image as needed.

[0044] When the electronic device is a server, the server can transmit the final captured image with the updated background to each user terminal, thereby allowing the user to view the final captured image with the replaced background.

[0045] When the electronic device is a live streaming terminal, the live streaming terminal can transmit the final captured screen with the updated background to the server, and when the final captured screen is transmitted from the server to the user terminal, the user can view the final captured screen with the replaced background.

[0046] The method for changing the background in the screen provided by the present application includes the steps of acquiring a base background image of a preset scene, the base background image being an image obtained by previously shooting the preset scene, acquiring a currently shot image obtained by currently shooting the preset scene, and determining a current background image of the preset scene based on the base background image and the shot image, and replacing the current background image in the shot image with the preset background image to obtain a final shot image. The method provided by the present application can recognize the current background image in the current shot image using the previously collected base background image, thereby replacing the current background image in the shot image with the preset background image. In this method, there is no need to place a specific background device in the preset scene, and the background in the shot image can be adjusted as needed, improving the flexibility of changing the background in the shot image.

[0047] FIG. 5 is a flowchart of a method for changing a background in a screen shown in another exemplary embodiment of the present application.

[0048] As shown in FIG. 5, the method for changing the background in a screen provided by the present application includes the following steps.

[0049] In step 501, a base background image of a preset scene is obtained, the base background image being an image obtained by photographing the preset scene in advance.

[0050] Step 501 and step 301 are similar in embodiment and principle, so they will not be described repeatedly here.

[0051] In step 502, a photographed image obtained by currently photographing a preset scene is acquired.

[0052] In step 502 and step 302, the means for acquiring the photographed image are similar, so they will not be described again here.

[0053] In step 503, base feature point information of the base background image is extracted, and shooting feature point information of the shooting screen is extracted.

[0054] The electronic device can recognize the current background image on the photographed screen based on the base background image, and more specifically, can extract feature point information of the base background image and the photographed screen.

[0055] Specifically, the extracted base feature point information can represent the features of the base background image, and the extracted shooting feature point information can represent the features of the shooting screen, so that the feature point information can be used to recognize parts of the shooting screen that match the base background image, and further to recognize the current background image.

[0056] Furthermore, the electronic device can use a scale invariant feature transform (SIFT) method to extract features of the base background image and the captured screen, thereby obtaining base feature point information of the base background image and the captured screen feature point information of the captured screen.

[0057] In step 504, a matching relationship between each piece of base feature information and each piece of photographing feature information is determined, and a current background image in the photographing screen is determined based on the matching relationship.

[0058] In practical application, the electronic device can determine a matching relationship between each piece of base feature point information and each piece of photographing feature point information by comparing the base feature point information and the photographing feature point information, for example, by comparing the base feature point information A1 with the photographing feature point information A2, and then comparing the base feature point information B1 with the photographing feature point information B2.

[0059] When the feature point information is extracted according to the SIFT method, each extracted feature point information can represent a stable local feature of an image, so that a stable matching result can be obtained based on the SIFT feature points.

[0060] Specifically, the electronic device can determine a current background image on the shooting screen based on the matching relationship.

[0061] Furthermore, the electronic device can determine the positional correspondence between the base background image and the photographed screen based on the matching relationship between each piece of base feature point information and each piece of photographed feature point information. For example, when a pixel point P1 of the base background image matches a pixel point P2 of the photographed screen, the pixel point P2 is a background part, and it can be understood that there is a positional correspondence between these two pixel points.

[0062] In practical application, the electronic device can determine the background part in the shooting screen based on the positional correspondence between the base background image and the shooting screen, and further stitch together the background part in the shooting screen and the base background image to obtain the current background image of the preset scene in the shooting screen.

[0063] Since the camera position when currently shooting the preset scene and the camera position when collecting the base background image may not be exactly the same, the background part in the shooting screen and the base background image may not completely overlap, so the base background image and the shooting screen can be spliced ​​together to obtain a larger size background image.

[0064] For example, if the camera position when currently shooting the preset scene is slightly higher than the camera position when collecting the base background image, the image content of the upper part of the shooting screen can be spliced ​​into the base background image to obtain a larger sized spliced ​​background image.

[0065] FIG. 6 is a schematic diagram of a background image shown in one exemplary embodiment of the present application.

[0066] As shown in FIG. 6, a base background image is shown at 61, and when a background portion 63 is recognized in a photographed screen 62, 63 and 61 are joined together to obtain a current background image 64.

[0067] After step 504, steps 507 and 510 can be performed, thereby directly generating a final shot image based on the current background image.

[0068] In step 505, the boundary of the current background image is expanded to obtain an expanded background image.

[0069] In one alternative embodiment, the solution provided herein can further extend the boundary of the current background image to obtain an extended background image.

[0070] In such an embodiment, when the camera angle changes, the method provided herein can accurately recognize the background portion in the shooting scene based on the augmented background image.

[0071] In such an embodiment, the augmented background image in the photographed image can be replaced with a preset background image based on the augmented background image to obtain the final photographed image. Specifically, the background portion in the photographed image can be recognized based on the augmented background image, a foreground image can be extracted, and the foreground image can be used to generate the final photographed image.

[0072] After step 505, steps 508 and 510 can be performed, thereby generating a final shot based on the augmented background image.

[0073] In step 506, the enhanced background image is subjected to either of the enhanced seam blurring and ray transformation processes to obtain an enhanced background image.

[0074] In one optional embodiment, the method provided by the present application further performs an enhancement process on the augmented background image to obtain an enhanced background image, so that the enhanced background image is more similar to the background part in the shooting scene, and the background part in the shooting scene can be accurately recognized based on the enhanced background image.

[0075] In such an embodiment, the enhanced background image in the photographed scene can be replaced with a preset background image based on the enhanced background image to obtain the final photographed scene. Specifically, the enhanced background image can be used to recognize a background part in the photographed scene, extract a foreground image, and generate a final photographed scene using the foreground image.

[0076] Specifically, a seam blurring process can be performed on the augmented background image to make the transition at the seam in the image more natural, and a light shift process can be performed on the augmented background image to increase the brightness of the image and further enhance the texture features of the image.

[0077] Specifically, the electronic device can determine a foreground image included in the shooting screen based on the current background image and the shooting screen. Specifically, the electronic device can directly extract the foreground image in the shooting screen based on the current background image, and can also extract the foreground image in the shooting screen using the augmented background image determined based on the current background image. Furthermore, the electronic device can extract the foreground image in the shooting screen using the enhanced background image determined based on the current background image.

[0078] After step 506, steps 509 and 510 can be performed, thereby generating a final shot based on the enhanced background image.

[0079] In step 507, the current background image and the photographed scene are input to a preset division model, and the preset division model is used to output a foreground image contained in the photographed scene.

[0080] In the electronic device, a model capable of outputting a foreground image included in the photographed screen based on a background image and the photographed screen can be arranged, and the model can be obtained by training in advance. For example, a preset model can be constructed in advance and trained to obtain a preset division model, and the feature extraction network in the preset model can be, for example, a mobilenet network (mobile terminal neural network).

[0081] By using the mobilenet network to construct a preset model, the size of the model can be made smaller, so that the model can be applied to the mobile terminal side.

[0082] In one alternative embodiment, the electronic device can input a current background image and a shooting scene into a preset splitting model, and the preset splitting model can output a foreground image.

[0083] In step 508, the augmented background image determined based on the current background image and the captured image is input into a preset division model, which is used to output a foreground image included in the captured image.

[0084] A model capable of outputting a foreground image included in a photographed screen based on a background image and the photographed screen can be arranged in the electronic device. The model can be obtained by training in advance, for example, a preset model can be constructed in advance and trained to obtain a preset segmentation model. The feature extraction network in the preset model can be, for example, a mobilenet network.

[0085] By constructing a preset model using the mobilenet network, the size of the model can be made smaller, and as a result, the model can be applied to the mobile terminal side.

[0086] In one alternative embodiment, the electronic device can input the augmented background image and the shooting scene into a preset splitting model, and the preset splitting model can output the foreground image.

[0087] In step 509, the enhanced background image determined based on the current background image and the captured image is input into a preset division model, and the preset division model is used to output a foreground image included in the captured image.

[0088] A model capable of outputting a foreground image included in a photographed screen based on a background image and the photographed screen can be arranged in the electronic device. The model can be obtained by training in advance, for example, a preset model can be constructed in advance and trained to obtain a preset segmentation model. The feature extraction network in the preset model can be, for example, a mobilenet network.

[0089] By constructing a preset model using the mobilenet network, the size of the model can be made smaller, and as a result, the model can be applied to the mobile terminal side.

[0090] In one alternative embodiment, the electronic device can input the enhanced background image and the shooting scene into a preset segmentation model, and the preset segmentation model can output the foreground image.

[0091] In step 510, a final shooting scene is generated based on the foreground image and the preset background image, where the background of the final shooting scene is the preset background image and the foreground of the final shooting scene is the foreground image.

[0092] Specifically, the electronic device can obtain a final image by fusing the foreground image with the acquired preset background image. By using such a means, the background in the image can be more appropriately captured, and the background part in the image can be accurately recognized based on the method according to the present application, so that an accurate foreground image can be extracted, and a clear final image can be obtained.

[0093] FIG. 7 is a schematic diagram of the final shot screen generation shown in one exemplary embodiment of the present application.

[0094] As shown in FIG. 7, the electronic device can extract a foreground image 71 from a shooting screen, and then fuse the foreground image 71 with a preset background image 72 to obtain a final shooting screen 73 with the foreground image 71 as the foreground and the preset background image 72 as the background.

[0095] In an alternative embodiment, the method provided herein may further include the steps of:

[0096] In step 511, the base background image is updated to an enhanced or augmented background image.

[0097] The electronic device can continue to process based on the updated base background image when changing the background for the shooting screen, and since the extended background image and the enhanced background image are both extended images based on the current shooting angle of view, when the camera is rotated, the background portion can still be accurately recognized from the shooting screen based on the updated base background image, thereby allowing an accurate foreground image to be extracted.

[0098] In step 512, the final captured image is transmitted.

[0099] The method provided by this embodiment can be implemented by a live streaming terminal, which can use a preset background image selected by a user to replace the background part in the captured image collected by the live streaming terminal, thereby achieving the effect of changing the background.

[0100] Specifically, the live streaming terminal can change the background in the captured image and then upload the resulting final captured image to a server. The server then transmits the final captured image to each user terminal, allowing the users to view the final captured image using their own terminals.

[0101] FIG. 8 is a structural schematic diagram of an apparatus for changing a background in a screen shown in one exemplary embodiment of the present application.

[0102] As shown in FIG. 8, the present application provides an apparatus 800 for changing a background in a screen, which comprises: A base background image of a preset scene is obtained, and the base background image is an image obtained by photographing the preset scene in advance; and acquiring a photographed image obtained by currently photographing the preset scene; An acquisition unit 810 for use in A recognition unit 820, which is used to determine a current background image of the preset scene based on the base background image and the shooting screen; a background replacement unit 830, which is used to replace a current background image in the shooting scene with a preset background image to obtain a final shooting scene.

[0103] The present application provides an apparatus for changing the background in a screen, which includes: an acquisition unit for acquiring a base background image of a preset scene, the base background image being an image obtained by previously shooting the preset scene, and for acquiring a shooting screen obtained by currently shooting the preset scene; a recognition unit for determining a current background image of the preset scene based on the base background image and the shooting screen; and a background replacement unit for replacing the current background image in the shooting screen with the preset background image to obtain a final shooting screen. The present application provides an apparatus for recognizing a current background image in the current shooting screen using a base background image collected in advance, thereby replacing the current background image in the shooting screen with the preset background image. In this manner, it is not necessary to place a specific background device in the preset scene, and the background in the shooting screen can be adjusted as needed, improving the flexibility of changing the background in the shooting screen.

[0104] FIG. 9 is a structural schematic diagram of an apparatus for changing a background in a screen shown in another exemplary embodiment of the present application.

[0105] As shown in FIG. 9, based on the above embodiment, in the device 900 for changing the background in the screen provided by the present application, in one optional embodiment, the recognition unit 820: A feature extraction module 821 is used to extract base feature point information of the base background image and extract shooting feature point information of the shooting screen; A matching module 822 is used to determine a matching relationship between each piece of base feature information and each piece of photographing feature information; and a background determination module 823, which is used to determine the current background image in the shooting screen based on the matching relationship.

[0106] In one alternative embodiment, the background determination module 823 specifically: Based on the matching relationship, a background portion in the shooting screen and the base background image are stitched together to obtain a current background image of the preset scene in the shooting screen.

[0107] In one alternative embodiment, the background determination module 823 further comprises: used to expand the boundary of the current background image to obtain an expanded background image; Accordingly, the background replacement unit 830 specifically: Based on the augmented background image, a preset background image is used to replace the augmented background image in the shot image to obtain the final shot image.

[0108] In one alternative embodiment, the background replacement unit 830 specifically comprises: For the augmented background image, A seam blurring and a light ray transformation are performed to obtain an enhanced background image; Accordingly, the background replacement unit 830 specifically: Based on the enhanced background image, the augmented background image in the shot is replaced with a preset background image, which is used to obtain the final shot.

[0109] In one alternative embodiment, the background replacement unit 830 comprises: a foreground determination module 831, which is used to determine a foreground image included in the photographed scene according to the current background image and the photographed scene; an image generation module 832 used to generate the final captured image based on the foreground image and the preset background image; The background of the final shot is the preset background image, and the foreground of the final shot is the foreground image.

[0110] In one alternative embodiment, the foreground determination module 831 specifically: inputting the current background image and the photographed scene into a preset division model, the preset division model being used to output a foreground image included in the photographed scene; Alternatively, an augmented background image determined based on the current background image and the photographed screen are input into a preset division model, and the preset division model is used to output a foreground image included in the photographed screen; Alternatively, an enhanced background image determined based on the current background image and the photographed screen are input into a preset division model, and the preset division model is used to output a foreground image included in the photographed screen.

[0111] In one alternative embodiment, the apparatus further comprises: An update unit 840 is used to update the base background image to the enhanced background image or the augmented background image.

[0112] In one alternative embodiment, the apparatus further comprises: A transmitting unit 850 is used to transmit the final captured image.

[0113] The apparatus provided by this embodiment may be used to implement the technical solutions related to the above method embodiments, and since the implementation principles and technical effects are similar, this embodiment will not be described repeatedly here.

[0114] The present embodiment further provides a computer program product including a computer program, which, when executed by a processor, realises the method of changing a background in a screen as described in any one of the above claims.

[0115] This embodiment further provides a computer program, which, when executed by a processor, realizes the method for changing a background in a screen as described in any one of the above.

[0116] Referring to FIG. 10, FIG. 10 shows a structural schematic diagram of an electronic device 1000 suitable for implementing the embodiments of the present disclosure, and the electronic device 1000 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (abbreviated as PDA), tablet computers (abbreviated as Portable Android Devices (abbreviated as PAD), portable multimedia players (abbreviated as PMP), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG. 10 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0117] As shown in FIG. 10, the electronic device 1000 may include a processing unit (such as a central processing unit or a graphic processor) 1001, which may perform various appropriate operations and processes according to a program stored in a read only memory (ROM) 1002 or loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0118] Input devices 1006, typically including a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007, typically including a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008, typically including a magnetic tape, hard disk, etc.; and communication devices 1009 may be connected to the I / O interface 1005. The communication devices 1009 may enable the electronic device 1000 to communicate and exchange data with other devices wirelessly or via wires. Although FIG. 10 illustrates the electronic device 1000 with various devices, it should be understood that not all of the devices illustrated need be implemented or deployed. Alternatively, more or fewer devices may be implemented or deployed.

[0119] In particular, according to an embodiment of the present disclosure, the above processes described with reference to the flowcharts may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product including a computer program carried on a computer-readable medium, the computer program including program code for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via a communication device 1009, or installed from a storage device 1008, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, it performs the above functions defined in the method according to the embodiment of the present disclosure.

[0120] It should be noted that the computer readable medium described in this disclosure may be a computer readable signal medium, or a computer readable storage medium, or any combination of the two. The computer readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a Random Access Memory (abbreviated as RAM), a Read-Only Memory (abbreviated as ROM), an Electrically Programmable Read-Only Memory (abbreviated as EPROM or flash memory), an optical fiber, a portable Compact Disc Read-Only Memory (abbreviated as CD-ROM), an optical storage device, a magnetic memory component, or any suitable combination of the above. In this disclosure, a computer-readable storage medium may be any tangible medium that includes or stores a program for use by or in combination with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal, propagated in baseband or as part of a carrier wave, with computer-readable program code embedded therein. Such a propagated data signal may take many forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.The program code contained in the computer readable medium may be transmitted using any suitable medium, including but not limited to electrical wire, fiber optic cable, RF (radio frequency), etc., or any suitable combination of the above.

[0121] The computer-readable medium may be included in the electronic device, or may exist independently without being assembled in the electronic device.

[0122] The computer-readable medium has one or more computer programs stored thereon, and when the one or more computer programs are executed by the electronic device, the electronic device performs the method shown in the above embodiments.

[0123] Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages, or a combination thereof. The program code can run entirely on the user's computer, partly on the user's computer, as a standalone software package, partly on the user's computer, partly on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any kind of network, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., connected via the Internet using an Internet Service Provider).

[0124] The flowcharts and block diagrams in the drawings illustrate the architecture, functions, and operations that can be realized by the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, a program segment, or a portion of code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions shown in the blocks may be executed in a different order than that shown in the figures. For example, two blocks shown as connected may actually be executed in parallel, or the blocks may be executed in the reverse order depending on the related functionality. It should be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or may be implemented using a combination of dedicated hardware and computer instructions.

[0125] The units described in the embodiments of the present disclosure can be implemented in software or hardware. The names of these units may not be intended to limit the units themselves in a particular situation, for example, the first acquisition unit may be described as "a unit for acquiring at least two Internet Protocol addresses."

[0126] The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used may include, but are not limited to, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0127] In the context of this disclosure, a computer-readable medium may be a tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer-readable storage media may include electrical connections based on one or more wires, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic memory components, or any suitable combination of the above.

[0128] In a first aspect, according to one or more embodiments of the present disclosure, there is provided a method for changing a background in a screen, the method comprising: A step of acquiring a base background image of a preset scene, the base background image being an image obtained by photographing the preset scene in advance; acquiring a currently captured image of the preset scene, and determining a current background image of the preset scene based on the base background image and the captured image; replacing a current background image in the shot scene with a preset background image to obtain a final shot scene.

[0129] According to one or more embodiments of the present disclosure, the step of determining a current background image of the preset scene based on the base background image and the shooting screen includes: extracting base feature point information of the base background image and extracting shooting feature point information of the shooting screen; determining a matching relationship between each piece of base feature point information and each piece of shooting feature point information; and determining the current background image in the shooting screen based on the matching relationship.

[0130] According to one or more embodiments of the present disclosure, the step of determining the current background image in the shooting screen based on the matching relationship includes: The method includes a step of splicing a background portion in the shooting screen and the base background image based on the matching relationship to obtain a current background image of the preset scene in the shooting screen.

[0131] According to one or more embodiments of the present disclosure, the method may further include: expanding a boundary of the current background image to obtain an expanded background image; Accordingly, based on the augmented background image, replacing the augmented background image in the captured image with a preset background image to obtain the final captured image.

[0132] According to one or more embodiments of the present disclosure, the method may further include: For the augmented background image, performing an enhanced process of either seam blurring or ray transformation to obtain an enhanced background image; Accordingly, based on the enhanced background image, replacing the augmented background image in the shot scene with a preset background image to obtain the final shot scene.

[0133] According to one or more embodiments of the present disclosure, the step of replacing a current background image in the shooting screen with a preset background image to obtain a final shooting screen includes: determining a foreground image to be included in the captured image based on the current background image and the captured image; generating the final shooting screen based on the foreground image and the preset background image, wherein the background of the final shooting screen is the preset background image and the foreground of the final shooting screen is the foreground image.

[0134] According to one or more embodiments of the present disclosure, the step of determining a foreground image included in the shooting screen based on the current background image and the shooting screen includes: inputting the current background image and the photographed scene into a preset division model, the preset division model being used to output a foreground image included in the photographed scene; or inputting the augmented background image determined based on the current background image and the photographed scene into a preset division model, the preset division model being used to output a foreground image included in the photographed scene; Or, the method includes a step of inputting an enhanced background image determined based on the current background image and the captured screen into a preset division model, and the preset division model is used to output a foreground image included in the captured screen.

[0135] According to one or more embodiments of the present disclosure, the method may further include: updating the base background image to the enhanced background image or the augmented background image.

[0136] According to one or more embodiments of the present disclosure, the method may further include: and transmitting the final captured image.

[0137] In a second aspect, in accordance with one or more embodiments of the present disclosure, there is provided an apparatus for changing a background in a screen, the apparatus comprising: A base background image of a preset scene is obtained, and the base background image is an image obtained by photographing the preset scene in advance; and an acquisition unit for acquiring a photographed image obtained by currently photographing the preset scene; a recognition unit used for determining a current background image of the preset scene based on the base background image and the shooting screen; a background replacement unit, which is used to replace a current background image in the shooting scene with a preset background image to obtain a final shooting scene.

[0138] According to one or more embodiments of the present disclosure, the recognition unit comprises: a feature extraction module used to extract base feature point information of the base background image and extract shooting feature point information of the shooting screen; a matching module used to determine a matching relationship between each piece of base feature information and each piece of photographing feature information; and a background determination module used for determining the current background image in the shooting scene based on the matching relationship.

[0139] According to one or more embodiments of the present disclosure, the context determination module specifically includes: Based on the matching relationship, a background portion in the shooting screen and the base background image are stitched together to obtain a current background image of the preset scene in the shooting screen.

[0140] According to one or more embodiments of the present disclosure, the context determination module may further include: used to expand the boundary of the current background image to obtain an expanded background image; Accordingly, the background replacement unit specifically: Based on the augmented background image, a preset background image is used to replace the augmented background image in the shot image to obtain the final shot image.

[0141] According to one or more embodiments of the present disclosure, the background replacement unit specifically includes: For the augmented background image, A seam blurring and a light ray transformation are performed to obtain an enhanced background image; Accordingly, the background replacement unit specifically: Based on the enhanced background image, the augmented background image in the shot is replaced with a preset background image, which is used to obtain the final shot.

[0142] According to one or more embodiments of the present disclosure, the background replacement unit comprises: a foreground determination module used for determining a foreground image included in the photographed scene according to the current background image and the photographed scene; an image generating module used to generate the final captured image based on the foreground image and the preset background image; The background of the final shot is the preset background image, and the foreground of the final shot is the foreground image.

[0143] According to one or more embodiments of the present disclosure, the foreground determination module specifically comprises: inputting the current background image and the photographed scene into a preset division model, the preset division model being used to output a foreground image included in the photographed scene; Alternatively, the augmented background image determined based on the current background image and the photographed screen are input into a preset division model, and the preset division model is used to output a foreground image included in the photographed screen; Alternatively, an enhanced background image determined based on the current background image and the photographed screen are input into a preset division model, and the preset division model is used to output a foreground image included in the photographed screen.

[0144] In accordance with one or more embodiments of the present disclosure, the apparatus may further include: An update unit is included which is used to update the base background image to the enhanced background image or the augmented background image.

[0145] In accordance with one or more embodiments of the present disclosure, the apparatus may further include: A sending unit is used to send the final captured image to a server.

[0146] In a third aspect, there is provided an electronic device according to one or more embodiments of the present disclosure, comprising at least one processor and a memory, The memory stores computer executable instructions, When the at least one processor executes the computer executable instructions stored in the memory, the at least one processor performs the method of changing a background in a screen described in the first aspect and various possible designs of the first aspect above.

[0147] In a fourth aspect, in accordance with one or more embodiments of the present disclosure, there is provided a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, realises a method of changing a background in a screen as described in the first aspect above and various possible designs of the first aspect.

[0148] In a fifth aspect, there is provided a computer program product according to one or more embodiments of the present disclosure, comprising a computer program, which, when executed by a processor, realises a method of changing a background in a screen as described in any one of the above claims.

[0149] In a sixth aspect, according to one or more embodiments of the present disclosure, there is provided a computer program, which, when executed by a processor, realises a method of changing a background in a screen as described in any one of the above paragraphs.

[0150] The above description is merely a description of some preferred embodiments of the present disclosure and the technical principles applied thereto. Those skilled in the art should understand that the scope of the invention according to the embodiments of the present disclosure is not limited to the solution formed by a specific combination of the above technical features, but should also cover other solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept, for example, by replacing the above features with technical features having similar functions disclosed in the present disclosure (but not limited thereto).

[0151] Additionally, although operations are described in a particular order, this should not be understood as requiring that these operations be performed in the particular order or sequence shown. Multitasking or parallel processing may be advantageous in certain circumstances. Similarly, the above description includes several specific implementation details, which should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments, either alone or in any suitable subcombination.

[0152] Although the present subject matter has been described using language specific to structural features and / or methodological logical operations, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or operations described above. Rather, the specific features and operations described above are merely example forms for implementing the claims.

[0153] This application claims priority to a Chinese patent application filed on May 12, 2021, bearing application number 202110518684.4 and entitled "Method, apparatus, storage medium and program product for changing the background on a screen," the entire contents of which are incorporated herein by reference.

Claims

1. 1. A method for changing a background in a screen, comprising: A step of acquiring a base background image of a preset scene, the base background image being an image obtained by photographing the preset scene in advance; acquiring a currently captured image of the preset scene, and determining a current background image of the preset scene based on the base background image and the captured image; replacing a current background image in the photographed scene with a preset background image to obtain a final photographed scene; Including, The step of determining a current background image of the preset scene based on the base background image and the shooting screen includes: extracting base feature point information of the base background image and extracting shooting feature point information of the shooting screen; determining a matching relationship between each piece of base feature point information and each piece of photographing feature point information, and determining the current background image in the photographing screen based on the matching relationship; Including, The step of determining the current background image in the photographing screen based on the matching relationship includes: splicing a background portion in the shooting screen and the base background image based on the matching relationship to obtain a current background image of the preset scene in the shooting screen; A method comprising:

2. The method further comprises: expanding a boundary of the current background image to obtain an expanded background image; accordingly, replacing the augmented background image in the photographed scene with a preset background image based on the augmented background image to obtain the final photographed scene; Including, 2. The method according to claim 1 .

3. The method further comprises: performing an enhanced process of either seam blurring or ray conversion on the augmented background image to obtain an enhanced background image; accordingly, replacing the augmented background image in the shot image with a preset background image based on the enhanced background image to obtain the final shot image; Including, 3. The method according to claim 2 .

4. The step of replacing the current background image in the photographed scene with the preset background image to obtain a final photographed scene includes: determining a foreground image to be included in the captured image based on the current background image and the captured image; generating the final photographed image based on the foreground image and the preset background image, the background of the final photographed image being the preset background image, and the foreground of the final photographed image being the foreground image; Including, 4. The method according to claim 3 .

5. The step of determining a foreground image included in the photographed screen based on the current background image and the photographed screen includes: inputting the current background image and the photographed scene into a preset division model, the preset division model being used to output a foreground image included in the photographed scene; or inputting the augmented background image determined based on the current background image and the captured image into a preset division model, the preset division model being used to output a foreground image included in the captured image; or inputting an enhanced background image determined based on the current background image and the photographed scene into a preset division model, the preset division model being used to output a foreground image included in the photographed scene; Including, 5. The method according to claim 4 .

6. The method further comprises: updating the base background image to the enhanced background image or the augmented background image; 4. The method according to claim 3 .

7. The method further comprises: transmitting the final captured image; 7. The method according to claim 1, wherein the first and second electrodes are connected to a first electrode.

8. A device for changing the background of a screen, comprising: is an acquisition unit, A base background image of a preset scene is obtained, the base background image being an image obtained by photographing the preset scene in advance; and acquiring a photographed image obtained by currently photographing the preset scene; An acquisition unit for use in a recognition unit used for determining a current background image of the preset scene based on the base background image and the shooting screen; a background replacement unit, which is used to replace a current background image in the photographed scene with a preset background image to obtain a final photographed scene; Including, The recognition unit includes: extracting base feature point information of the base background image and extracting shooting feature point information of the shooting screen; determining a matching relationship between each piece of base feature point information and each piece of photographing feature point information, and determining the current background image in the photographing screen based on the matching relationship; determining the current background image in the photographing screen based on the matching relationship, splicing a background part in the shooting screen and the base background image based on the matching relationship to obtain a current background image of the preset scene in the shooting screen; An apparatus comprising:

9. An electronic device including at least one processor and a memory, The memory stores computer executable instructions, 7. The method of claim 1, wherein the at least one processor includes a first processor and a second processor.

8. The method of claim 1, wherein the at least one processor includes a second processor and a third processor. An electronic device comprising:

10. 1. A computer-readable storage medium, comprising: The computer-readable storage medium has computer-executable instructions stored thereon, When the computer-executable instructions are executed by a processor, 7. causing the processor to carry out a method according to any one of claims 1 to 6; A computer-readable storage medium comprising:

11. A computer program comprising: The computer program, when executed by a processor, 7. causing the processor to carry out a method according to any one of claims 1 to 6; A computer program comprising:

Citation Information

Patent Citations

  • Image processing device

    JP2009098718A

  • Image capturing apparatus, image processing method, and program

    JP2010183560A

  • Imaging apparatus, imaging method and program

    JP2011041041A

  • Subject image extraction device and subject image extraction / synthesis device

    JP2014192557A

  • Controller, control method, and program

    JP2017212592A

Cited By

  • Method for replacing background in picture, device, storage medium and program product

    US12744861B2