Display method, electronic equipment and readable storage medium

Abstract

The invention provides a display method, electronic equipment and a readable storage medium. The method is applied to electronic equipment and comprises the steps that a video picture is displayed in a first application interface, the video picture comprises a first virtual background and a foreground target, and the foreground target has a first display parameter; it is detected that a first display parameter of a foreground target in the video picture is changed into a second display parameter, and the first display parameter and the second display parameter comprise a position parameter and a size parameter; the first virtual background is replaced with a second virtual background, the first virtual background and the second virtual background are a first area and a second area of the virtual background source image respectively, and the size of the second area and the position of the second area in the virtual background source image are related to the second display parameter. In a scene in which background replacement is carried out on the display picture of which the director function is started, the virtual background can be changed along with the change of the position and / or the size of the target person, so that the visual experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a display method, an electronic device, and a readable storage medium. Background Art

[0002] With the widespread application of video conferencing, video conferencing functions such as directing and background replacement have gradually become popular. The directing function refers to the function of identifying and tracking the foreground target in the shooting picture, and then always displaying the foreground target in the center of the display picture. The background replacement function refers to the function of replacing the actual background in the display picture with a virtual background.

[0003] It is understandable that the directing function can be used together with the background replacement function. For example, in some directing scenarios, the user may need to continuously display the screen (display screen) where the speaker is located based on the directing function, and replace the background of the speaker in the display screen with a virtual background. In the solution to achieve this user demand, the speaker's portrait can be separated from the shot screen and placed in the center of the virtual background to form a display screen after the background is replaced. However, the replaced virtual background in this solution remains unchanged and cannot adapt to changes in the position and / or size of the speaker's portrait, resulting in a strong sense of disharmony in the replaced display screen. Summary of the invention

[0004] The present application provides a display method, an electronic device and a readable storage medium. In a scenario where the background of a display screen with a director function turned on is replaced, the virtual background in the display screen can change with the position and / or size of the target person, thereby improving the user's visual experience.

[0005] In a first aspect, the present application provides a display method, applied to an electronic device, the method comprising: displaying a video screen in a first application interface, wherein the video screen includes a first virtual background and a foreground target, and the foreground target has a first display parameter; detecting that the first display parameter of the foreground target in the video screen is changed to a second display parameter, wherein the first display parameter and the second display parameter include a position parameter and a size parameter; replacing the first virtual background with a second virtual background, wherein the first virtual background and the second virtual background are respectively a first area and a second area of ​​a virtual background source image, and the size of the second area and its position in the virtual background source image are related to the second display parameter.

[0006] Exemplarily, the aforementioned video screen may be a display screen after background replacement based on the display method of the present application, the first virtual background and the second virtual background may be the virtual image area described below, and the foreground target may be the host or speaker described below. During the playback of the display screen, if the first display parameter of the foreground target changes to the second display parameter, for example, the first position parameter and / or the first size parameter in the first display parameter changes to the second position parameter and / or the second size parameter in the second display parameter, correspondingly, the first virtual background corresponding to the first display parameter in the display screen will change to the second virtual background corresponding to the second display parameter.

[0007] In a possible implementation of the first aspect above, the first display parameter and the second display parameter are related to the position parameter and display parameter corresponding to the target frame that frames the foreground target in the captured image, and the method includes: determining the first display parameter based on the first position and the first size of the target frame in the captured image; determining the second display parameter based on the second position and the second size of the target frame in the captured image.

[0008] Exemplarily, the captured image may be the current video frame described below, and the target frame that frames the foreground target in the captured image may be the close-up area described below. The first position and the second position of the target frame in the captured image may be the coordinates of the upper left pixel point of the close-up area of ​​the presenter before and after walking in the current video frame. For example, the first position may be the coordinates of the upper left pixel point of the close-up area 510a / 610a described below in the current video frame; the second position may be the coordinates of the upper left pixel point of the close-up area 511a / 611a described below in the current video frame. The first size and the second size of the target frame in the captured image may be the width and height of the close-up area of ​​the presenter described below before and after scaling. For example, the first size may be the width and height of the close-up area 510a / 610a described below in the current video frame; the second size may be the width and height of the close-up area 511a / 611a described below in the current video frame. Accordingly, the first display parameter may be the coordinates of the upper left pixel of the close-up area in the current video frame before the speaker moves and / or the close-up area is scaled, as well as the width and height in the current video frame. For example, the first display parameter may be the coordinates of the upper left pixel of the close-up area 510a / 610a in the current video frame, as well as the width and height. The second display parameter may be the coordinates of the upper left pixel of the close-up area in the current video frame after the speaker moves and / or the close-up area is scaled, as well as the width and height in the current video frame. For example, the second display parameter may be the coordinates of the upper left pixel of the close-up area 511a / 611a in the current video frame, as well as the width and height.

[0009] In a possible implementation of the first aspect above, replacing the first virtual background with the second virtual background includes: mapping the first position of the target frame in the captured image to the first position parameter of the first area in the virtual background source image, and mapping the first size of the target frame in the captured image to the first size parameter of the first area in the virtual background source image; mapping the second position of the target frame in the captured image to the second position parameter of the second area in the virtual background source image, and mapping the second size of the target frame in the captured image to the second size parameter of the second area in the virtual background source image; replacing the picture within the first area framed in the virtual background source image based on the first position parameter and the first size parameter with the picture within the second area framed in the virtual background source image based on the second position parameter and the second size parameter.

[0010] Exemplarily, the first position parameter of the first area in the virtual background source image can be the coordinates of the upper left pixel of the virtual image area corresponding to the close-up area in the virtual background source image before the presenter walks and / or the close-up area is scaled. For example, it can be the coordinates of the upper left pixel of 510b / 610b in the virtual background source image. The second position parameter of the second area in the virtual background source image can be the coordinates of the upper left pixel of the virtual image area corresponding to the close-up area in the virtual background source image after the presenter walks and / or the close-up area is scaled. For example, it can be the coordinates of the upper left pixel of 511b / 611b in the virtual background source image. The first size parameter of the first area in the virtual background source image can be the width and height of the virtual image area corresponding to the close-up area in the virtual background source image before the presenter walks and / or the close-up area is scaled. For example, it can be the width and height of 510b / 610b in the virtual background source image. The second size parameter of the second area in the virtual background source image may be the width and height of the virtual image area corresponding to the close-up area of ​​the presenter after the presenter moves and / or the close-up area is zoomed in the virtual background source image. For example, it may be the width and height of 511b / 611b in the virtual background source image.

[0011] It can be understood that the picture within the first area framed in the virtual background source image based on the first position parameter and the first size parameter is the above-mentioned first virtual background, and the picture within the second area framed in the virtual background source image based on the second position parameter and the second size parameter is the above-mentioned second virtual background. Since the first size parameter and the first position parameter of the first area correspond to the first size and the first position of the target frame in the captured image, the second size parameter and the second position parameter of the second area correspond to the second size and the second position of the target frame in the captured image. Therefore, the virtual background of the playback screen will change synchronously with the change of the position and / or size of the foreground target in the captured image, thereby enhancing the user's visual experience.

[0012] In a possible implementation of the first aspect, mapping the first position of the target frame in the captured image to the first position parameter of the first area in the virtual background source image includes: determining a normalized position parameter (x1, y1) of the target frame according to the coordinates (x1, y1) corresponding to the first position of the target frame in the captured image and a third size parameter (W, H) corresponding to the captured image. 01 ,y 01 ):

[0013]

[0014]

[0015] According to the normalized position parameter (x 01 ,y 01 ) and the fourth size parameter (W0, H0) corresponding to the virtual background source image determine the first position parameter (x1′, y1′) of the first area:

[0016] x1′=x 01 ×W0

[0017] y1′=y 01 ×H0

[0018] Among them, in the third size parameter (W, H), W represents the width of the shooting picture, and H represents the height of the shooting picture; x1≤W, y1≤H;

[0019] In the fourth size parameter (W0, H0), W0 represents the width of the virtual background source image, and H0 represents the height of the virtual background source image.

[0020] In a possible implementation of the first aspect, mapping the first size of the target frame in the captured image to the first size parameter of the first area in the virtual background source image includes: determining a normalized size parameter (w1, h1) of the target frame according to the first size (w1, h1) of the target frame in the captured image and a third size parameter (W, H) of the captured image. 01 ,h01 ):

[0021]

[0022]

[0023] According to the normalized size parameter (w 01 ,h 01 ) and the fourth size parameter (W0, H0) of the virtual background source image determine the first size parameter (w1′, h1′) of the first area:

[0024] w1′=w 01 ×W0

[0025] h1′=h 01 ×H0

[0026] Among them, the normalized size parameter (w 01 ,h 01 ) 01 represents the normalized width parameter, h 01 represents the normalized height parameter,

[0027] In the first size (w1, h1), w1 represents the width of the target box and h1 represents the height of the target box;

[0028] In the first size parameters (w1′, h1′), w1′ represents the width of the first region, and h1′ represents the height of the first region.

[0029] In a possible implementation of the first aspect, mapping the second position of the target frame in the captured image to a second position parameter of the second area in the virtual background source image includes:

[0030] The normalized position parameter (x2, y2) of the target frame is determined according to the coordinates (x2, y2) corresponding to the second position of the target frame in the captured image and the third size parameter (W, H) corresponding to the captured image. 02 ,y 02 ):

[0031]

[0032]

[0033] According to the normalized position parameter (x 02 ,y 02 ) and the fourth size parameter (W0, H0) corresponding to the virtual background source image determine the second position parameter (x2′, y2′) of the second area:

[0034] x2′=x 02 ×W0

[0035] y2′=y 02 ×H0

[0036] Among them, in the third size parameter (W, H), W represents the width of the shooting picture, and H represents the height of the shooting picture; x2≤W, y2≤H;

[0037] In the fourth size parameter (W0, H0), W0 represents the width of the virtual background source image, and H0 represents the height of the virtual background source image.

[0038] In a possible implementation of the first aspect, mapping the second size of the target frame in the captured image to a second size parameter of the second area in the virtual background source image includes:

[0039] The normalized size parameter (w2, h2) of the target frame is determined according to the second size (w2, h2) of the target frame in the captured image and the third size parameter (W, H) of the captured image. 02 ,h 02 ):

[0040]

[0041]

[0042] According to the normalized size parameter (w 02 ,h 02 ) and the fourth size parameter (W0, H0) of the virtual background source image determine the second size parameter (w2′, h2′) of the second area:

[0043] w2′=w 02 ×W0

[0044] h2′=h 02 ×H0

[0045] Among them, the normalized size parameter (w 02 ,h 02 ) 02 represents the normalized width parameter, h 02 Represents the normalized height parameter, the second size (w2, h2) in which w2 represents the width of the virtual background source image, h2 represents the height of the virtual background source image, the second size parameter (w2′, h2′) in which w2′ represents the width of the second area, and h2′ represents the height of the second area.

[0046] In a possible implementation of the first aspect above, the method also includes: determining the target width of the virtual background source image according to the ratio between the first depth of field corresponding to the captured picture and the second depth of field of the virtual background source image, and the product of the third width parameter of the captured picture; determining the target height of the virtual background source image according to the ratio between the first depth of field and the second depth of field, and the product of the third height parameter of the captured picture; adjusting the fourth width parameter of the virtual background source image to the target width; and adjusting the fourth height parameter of the virtual background source image to the target height.

[0047] Exemplarily, the second depth of field of the virtual background source image may be the current virtual depth of field corresponding to the virtual background source image described below. The target width and target height of the virtual background source image may be the target size of the virtual background source image described below. The ratio between the first depth of field corresponding to the captured image and the second depth of field of the virtual background source image may be the size change ratio S described below.

[0048] In a possible implementation of the first aspect above, the first depth of field corresponding to the captured image is between the third depth of field and the fourth depth of field, wherein the third depth of field is the maximum value of the depth of field of the captured image, and the fourth depth of field is the depth of field of the foreground target in the captured image.

[0049] For example, the maximum value of the depth of field of the captured image (third depth of field) may be the maximum value of the depth of field in the current video frame (current maximum depth of field D max ), the depth of field of the foreground target in the shooting picture (the fourth depth of field) can be the depth of field value of the speaker below (the depth of field of the speaker D p ), the first depth of field can be the target virtual depth of field D0 of the virtual background source image below. It can be understood that since the fourth depth of field will change accordingly with the movement of the foreground target, the first depth of field determined based on the third depth of field and the fourth depth of field is a dynamic value, and accordingly, the size change ratio S determined based on the first depth of field and the second depth of field is also a dynamic value, which will change accordingly according to the movement of the foreground target. Therefore, when the size of the virtual background source image is adjusted based on the size change ratio S, the size of the virtual background source image can be dynamically changed, and then, the dynamic adjustment of the depth of field of the virtual background source image is realized, so that the visual distance between the virtual background source image and the foreground target always conforms to the real scene in the captured picture. In this way, the picture after the background replacement is made more realistic and the user's visual experience is more comprehensively improved.

[0050] In a possible implementation of the first aspect above, a first projection mode of the virtual background source image is adjusted to a second projection mode of the captured image, wherein the first projection mode and the second projection mode include orthogonal projection or perspective projection.

[0051] For example, by unifying the projection method of the virtual background source image and the captured image, it is possible to avoid the visual disharmony caused by the inconsistent perspective effect of the video image after the background is replaced.

[0052] In a possible implementation of the first aspect above, a resolution of the virtual background source image is the same as a resolution of the captured image.

[0053] For example, the resolution of the virtual background source image may be consistent with the resolution of the captured image, so that the foreground object after the background is replaced has the same image granularity as the background, thereby making the display effect more uniform.

[0054] In a second aspect, the present application provides an electronic device, comprising: one or more processors; one or more memories; one or more memories storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the display method provided by the first aspect and various possible implementations of the first aspect.

[0055] In a third aspect, the present application provides a computer-readable medium having instructions stored thereon, which, when executed on a computer, causes the computer to execute the display method provided by the first aspect and various possible implementations of the first aspect.

[0056] In a fourth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a display method provided by any possible implementation of the first aspect.

[0057] The beneficial effects of the second to fourth aspects mentioned above can be referred to the relevant descriptions in the first aspect mentioned above and various possible implementations of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1a A close-up display effect diagram of a presenter of a display method provided by the present application is shown;

[0059] Figure 1b A display effect diagram of tracking the movement of a speaker in a display method provided by the present application is shown;

[0060] Figure 1c A schematic diagram of a shooting screen provided by the present application is shown;

[0061] Figure 1d A schematic diagram of a display screen provided by the present application is shown;

[0062] Figure 1e A schematic diagram showing changes in a display screen during movement of a presenter provided by the present application is shown;

[0063] Figure 1f A schematic diagram showing changes in a display screen during a close-up display of a speaker provided by the present application is shown;

[0064] Figure 2 A flow chart of a display method provided by the present application is shown;

[0065] Figure 3a A flowchart of another display method provided by the present application is shown;

[0066] Figure 3b The following is an operation flow chart of determining the close-up area where the speaker is located provided by the present application;

[0067] Figure 3c The operation flow chart of determining the virtual image area based on the close-up area of ​​the speaker provided by the present application is shown;

[0068] Figure 4a A schematic diagram of a calculation process for determining a close-up area of ​​a speaker provided by the present application is shown;

[0069] Figure 4b A schematic diagram of a calculation process for determining a virtual image area provided by the present application is shown;

[0070] Figure 4c A schematic diagram showing the process of separating the speaker's portrait and close-up area provided by the present application is shown;

[0071] Figure 4d A schematic diagram showing the process of merging the speaker's portrait and the virtual image area provided by the present application is shown;

[0072] Figure 5a A schematic diagram of the movement trajectory of a speaker in a broadcasting scenario provided by the present application is shown;

[0073] Figure 5b A schematic diagram showing the change of a virtual image area during the movement of a speaker provided by the present application is shown;

[0074] Figure 5c A schematic diagram of a process for generating target video frames corresponding to the speaker before and after movement provided by the present application is shown;

[0075] Figure 6a A schematic diagram of a directing scene showing a close-up of a speaker provided by the present application is shown;

[0076] Figure 6b A schematic diagram showing changes in a virtual image area during a close-up display of a speaker provided by the present application is shown;

[0077] Figure 6cA schematic diagram of a process for generating a target video frame corresponding to the front and back of a speaker in close-up display provided by the present application is shown;

[0078] Figure 7 A schematic diagram of a module interaction process of the display method provided by the present application is shown;

[0079] Figure 8a A flowchart of another display method provided by the present application is shown;

[0080] Figure 8b The following is an operation flow chart of adjusting the size of a virtual background source image provided by the present application;

[0081] Figure 8c A flowchart for determining a virtual background source image provided by the present application is shown;

[0082] Fig. 9 A schematic diagram of the module interaction process of another display method provided by the present application is shown;

[0083] Fig.10a A schematic diagram of the architecture of a video conferencing system with director and background replacement functions enabled provided by the present application is shown;

[0084] Fig.10b A schematic diagram of the architecture of another video conferencing system with director and background replacement functions enabled provided by the present application is shown;

[0085] Fig.11 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application is shown;

[0086] Fig.12 A software structure block diagram of a terminal device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0088] Figure 1a A schematic diagram showing changes in display screens displayed in a directing scenario is shown.

[0089] See also Figure 1aAs shown, in a directing scenario of a video conference, a mobile phone or other electronic device can display a display screen 101, and the background behind the speaker in the display screen 101 can be a replaced virtual background. In the process of displaying the display screen 101, the speaker's portrait in the display screen 101 may be gradually enlarged based on the directing function, so that the mobile phone or other electronic device can display a close-up image of the speaker's face. Correspondingly, the speaker can be enlarged from the close-up face size corresponding to the portrait 100a to the close-up face size corresponding to the portrait 100b. However, based on the image processing scheme currently applied to directing scenarios, the virtual background behind the speaker cannot be enlarged accordingly with the enlargement of the portrait. For example, it may always be displayed as Figure 1a The virtual background 101a shown in the figure will make the display screen feel more inconsistent after the background is replaced, and the user's visual experience will be poor.

[0090] Figure 1b A schematic diagram showing changes in display screens in another directing scenario is shown.

[0091] See also Figure 1b As shown, in another directing scenario, the display screen 102 displayed by an electronic device such as a mobile phone, the background behind the speaker in the display screen 102 can still be a replaced virtual background. The portrait of the speaker while walking may always be located in the center of the display screen 102 based on the directing function. For example, in the process of the speaker walking from position A to position B at the conference site, the speaker's portrait is always located in the center of the display screen 102, and the portrait may change from portrait 102a corresponding to position A to portrait 102b corresponding to position B. However, based on the image processing scheme currently applied to directing scenarios, the virtual background behind the speaker in this scenario cannot move accordingly with the movement of the speaker. For example, in the process of portrait 102a changing to portrait 102b, the virtual background behind the speaker may always be... Figure 1b As shown in the virtual background 101a. Therefore, the user cannot perceive the real movement of the speaker according to the change of the background, which will also cause the display screen after the background is replaced to feel more inconsistent, thereby resulting in a poor visual experience for the user.

[0092] Based on the above content, it can be imagined that if the background after the director function and background replacement can be changed accordingly with the change of the position and / or size of the speaker's portrait, the display screen can be more in line with the display requirements of the actual director scene, enhance the realism of the display screen, and improve the user's actual experience.

[0093] It can be understood that the technical solution of the present application is applicable to various video playback scenarios in which the display screen displayed to the user is different from the actual screen captured by the electronic device, for example, a directing scenario in which a portion of the screen captured by the electronic device is displayed to the user. Specifically, in the directing scenario, when the electronic device captures the host in the studio, the captured screen includes the host and the background behind the host (hereinafter referred to as the captured screen), and the directing function needs to display a smaller area of ​​the captured screen that includes the host (hereinafter referred to as the displayed screen). In this scenario, the displayed screen is a portion of the captured screen.

[0094] In addition, for another example, the scenario to which the technical solution of the present application is applicable may also be a video conferencing scenario in which the display screen displayed to the user is different from the actual screen captured by the electronic device, that is, a directing scenario in which a portion of the screen captured by the electronic device is displayed to the user. For example, in this directing scenario, when the electronic device captures the speaker at the conference site, the captured screen also includes the speaker and the background behind the speaker, that is, the captured screen. The directing function needs to display a smaller area of ​​the captured screen that includes the speaker, that is, the display screen. In this scenario, the display screen is still a portion of the captured screen. As an example, Figure 1c The following is a screenshot of the video conference scene above. Figure 1d The display screen in the above-mentioned video conference scene is shown, and the display screen is a part of the shooting screen.

[0095] In response to the above problems, the present application provides a display method. In the display method, in the video playback scenario mentioned above where the display screen displayed to the user is different from the actual screen captured by the electronic device, if it is necessary to replace the background of the display screen with a virtual background, the area (virtual image area) suitable for being the virtual background can be selected from the image providing the virtual background (hereinafter referred to as the virtual background source image) based on the position and size of the foreground target in the display screen (such as the host or speaker mentioned above) in the captured screen, and the size and / or position of the virtual image area selected from the virtual background source image can be adjusted in real time as the position and / or size of the foreground target in the captured screen changes. Specifically, the position and size of the foreground target in the captured screen can be mapped to the virtual background source image, and then the mapped area in the virtual background source image is determined as the aforementioned virtual image area.

[0096] It can be understood that since the change in the position and / or size of the foreground target in the captured image corresponds to the change in the position and / or size of the virtual image area in the virtual background source image, after replacing the background of the display screen with the virtual image area, the replaced background (virtual image area) can change with the position and / or size of the foreground target, thereby creating a realistic visual experience.

[0097] As an example, Figure 1e As shown, when the speaker moves, the position of the speaker in the shooting picture will change, and accordingly, the position of the virtual image area in the virtual background source image will also change accordingly, for example, from area 103a to area 103b. Then, after the background of the display picture is replaced, the visual experience of the virtual background moving accordingly with the person walking can be achieved.

[0098] Another example Figure 1f As shown, when the speaker is shown in close-up, the size of the speaker in the shooting picture will be reduced, and accordingly, the size of the virtual image area in the virtual background source image will also be reduced, for example, from area 104a to 104b. Furthermore, after the background of the display picture is replaced, the visual experience of the virtual background being enlarged accordingly with the enlargement of the person can be achieved.

[0099] In the above example, the display method provided by the present application can make the display characters and background in the director's scene more harmoniously integrated and more realistic, which is ultimately beneficial to improving the user viewing experience of electronic devices such as mobile phones.

[0100] In addition, before determining the area in the virtual background source image suitable for being the virtual background according to the position and / or size of the foreground target in the shooting picture, the display method provided by the present application can also determine the size change ratio of the virtual background source image based on the depth of field of the virtual background image and the depth of field of the actual image and the depth of field of the virtual background source image. Furthermore, the electronic device can dynamically adjust the size of the virtual background source image based on the above-determined size change ratio so that the virtual background source image presents a dynamic enlargement or reduction effect relative to the portrait of the speaker in the actual image. For example, when the virtual background source image is enlarged relative to the portrait of the speaker, a visual feeling that the distance between the virtual background source image and the speaker is gradually getting closer can be created; accordingly, when the virtual background source image is reduced relative to the portrait of the speaker, a visual feeling that the distance between the virtual background and the speaker is gradually getting farther can be created. Furthermore, in the virtual background source image whose size has been adjusted, the virtual image area is determined based on the above-mentioned normalized position and normalized size to adjust the visual distance between the virtual background and the speaker to be close to the visual distance between the background and the speaker in the actual image. In this way, the picture after the background replacement is made more realistic and the user's visual experience is more comprehensively enhanced.

[0101] It can be understood that the execution subject of the display method provided in the present application can be a terminal device and / or a server. The aforementioned terminal device can include, for example, but is not limited to, mobile phones, tablet computers, desktops, laptops, handheld computers, netbooks, and wearable devices such as augmented reality (AR) / virtual reality (VR) devices, smart TVs, smart watches, and other electronic devices with one or more processors; the aforementioned server includes, but is not limited to, a cloud server, a physical server, etc. When describing the specific execution content of each step below, the execution subject of each step will not be described in detail to avoid repetition.

[0102] Below, the specific implementation process of the display method provided by the present application will be described in detail in conjunction with specific embodiments.

[0103] Example 1

[0104] The embodiment of the present application specifically describes the specific implementation process of the display method provided by the present application to realize the movement and / or scaling of the virtual background corresponding to the movement and / or scaling of the presenter.

[0105] In the present application embodiment, firstly Figure 1a and Figure 1b The corresponding display method that cannot make the virtual background move and / or scale accordingly with the movement and / or scaling of the presenter is described in detail. Figure 2 The image processing flow chart shown in FIG. 1 includes:

[0106] 201: Get the current video frame.

[0107] Exemplarily, real-time video stream data of a video conference is acquired, and a current video frame is obtained based on the real-time video stream data. The current video frame may be an actual image captured by the aforementioned electronic device.

[0108] 202: Perform face detection based on the current video frame.

[0109] Exemplarily, face detection can be performed on the current video frame based on a deep learning model to identify the face area in the current video frame and output the position and size information of the face.

[0110] 203: Determine the speaker based on the face detection result and the audio data.

[0111] For example, the speaker's voice in the current video frame can be identified by audio processing technology, and the face area corresponding to the speaker can be determined in combination with the face detection result. At the same time, the speaker's position can be tracked and matched according to the movement trajectory of the speaker's voice and the face area corresponding to the speaker.

[0112] 204: Determine the close-up area of ​​the speaker.

[0113] Exemplarily, the current video frame may be cropped according to the speaker position information determined in the aforementioned step 203 to obtain a close-up area of ​​the speaker.

[0114] 205: Obtain a mask of the speaker based on the close-up region of the speaker.

[0115] For example, the close-up area of ​​the speaker may be segmented at the pixel level based on the deep learning model to obtain a mask of the speaker, which may be used for background replacement and portrait fusion operations in step 206 described below.

[0116] 206: Perform background replacement based on the speaker's mask and virtual background.

[0117] For example, the speaker's portrait can be separated from the current video frame according to the aforementioned speaker's mask, and placed in the center of the virtual background source image, so that the speaker's portrait can always be displayed in the center of the video screen. In this way, the background of the current video frame is replaced with the virtual background, and a target video frame after background replacement is generated. Here, since the background replacement scheme only superimposes the virtual background source image with the separated speaker's portrait, see Figure 1a and Figure 1b As shown in the background replacement effect diagram, the background in the target video frame generated based on this solution is always the virtual background source image 101a. The background cannot move and / or scale accordingly according to the movement of the speaker and / or the scaling of the speaker's portrait, which will result in a strong sense of incongruity in the target video frame after the background replacement.

[0118] In view of the problem that the target video frame after the background replacement has a strong sense of disharmony, the embodiment of the present application provides a display method to make the display effect of the target video frame more consistent with the real video conference scene, enhance the realism of the picture, and improve the user's visual experience. Figure 3a The image processing implementation flow chart provided in the embodiment of the present application is shown, and the implementation flow may include the following steps:

[0119] 301: Get the current video frame.

[0120] For example, the current video frame can be captured from the real-time video stream data collected at the conference site. The real-time video stream data can be a panoramic video of the conference site, and accordingly, the current video frame can be a panoramic image of the conference site. Here, the real-time video stream data collection device can be a camera or one or more cameras of a local terminal.

[0121] 302: Perform face detection based on the current video frame.

[0122] Exemplarily, face detection may be performed on the current video frame to obtain facial information of all participants in the current video frame. The facial information may include facial position information capable of determining the position of the face frame, size information, and posture information capable of representing the angle of the face.

[0123] 303: Determine the speaker based on the face detection result and the collected audio data.

[0124] Exemplarily, when collecting the aforementioned real-time video stream data, real-time audio data can be collected simultaneously based on a collection device. Here, the collection device for collecting audio data can be the same as the collection device for collecting the aforementioned real-time video stream data, for example, both can be cameras, or can be different from the collection device for collecting the aforementioned real-time video stream data, for example, it can be an audio collection device such as a microphone of a local terminal.

[0125] For example, see Figure 3b The operation flow chart for determining the close-up area where the speaker is located can be based on the real-time audio data collected by the microphone, and the sound source of the speaker can be located based on the audio data to generate the location coordinates of the speaker. Here, the processing logic for locating the sound source and generating the location coordinates can be integrated in the audio module. Then, based on the location coordinates and the result of the face detection in the aforementioned step 302, the sound and image matching is performed to obtain the close-up area of ​​the speaker based on the following step 304. Among them, the real-time video stream data for face detection in the aforementioned step 302 can be an RGB image collected by a camera.

[0126] Specifically, the speaker's audio can be determined based on the collected audio data. Here, no restrictive description is given to the method of determining the speaker's audio. In one embodiment, the voiceprint features of the audio data can be extracted based on the audio processing technology, and the audio of each participant can be determined based on the extracted voiceprint features. Therefore, the speaker's audio can be determined from all the participant's audios based on the continuous speaking time of the audio of each participant and the set time threshold. For example, the audio of a participant whose continuous speaking time exceeds 4 seconds can be determined as the speaker's audio.

[0127] Further, the speaker's facial information can be determined from the facial information of all the participants determined in the aforementioned step 302 based on the speaker's audio. Here, no limitation is given to the method of determining the speaker's facial information based on the speaker's audio. In one possible implementation, see Figure 3bIn the schematic diagram of the audio-visual matching process shown in the figure, after the speaker's audio is determined in the audio data collected by the microphone based on the above method, the speaker's audio can be localized to determine the speaker's location coordinates based on the sound source localization. Based on the above step 302, it can be known that the face information of the participants can be identified based on face detection in the current video frame collected by the camera. Therefore, the location coordinates determined by the sound source localization and the face information of the participants can be matched with the audio-visual to determine the speaker's face information.

[0128] 304: Determine the close-up area of ​​the speaker.

[0129] Exemplarily, after the face information of the speaker is determined based on the aforementioned step 303, a close-up area of ​​the speaker can be determined in the current video frame based on the face position information that can determine the position of the face frame, and the close-up area can be used as a video frame displayed in the video screen after the director function is turned on. For example, after determining the position of the face frame of the speaker, the face frame can be expanded based on a fixed size to obtain a close-up area of ​​a fixed size. Here, the fixed size can be determined based on the display ratio of the terminal display screen that displays the video conference. For example, the terminal that displays the video conference has a display ratio of 16:9, so a screen area with an aspect ratio of 16:9 can be determined in the current video frame based on the face frame of the speaker, and the aforementioned face frame can be located at the center of the screen area. Furthermore, the screen area can be used as a close-up area of ​​the speaker.

[0130] Here, no restrictive description is given to the specific method of determining the speaker's face frame at the center of the close-up area. In one embodiment, the center coordinates of the speaker's face frame in the current video frame can be determined, and then the aforementioned rectangular area of ​​fixed size can be determined with the center coordinates as the center, and the rectangular area can be used as the speaker's close-up area. It can be understood that since the center coordinates of the speaker's face frame are the center coordinates of the close-up area, the speaker can always be located at the center of the close-up area, thereby realizing the director function of always displaying the speaker in the center of the screen.

[0131] In another possible implementation, the specific position of the face frame in the close-up area can be set based on the face frame of the speaker and the actual display effect. For example, based on the test of the display effect, the length of the upper and lower edges of the face frame can be set to 20% of the close-up area, and the length of the left and right edges of the face frame can be set to 30% of the characteristic area, and the alignment of the face frame and the characteristic area can be set based on the test results to determine the close-up area of ​​the speaker. It can be understood that this method can also realize the director function of always displaying the speaker in the center of the screen.

[0132] Here, it should be made clear that the center of the screen involved in the director function means that when the user is watching the video screen with the director function turned on, the speaker is always in the center area of ​​the screen. For example, the left and right offset of the center point of the speaker's face frame from the center point of the display screen does not exceed the preset horizontal threshold, and the up and down offset does not exceed the vertical threshold. Exemplarily, the aforementioned horizontal threshold may be in the range of 0 to 20%, and the aforementioned vertical threshold may be in the range of 0 to 30%. Here, no restrictive description is made on the value ranges of the horizontal threshold and the vertical threshold.

[0133] In addition, after determining the close-up area corresponding to the speaker in the current video frame, the image features of the previous and next video frames of the current video frame can be matched to predict the movement vector of the speaker in the next video frame, and then the tracking frame of the speaker can be determined based on the movement vector and the preset tracking frame size, and then the position of the speaker in the next video frame can be predicted based on the tracking frame, so as to achieve the tracking of the speaker in the director function. Here, no restrictive description is made on the specific method of predicting the movement vector of the speaker in the next video frame based on the matching results of the image features of the previous and next video frames. In addition, the aforementioned tracking frame can be smoothed so that the visual effect of smooth display without stuttering can be achieved during the movement of the speaker or in the process of switching the close-up shot of the speaker. Here, no restrictive description is made on the specific smoothing algorithm.

[0134] 305: Normalize the close-up area based on the current video frame.

[0135] Exemplarily, the close-up region may be normalized based on the size of the current video frame to obtain normalized coordinates and normalized size of the close-up region. It is understood that based on the normalized coordinates and normalized size of the close-up region, the close-up region may be mapped in the virtual background source image source image in the following step 306 to obtain a virtual image region whose position and size correspond to the close-up region.

[0136] Specifically, see Figure 4a The schematic diagram of the calculation process for determining the close-up area of ​​the speaker is shown in FIG. 1 , taking the current video frame with a width of W and a height of H, and the close-up area with the coordinates of the upper left corner pixel point in the current video frame as (x, y), and a width of w and a height of h as an example, the normalized coordinates (x0, y0) of the upper left corner pixel point of the close-up area can be:

[0137]

[0138]

[0139] The normalized width w0 of the close-up area can be:

[0140]

[0141] The normalized height h0 of the close-up area can be:

[0142]

[0143] It can be understood that based on the normalized coordinates (x0, y0), normalized width w0, and normalized height h0 of the upper left corner pixel of the close-up area, when the size of the virtual background source image is inconsistent with that of the current video frame, the virtual image area whose position ratio and size ratio correspond to the close-up area can be determined in the virtual background source image.

[0144] Here, the aspect ratio of the virtual background source image may be consistent with the aspect ratio of the current video frame, so as to determine a virtual image region in the virtual background source image whose position ratio and size ratio are the same as those of the close-up region. If the aspect ratio of the virtual background source image is inconsistent with the aspect ratio of the current video frame, the aspect ratio of the virtual background source image may be adjusted to the aspect ratio of the current video frame. Alternatively, a virtual image region with a position ratio and size ratio similar to the normalized close-up region may be determined in the virtual background source image.

[0145] 306: Determine a virtual image region in the virtual background source image based on the normalized close-up region.

[0146] For example, see Figure 3c The operation flow chart of determining the virtual image area based on the speaker's close-up area is shown. After the speaker's close-up area is normalized based on the aforementioned step 305, the virtual image area mapping can be performed based on the normalized close-up area. Specifically, based on the normalized coordinates (x0, y0) of the upper left corner pixel of the aforementioned close-up area, the normalized width w0, the normalized height h0, and the size of the virtual background source image, the corresponding virtual image area can be determined in the virtual background source image. It can be understood that the normalized coordinates of the upper left corner pixel of the virtual image area are also (x0, y0), the normalized width is also w0, and the normalized height is also h0. See Figure 4b The schematic diagram of the calculation process for determining the virtual image area shown in FIG. 1 is a diagram of the calculation process for determining the virtual image area. Taking the virtual background source image with a width of W0 and a height of H0 in the aforementioned step 305 as an example, the coordinates (x′, y′) of the upper left corner pixel point of the virtual image area in the virtual background source image can be:

[0147] x′=x0×W0

[0148] y′=y0×H0

[0149] The width w′ of the virtual image area in the virtual background source image can be:

[0150] w′=w0×W0

[0151] The height h′ of the virtual image area in the virtual background source image can be:

[0152] h′=h0×H0

[0153] Furthermore, based on the coordinates (x′, y′), width w′, and width h′ of the upper left corner pixel of the virtual image area, the virtual image area can be determined in the virtual background source image, and the position and size of the virtual image area in the virtual background source image correspond to the position and size of the close-up area in the current video frame.

[0154] Therefore, when the director function is turned on to track the moving speaker, the position of the close-up area will move in the current video frame due to the movement of the speaker, so that the speaker is always located at the center of the close-up area. Correspondingly, the position of the virtual image area in the virtual background source image will also move accordingly. Furthermore, when the virtual image area is used as the replaced background based on the following step 308, a visual effect will be created that the background moves synchronously with the movement of the speaker, making the display image after the background replacement more realistic and enhancing the user's visual experience.

[0155] For another example, when the director function is turned on to switch the close-up shot of the speaker, the proportion of the close-up area in the current video frame will be reduced to create a visual effect of magnifying the person. Correspondingly, the proportion of the virtual image area in the virtual background source image will also be reduced accordingly. Furthermore, when the virtual image area is used as the replaced background based on the following step 308, a visual effect will be created that the background is synchronously magnified as the speaker is magnified, making the display screen after the background replacement more realistic and enhancing the user's visual experience.

[0156] 307: Obtain a mask of the speaker based on the close-up region of the speaker.

[0157] Exemplarily, the close-up area of ​​the speaker can be pixel-wise segmented based on the deep learning model to obtain the speaker's mask. Here, the mask can be used to block images other than the speaker's portrait in the current video frame, and then extract the speaker's portrait from the current video frame. Here, the algorithm for obtaining the speaker's mask includes but is not limited to a threshold-based image segmentation algorithm, such as the Sauvola algorithm, the Otsu algorithm, etc.; an edge detection-based image processing algorithm, such as the fast Fourier transformation (FFT) algorithm, the fast Hartley transform (FHT) algorithm; an image segmentation algorithm based on region growth, a convolutional neural network (CNN) algorithm based on deep learning, etc. Here, no restrictive description is given to the specific algorithm for determining the speaker's mask.

[0158] 308: Based on the speaker's mask, the speaker in the current video frame is merged with the virtual image region to obtain a target video frame after background replacement.

[0159] Exemplarily, after the speaker's mask is obtained based on the aforementioned step 307, see Figure 4c The schematic diagram of the separation process of the speaker portrait and the close-up area shown in FIG. 4 shows, based on the mask, the speaker portrait 400a in the current video frame can be separated out, and then, see Figure 4d The schematic diagram of the fusion process of the speaker's portrait and the virtual image area is shown, and the speaker's portrait 400a and the virtual image area 400b are pixel-fused to obtain a target video frame after background replacement.

[0160] In an example scenario, see Figure 5a The schematic diagram of the movement trajectory of the speaker is shown in FIG. 5 . The speaker can move from position 510 to position 511. During this process, the speaker close-up area determined based on the aforementioned step 305 can move from close-up area 510a to close-up area 511a. Figure 5b The virtual image area changes during the movement of the presenter shown in FIG. 3 , based on the virtual image area determined in step 306 above, can be moved from the replacement area 510b to the replacement area 511b. Further, based on step 308, the portrait 510c of the presenter corresponding to the position 510 and the portrait 511c of the presenter corresponding to the position 511 are obtained, see Figure 5cThe process diagram of generating the target video frames corresponding to the speaker before and after the movement is shown, after the portrait 510c is merged with the replacement area 510b, and the portrait 511c is merged with the replacement area 511b, the target video frame 510d corresponding to the position 510 and the target video frame 511d corresponding to the position 511 will be obtained. It can be understood that when the user watches the conference video between the target video frame 510d and the target video frame 511d, because the virtual image area moves from the replacement area 510b to the replacement area 511b, and the portrait posture of the speaker changes from the portrait 510c to the portrait 511c, the user can feel the realistic walking process of the speaker. In this way, the realism of the background replacement effect in the directing process is improved, and the user's visual experience is improved.

[0161] In another example scenario, see Figure 6a The schematic diagram of the broadcasting scene showing the speaker in close-up is shown. Based on the close-up area of ​​the speaker determined in step 304, the close-up area 610a can be reduced to the close-up area 611a. Figure 6b Schematic diagram of the change of the virtual image area during the close-up display of the speaker. The virtual image area determined based on the aforementioned step 306 can be reduced from the replacement area 610b to the replacement area 611b. Based on the aforementioned step 308, the portrait 610c of the speaker corresponding to the close-up area 610a and the portrait 611c of the speaker corresponding to the close-up area 611a are obtained. Figure 6c The process diagram of generating the target video frames corresponding to the close-up display of the speaker before and after the close-up display is shown, after the portrait 610c is merged with the replacement area 610b, and the portrait 611c is merged with the replacement area 611b, the target video frame 610 corresponding to the close-up display before and the target video frame 611 corresponding to the close-up display after will be obtained. It can be understood that when the user watches the conference video between the target video frame 610 and the target video frame 611, since the virtual image area changes from the replacement area 610b to the replacement area 611b, and the portrait of the speaker changes from the portrait 610c to the portrait 611c, the user can feel the realistic enlargement process of the speaker. In this way, the realism of the background replacement effect in the directing process is improved, and the user's visual experience is improved.

[0162] The module interaction process for implementing the display method provided by the aforementioned steps 301 to 308 is described in detail below with reference to the accompanying drawings.

[0163] See also Figure 7The module interaction diagram of the display method shown in the figure can implement the display method provided by the present application based on the AI ​​processing module 710, the adaptive module 720 and the system module 730. For example, the AI ​​processing module 710 may include a collection module 711, an audio-visual matching module 712, a tracking module 713, a portrait segmentation module 714, and a replacement module 715. The adaptive module 720 may include a background adaptive replacement module 721. The system module 730 may include a user operation module 731, a prefabricated background library module 732, and a background upload module 733.

[0164] Specifically, the acquisition module 711 may be used to acquire the aforementioned real-time video stream data and the aforementioned audio data at the conference site, and transmit the acquired real-time video stream data and audio data to the audio-visual matching module 712 .

[0165] The audio-visual matching module 712 may include a face detection module 712a and a sound source localization module 712b. The face detection module 712a may be used to determine the current video frame based on the received real-time video stream data, and perform face detection based on the current video frame. The face detection result is sent to the sound source localization module 712b. Here, the specific content of obtaining the current video frame can refer to the relevant description in the aforementioned step 301, and the specific content of performing face detection based on the current video frame can refer to the relevant description in the aforementioned step 302, which will not be repeated here.

[0166] Among them, the sound source localization module 712b can be used to determine the speaker's audio based on the received audio data, and determine the speaker's location coordinates based on the sound source localization of the speaker's audio. Furthermore, the speaker's face information can be determined in the received face detection result according to the location coordinates. Here, the specific content of the above-mentioned determination of the speaker's face information based on the audio data and the face detection result can refer to the relevant description in the above-mentioned step 303, which will not be repeated here. In addition, the sound source localization module 712b can also be used to send the speaker's face information to the tracking module 713.

[0167] The tracking module 713 can be used to determine the speaker's close-up area based on the received speaker's facial information, and track the speaker based on the close-up area to achieve the director function. Here, the specific content of the above-mentioned determination of the speaker's close-up area and tracking the speaker based on the close-up area can be referred to the relevant description in the above-mentioned step 304, which will not be repeated here. In addition, the tracking module 713 can also be used to send the speaker's close-up area to the portrait segmentation module 714 and the adaptive adjustment module 720.

[0168] The portrait segmentation module 714 can separate the portrait of the speaker from the current video frame based on the received close-up region of the speaker, and send the separated portrait of the speaker to the replacement module 715. Here, the specific content of separating the portrait of the speaker from the current video frame based on the received close-up region of the speaker can be referred to the relevant description in the aforementioned step 307 and step 308, which will not be repeated here.

[0169] The user operation module 731 can be used to receive user operations, which can include selecting to manually upload the aforementioned virtual background source image, or selecting to determine the virtual background source image in the aforementioned prefabricated background library. When the user selects to manually upload, the user operation module 731 can send the selection operation to the background upload module 733 through broadcasting or other means; when the user chooses to access the preset background library, the user operation module can send the selection operation to the prefabricated background library module 732 through event broadcasting or other means.

[0170] The background upload module 733 is used to receive a picture uploaded by a user, use the picture as a virtual background source image, and send the virtual background source image to the background adaptive replacement module 721 .

[0171] The prefabricated background library module 732 is used to determine the virtual background source image selected by the user in the preset background library, so as to send the virtual background source image to the background adaptive replacement module 721 .

[0172] The background adaptive replacement module 721 can be used to determine the virtual image area based on the current video frame, the virtual background source image and the close-up area. Here, the specific content of determining the virtual image area based on the current video frame, the virtual background source image and the close-up area can refer to the relevant descriptions in the above-mentioned steps 305 and 306, which will not be repeated here. In addition, the background adaptive replacement module 721 can also be used to send the virtual image area to the replacement module 715.

[0173] The replacement module 715 can be used to merge the received speaker's portrait with the virtual image area to obtain a target video frame after background replacement. Here, the specific content of merging the speaker's portrait with the virtual image area can be referred to the relevant description in the aforementioned step 308, which will not be repeated here.

[0174] It can be understood that based on the display method provided in the embodiment of the present application, after the director function and the background replacement function are turned on, the background can change accordingly with the position and / or size of the speaker's portrait, thereby enhancing the realism of the picture and improving the user's visual experience.

[0175] In conjunction with another embodiment 2, the following describes a specific implementation process of a display method for adjusting the depth of field of a virtual background to further enhance the realism of a picture based on the display method of embodiment 1 in another scenario where a director function and a background replacement function are turned on.

[0176] Example 2

[0177] The embodiment of the present application is based on the display method provided in the aforementioned embodiment 1, and further adjusts the size of the virtual background so that the background depth after background replacement corresponds to the actual background depth, so as to further improve the realism of the background replacement effect during the directing process.

[0178] The specific implementation process of the display method provided in the embodiment of the present application is described in detail below. Figure 8a The image processing implementation flow chart provided in the embodiment of the present application is shown, and the implementation flow may include the following steps:

[0179] 801: Get the current video frame.

[0180] It can be understood that the above process of obtaining the current video frame can be specifically referred to the relevant description of the above step 301, which will not be repeated here.

[0181] 802: Perform face detection based on the current video frame.

[0182] It can be understood that the above process of performing face detection based on the current video frame can be specifically referred to the relevant description of the aforementioned step 302, and will not be repeated here.

[0183] 803: Determine the speaker based on the face detection result and the collected audio data.

[0184] It can be understood that the process of determining the speaker based on the face detection result and the collected audio data can be specifically referred to the relevant description of the aforementioned step 303, which will not be described in detail here.

[0185] 804: Determine the close-up area of ​​the speaker.

[0186] It can be understood that the process of determining the close-up area of ​​the speaker can be specifically described in the aforementioned step 304, and will not be described in detail here.

[0187] 805: Adjust the size of the virtual background source image based on the current video frame and the close-up area.

[0188] Exemplarily, the user can manually upload a virtual background source image that he wants to display as the speaker's conference background, or preset several background images in the preset background library for the user to select one as the aforementioned virtual background source image. Here, it can be understood that the visual distance between the speaker and the replaced background will affect the user's sense of reality when watching the video. For example, if the visual distance between the replaced background and the speaker is much larger than the actual distance between the conference background and the speaker in the current video frame, when the background is replaced, the size of the replaced background may be inconsistent with the size of the speaker's portrait. For example, the size of the replaced background may be too small, resulting in the visual distance between the replaced background and the speaker being too far, making the picture look more false, affecting the user's perception.

[0189] Therefore, see Figure 8b As shown in the operation flow chart, the depth of field of the current video frame and the virtual background source image can be first estimated. Here, the depth of field estimation includes monocular depth of field estimation for the monocular image; then, the background scale of the virtual background source image is adjusted based on the depth of field estimation result so that the depth of field of the replaced background matches the depth of field of the actual scene; further, the projection method of the virtual background source image with the adjusted scale can be aligned relative to the current video frame to avoid the sense of incongruity of the picture caused by different projection methods.

[0190] The detailed process of resizing the virtual background source image based on the current video frame and close-up area can be found in the following Figure 8c The steps shown and described are not limiting.

[0191] 806: Normalize the close-up area based on the current video frame.

[0192] It can be understood that the above process of normalizing the close-up area based on the current video frame can be specifically referred to the relevant description of the above step 305, which will not be described in detail here.

[0193] 807: Determine a virtual image region in the virtual background source image based on the normalized close-up region.

[0194] It can be understood that the process of determining the virtual image area in the virtual background source image based on the normalized close-up area can be specifically referred to the relevant description of the aforementioned step 306, which will not be repeated here. Here, the virtual background in the aforementioned step 306 includes the aforementioned virtual background source image.

[0195] 808: Obtain a mask of the speaker based on the close-up region of the speaker.

[0196] It can be understood that the process of obtaining the speaker's mask based on the speaker's close-up area may be specifically described in the aforementioned step 307, and will not be described in detail here.

[0197] 809: Based on the speaker's mask, the speaker in the current video frame is merged with the virtual image region to obtain a target video frame after background replacement.

[0198] It can be understood that the process of fusing the speaker in the current video frame with the virtual image area based on the speaker's mask to obtain the target video frame after background replacement can be specifically referred to the relevant description of the aforementioned step 308, which will not be repeated here.

[0199] Regarding the size adjustment process described in the aforementioned step 805, Figure 8c The flowchart shown is described in detail.

[0200] Specifically, see Figure 8c The flowchart for determining the virtual background source image shown in the figure can estimate the depth of field of the current video frame through the depth of field estimation method based on the following steps 805a to 805f, determine the depth of field of the replaced target background based on the depth of field of the speaker and the depth of field of the real background, and resize the virtual background source image based on the target background depth of field and the original depth of field of the aforementioned virtual background source image, so that the depth of field of the replaced background conforms to the actual scene, thereby improving the realism of the picture.

[0201] 805a: Determine the current maximum depth of field and the speaker's depth of field based on the current video frame.

[0202] Exemplarily, the depth of field of the current video frame can be estimated based on the monocular depth of field estimation. Here, no restrictive description is given to the monocular depth of field estimation method adopted. For example, the monocular depth of field estimation method adopted may include a traditional monocular depth of field estimation method based on motion estimation (ME), and the depth of field of the object to be measured in the image is determined by analyzing the motion trajectory of the object to be measured in the current video frame. Specifically, a frame sequence of several video frames including the current video frame can be obtained, and the motion trajectory of the object to be measured in the frame sequence can be analyzed based on methods such as optical flow and feature matching, and then the depth of field of the object to be measured is calculated according to the aforementioned motion trajectory based on methods such as triangulation.

[0203] For another example, the monocular depth of field estimation method adopted may include a monocular depth of field estimation method based on deep learning, and a deep neural network (DNN) model is trained by a large amount of annotated data to learn the mapping from a single image to a depth map. Specifically, the current video frame can be mapped to a depth map based on a CNN using an encoder-decoder structure, wherein the encoder is used to extract the image features of the current video frame, and the decoder is used to map the aforementioned image features to the depth map. In addition, the accuracy of the aforementioned depth map can be improved based on the contextual information in the current video frame in combination with a recurrent neural network (RNN) or an attention mechanism. The depth of field of the object to be measured in the current video frame is determined based on the depth map. Here, the depth of field of the aforementioned object to be measured may be the distance information between the camera that photographs the object to be measured and the object to be measured.

[0204] Furthermore, based on the above method, the maximum depth of field in the current video frame can be determined as the current maximum depth of field, and the depth of field value of the speaker can be determined as the speaker's depth of field. It can be understood that the current maximum depth of field can be the depth of field corresponding to the actual background in the current video frame.

[0205] 805b: Determine a target virtual depth of field of the virtual background source image based on the current maximum depth of field and the depth of field of the speaker.

[0206] For example, based on the current maximum depth of field D max And the speaker's depth of field D p The target virtual depth of field D0 of the virtual background source image can be determined. It can be understood that the target virtual depth of field D0 can be greater than the speaker's depth of field D p , in order to achieve the visual effect that the virtual background source image is located behind the speaker, the target virtual depth of field D0 may not exceed the current maximum depth of field D max , to meet the actual video conferencing scenario. Then, the value range of the target virtual depth of field D0 can be: D p <D0<D max Furthermore, the target virtual depth of field D0 can be set to be equal to the current maximum depth of field D max , so that the visual distance after background replacement is more in line with the actual video conference scene. Here, no restrictive description is given on the specific value of the target virtual depth of field D0.

[0207] 805c: Determine the current virtual depth of field based on the acquired virtual background source image.

[0208] Exemplarily, based on the aforementioned step 805, it can be known that the virtual background source image can be manually uploaded by the user, or can be preset in a preset background library.

[0209] Here, after obtaining the virtual background source image selected by the user, the current virtual depth of field corresponding to the virtual background source image can be determined based on the monocular depth of field estimation method in the aforementioned step 805a, and the current virtual depth of field can be the distance information between the camera that shoots the virtual background source image and the object farthest from the camera. For example, the virtual background source image selected by the user is a classroom scene, and the classroom scene includes objects such as desks, podiums, and blackboards, wherein the distance between the camera that shoots the virtual background source image and the blackboard is the farthest, then the aforementioned current virtual depth of field can be the depth of field corresponding to the blackboard, that is, it can be the distance information between the blackboard and the camera that shoots the virtual background source image.

[0210] 805d: Determine a size change ratio of the virtual background source image based on the target virtual depth of field and the current virtual depth of field.

[0211] Exemplarily, the size change ratio S of the virtual background source image may be determined based on the target virtual depth of field D0 and the current virtual depth of field D1:

[0212]

[0213] Here, it can be understood that when the target virtual depth of field D0 is greater than the current virtual depth of field D1, it means that the distance between the virtual background source image and the speaker is far, which may cause a false sense of disharmony between the replaced background and the speaker. At this time, the size change ratio S>1, and the virtual background source image can be enlarged based on the aforementioned step 305. It can be understood that based on the perspective principle of near big and far small, enlarging the virtual background source image relative to the speaker can create a visual effect of reducing the distance between the virtual background source image and the speaker, so that the visual distance between the replaced background and the speaker is more in line with the real video conference scene.

[0214] On the contrary, when the target virtual depth of field D0 is less than the current virtual depth of field D1, it means that the distance between the virtual background source image and the speaker is small, which may cause a false sense of disharmony between the replaced background and the speaker. At this time, the size change ratio S<1, and the virtual background source image can be reduced based on the following step 805e. It can be understood that based on the perspective principle of near big and far small, the reduction of the virtual background source image relative to the speaker can create a visual effect that the distance between the virtual background source image and the speaker becomes farther, so that the visual distance between the replaced background and the speaker is more in line with the real video conference scene, and the user's visual sense of reality is enhanced in the above way.

[0215] 805e: Determine the target size of the virtual background source image based on the size change ratio and the size of the current video frame to obtain a virtual background source image after size adjustment.

[0216] For example, the virtual background source image can be scaled based on the size change ratio S and the size of the current video frame to obtain a resized virtual background source image. Specifically, taking the current video frame with a width of W and a height of H as an example, the width W0 and the height H0 of the virtual background source image can be:

[0217] W0=S×W

[0218] H0=S×H

[0219] Here, the size of the virtual background source image is determined based on the size change ratio S and the size of the current video frame. It can be understood that since the speaker is located in the current video frame, determining the size of the virtual background source image based on the size of the current video frame can make the virtual background source image relatively enlarged or relatively reduced relative to the speaker, thereby creating a visual effect that the virtual background source image is farther or closer to the speaker, so that the visual distance between the replaced background and the speaker is more in line with the real video conferencing scene.

[0220] 805f: Adjust the projection mode of the virtual background source image based on the projection mode of the current video frame.

[0221] Exemplarily, if the projection mode of the virtual background source image is inconsistent with the projection mode of the current video frame, the projection mode of the virtual background source image can be adjusted to the projection mode of the current video frame to enhance the visual effect after replacement.

[0222] It can be understood that the projection methods for capturing videos include plane projection and curved surface projection. Common curved surface projections include cylindrical projection and spherical projection, which are mostly used to build panoramic maps in panoramic cameras. In the application scenario of video conferencing, screen projection is usually used to capture conference videos. Common plane projections include orthogonal projection and perspective projection. Among them, the angle between the projection line of orthogonal projection and the surface of the object is 90 degrees. The size and shape of the object on the projection plane are the same as the actual object, and there is no perspective effect; the projection lines of perspective projection all project the object from a projection center to a single projection surface, so that the size and shape of the object on the projection plane may be different from the actual object, and there is a perspective effect.

[0223] Based on the above, if the projection mode of the virtual background source image is inconsistent with the projection mode of the current video frame, replacing the real background of the current video frame with the virtual background source image may cause the perspective effect of the picture to be inconsistent, thereby creating a visual sense of disharmony. Therefore, the projection mode of the virtual background source image can be adjusted to the projection mode of the current video frame to enhance the realism of the replaced picture.

[0224] Specifically, the projection mode of the virtual background source image can be determined based on the camera parameters of the camera that shoots the virtual background source image, and the projection mode of the current video frame can be determined based on the camera parameters of the camera that collects real-time video stream data, and then it is determined whether the projection mode of the virtual background source image is consistent with the projection mode of the current video frame. For example, if the projection mode of the virtual background source image is orthogonal projection, and the projection mode of the current video frame is perspective projection, the projection mode of the virtual background source image can be adjusted to perspective projection based on the transformation matrix M that converts the orthogonal projection into perspective projection. Among them, the transformation matrix M is:

[0225]

[0226] Among them, n represents the front of the regular cube, f represents the back of the regular cube, p represents perspective, and o represents orthogonal.

[0227] The module interaction process for implementing the display method provided by the aforementioned steps 801 to 809 is described in detail below with reference to the accompanying drawings.

[0228] See also Fig. 9 The module interaction diagram of the display method shown in the figure can implement the display method provided by the present application based on the AI ​​processing module 910, the adaptive module 920 and the system module 930. Exemplarily, the adaptive module 920 may include a size adaptive adjustment module 921, a projection mode adaptive adjustment module 922, and a background adaptive replacement module 923. It can be understood that the specific modules included in the AI ​​processing module 910 and the system module 930 can refer to the aforementioned Figure 7 The relevant description is not repeated here.

[0229] Here, the specific description and interaction process of the acquisition module 911, the sound image matching module 912, the face detection module 912a, the sound source localization module 912b, and the tracking module 913 can be referred to in the aforementioned Figure 7 The relevant description is not repeated here.

[0230] Specifically, the user operation module 931 can be used to receive the user's selection operation, which can include selecting to manually upload the aforementioned virtual background source image, or selecting to determine the virtual background source image in the aforementioned prefabricated background library. When the user selects manual upload, the user operation module 931 can send the selection operation to the background upload module 933 through broadcasting or the like; when the user chooses to access the preset background library, the user operation module 931 can send the selection operation to the prefabricated background library module 932 through event broadcasting or the like.

[0231] The background uploading module 933 is used to receive a picture uploaded by a user, use the picture as a virtual background source image, and send the virtual background source image to the size adaptive adjustment module 921 .

[0232] The prefabricated background library module 932 is used to determine the virtual background selected by the user in the preset background library, to use the virtual background as a virtual background source image, and to send the virtual background source image to the size adaptive adjustment module 921 .

[0233] After receiving the speaker close-up area sent by the tracking module 913 and the virtual background source image sent by the background upload module 933 or the prefabricated background library module 932, the size adaptive adjustment module 921 adjusts the size of the aforementioned virtual background source image based on the close-up area and the current video frame to obtain the virtual background source image. Here, the specific content of the size adjustment of the virtual background source image based on the current video frame and the close-up area can refer to the relevant description of the aforementioned steps 805a to 805e, which will not be repeated here. In addition, the size adaptive adjustment module 921 can also be used to send the virtual background source image to the projection mode adaptive adjustment module 922.

[0234] The projection mode adaptive adjustment module 922 can be used to adjust the projection mode of the virtual background source image based on the projection mode of the current video frame. Here, the specific content of adjusting the projection mode of the virtual background source image based on the projection mode of the current video frame can refer to the relevant description in the aforementioned step 805f, which will not be repeated here. In addition, the projection mode adaptive adjustment module 922 can also be used to send the virtual background source image after the projection mode is adjusted to the background adaptive replacement module 923.

[0235] The background adaptive replacement module 923 may be used to determine the virtual image area based on the current video frame, the virtual background source image and the close-up area. Here, the specific content of determining the virtual image area based on the current video frame, the virtual background source image and the close-up area may refer to the related descriptions in the aforementioned steps 806 to 807, which will not be repeated here. In addition, the background adaptive replacement module 923 may also be used to send the virtual image area to the replacement module 915.

[0236] The portrait segmentation module 914 can separate the portrait of the speaker from the current video frame based on the received close-up region of the speaker, and send the separated portrait of the speaker to the replacement module 915. Here, the specific content of separating the portrait of the speaker from the current video frame based on the received close-up region of the speaker can be referred to the relevant description in the aforementioned step 808, which will not be repeated here.

[0237] The replacement module 915 can be used to merge the speaker portrait received from the portrait segmentation module 914 with the virtual image area to obtain a target video frame after background replacement. Here, the specific content of merging the speaker portrait with the virtual image area can be referred to the relevant description in the aforementioned step 809, which will not be repeated here.

[0238] Next, based on the above description of the module interaction process for implementing the display method provided by the present application, the system architecture for implementing the display method provided by the present application is specifically described.

[0239] See also Fig.10a The schematic diagram of the architecture of a video conferencing system with director and background replacement functions enabled is shown. The video conferencing system can implement the display method provided in the present application based on a local terminal and a remote terminal.

[0240] Exemplarily, the local terminal may include the aforementioned AI processing module, the aforementioned adaptive module, and the aforementioned system module. Here, the specific structure and functions of the AI ​​processing module, the adaptive module, and the system module refer to the aforementioned Figure 7 and Fig. 9 The specific description will not be repeated here.

[0241] Specifically, the local terminal can collect real-time video stream data and audio data based on the acquisition module in the AI ​​processing module, and can display the target video frame obtained based on the real-time video stream data and audio data based on the AI ​​processing module, the adaptive module and the system module on the display of the local terminal, and encode the target video frame and send it to the remote terminal, so that the remote terminal can display the target video frame on the display of the remote terminal after decoding the target video frame, thereby realizing the background replacement in the directing process of the remote video conference. It can be understood that the aforementioned local terminal may include the terminal equipment used by the speaker of the video conference and the participants in the same space, and accordingly, the aforementioned remote terminal may include the terminal equipment used by the participants in different spaces from the speaker.

[0242] See also Fig.10b Another architecture diagram of a video conferencing system with director and background replacement functions enabled is shown. The video conferencing system can implement the display method provided in the present application based on a local terminal, a server, and a remote terminal.

[0243] For example, the local terminal may include the acquisition module in the aforementioned AI processing module. The server may include the audio-visual matching module, the tracking module, the portrait segmentation module, the replacement module, the aforementioned adaptive module, and the aforementioned system module in the aforementioned AI processing module. Here, the specific structure and functions of the AI ​​processing module, the adaptive module, and the system module refer to the aforementioned Figure 7 and Fig. 9The specific description will not be repeated here.

[0244] Specifically, the local terminal can collect the aforementioned real-time video stream data and audio data based on the acquisition module, and send the real-time video stream data and audio data to the server. The server can determine the target video frame based on the audio-visual matching module, tracking module, portrait segmentation module, replacement module, adaptive module and system module according to the received real-time video stream data and audio data. And encode the target video frame and send it to the local terminal and the remote terminal, so that the local terminal and the remote terminal can display the target video frame on the display of the local terminal and the remote terminal after decoding the target video frame, so as to realize the background replacement in the directing process of the remote video conference.

[0245] Fig.11 According to an embodiment of the present application, a schematic diagram of the structure of a terminal device 100 is shown. Here, the terminal device 100 includes the local terminal and the remote terminal.

[0246] The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0247] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0248] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0249] The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. In the embodiment of the present application, the controller can control the execution of the instructions of the aforementioned steps 301 to 308 and steps 701 to 709.

[0250] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0251] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0252] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0253] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to implement the function of answering calls via a Bluetooth headset. Both the above-mentioned I2S interface and the above-mentioned PCM interface can be used for audio communication.

[0254] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0255] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0256] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0257] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0258] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0259] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0260] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0261] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0262] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0263] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0264] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0265] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0266] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0267] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0268] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0269] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0270] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0271] Fig.12 It is a software structure block diagram of the terminal device according to an embodiment of the present invention.

[0272] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0273] like Fig.12 As shown, the application layer may include a series of application packages, wherein the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0274] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0275] like Fig.12 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0276] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0277] Content providers are used to store and retrieve data and make it accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0278] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0279] The phone manager is used to provide communication functions for terminal devices, such as the management of call status (including answering, hanging up, etc.).

[0280] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0281] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0282] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0283] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0284] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0285] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as openGL ES), 2D graphics engine (such as SGL), etc.

[0286] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0287] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0288] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0289] A 2D graphics engine is a drawing engine for 2D drawings.

[0290] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0291] The embodiments of the present application also provide a computer program product for implementing the display methods provided in the above embodiments.

[0292] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program module or module code executed on a programmable system, and the programmable system includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0293] A computer program module or module code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0294] Module code can be implemented with high-level modular language or object-oriented programming language to communicate with the processing system. When necessary, module code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0295] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0296] References to "one embodiment" or "an embodiment" in the specification mean that the specific features, structures, or characteristics described in conjunction with the embodiment are included in at least one exemplary implementation or technology disclosed according to the embodiment of the present application. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily all refer to the same embodiment.

[0297] The disclosure of the embodiment of the present application also relates to an operating device for executing the text. The device can be specially constructed for the required purpose or it can include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer-readable medium, such as, but not limited to any type of disk, including a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application-specific integrated circuit (ASIC) or any type of medium suitable for storing electronic instructions, and each can be coupled to a computer system bus. In addition, the computer mentioned in the specification may include a single processor or may be an architecture involving multiple processors for increased computing power.

[0298] In addition, the language used in this specification has been primarily selected for readability and instructional purposes and may not be selected to describe or limit the disclosed subject matter. Therefore, the present application embodiment disclosure is intended to illustrate rather than limit the scope of the concepts discussed herein.

Claims

1. A display method, applied to an electronic device, characterized in that: The method comprises: Displaying a video screen in a first application interface, wherein the video screen includes a first virtual background and a foreground object, and the foreground object has a first display parameter; It is detected that a first display parameter of a foreground object in the video screen is changed to a second display parameter, wherein the first display parameter and the second display parameter include a position parameter and a size parameter; The first virtual background is replaced with a second virtual background, wherein the first virtual background and the second virtual background are respectively a first area and a second area of ​​a virtual background source image, and a size of the second area and a position thereof in the virtual background source image are related to the second display parameter.

2. The method according to claim 1, characterized in that The first display parameter and the second display parameter are related to the position and size of a target frame that frames the foreground target in the captured image, and the method includes: determining the first display parameter according to a first position and a first size of the target frame in the captured image; The second display parameter is determined according to a second position and a second size of the target frame in the captured image.

3. The method according to claim 2, characterized in that The replacing the first virtual background with the second virtual background comprises: Mapping a first position of the target frame in the captured image to a first position parameter of the first region in the virtual background source image, and mapping a first size of the target frame in the captured image to a first size parameter of the first region in the virtual background source image; Mapping a second position of the target frame in the captured image to a second position parameter of the second region in the virtual background source image, and mapping a second size of the target frame in the captured image to a second size parameter of the second region in the virtual background source image; A picture within a first area framed in the virtual background source image based on the first position parameter and the first size parameter is replaced by a picture within a second area framed in the virtual background source image based on the second position parameter and the second size parameter.

4. The method according to claim 3, characterized in that The mapping of the first position of the target frame in the captured image to a first position parameter of the first area in the virtual background source image includes: The normalized position parameter (x1, y1) of the target frame is determined according to the coordinates (x1, y1) corresponding to the first position of the target frame in the captured image and the third size parameter (W, H) corresponding to the captured image. 01 ,y 01 ): According to the normalized position parameter (x 01 ,y 01 ) and the fourth size parameter (W0, H0) corresponding to the virtual background source image determine the first position parameter (x1′, y1′) of the first area: x1′=x 01 ×W0 y1′=y 01 ×H0 Wherein, in the third size parameter (W, H), W represents the width of the shooting picture, H represents the height of the shooting picture, and x1≤W, y1≤H; In the fourth size parameter (W0, H0), W0 represents the width of the virtual background source image, and H0 represents the height of the virtual background source image.

5. The method according to claim 4, characterized in that The mapping the first size of the target frame in the captured image to a first size parameter of the first area in the virtual background source image includes: The normalized size parameter (w1, h1) of the target frame is determined according to the first size (w1, h1) of the target frame in the captured image and the third size parameter (W, H) of the captured image. 01 ,h 01 ): According to the normalized size parameter (w 01 ,h 01 ) and the fourth size parameter (W0, H0) of the virtual background source image determine the first size parameter (w1′, h1′) of the first area: w1′=w 01 ×W0 h1′=h 01 ×H0 Among them, the normalized size parameter (w 01 ,h 01 ) 01 represents the normalized width parameter, h 01 represents the normalized height parameter; In the first size (w1, h1), w1 represents the width of the target frame, and h1 represents the height of the target frame; In the first size parameters (w1′, h1′), w1′ represents the width of the first region, and h1′ represents the height of the first region.

6. The method according to claim 3, characterized in that The mapping the second position of the target frame in the captured image to a second position parameter of the second area in the virtual background source image includes: The normalized position parameter (x2, y2) of the target frame is determined according to the coordinates (x2, y2) corresponding to the second position of the target frame in the captured image and the third size parameter (W, H) corresponding to the captured image. 02 ,y 02 ): According to the normalized position parameter (x 02 ,y 02 ) and the fourth size parameter (W0, H0) corresponding to the virtual background source image determine the second position parameter (x2′, y2′) of the second area: x2′=x 02 ×W0 y2′=y 02 ×H0 Wherein, in the third size parameter (W, H), W represents the width of the shooting picture, H represents the height of the shooting picture, and x2≤W, y2≤H; In the fourth size parameter (W0, H0), W0 represents the width of the virtual background source image, and H0 represents the height of the virtual background source image.

7. The method according to claim 6, characterized in that The mapping the second size of the target frame in the captured image to a second size parameter of the second area in the virtual background source image includes: The normalized size parameter (w2, h2) of the target frame is determined according to the second size (w2, h2) of the target frame in the captured image and the third size parameter (W, H) of the captured image. 02 ,h 02 ): According to the normalized size parameter (w 02 ,h 02 ) and the fourth size parameter (W0, H0) of the virtual background source image determine the second size parameter (w2′, h2′) of the second area: w2′=w 02 ×W0 h2′=h 02 ×H0 Among them, the normalized size parameter (w 02 ,h 02 ) 02 represents the normalized width parameter, h 02 Represents a normalized height parameter, wherein the w2 in the second size (w2, h2) represents the width of the virtual background source image, and h2 represents the height of the virtual background source image, and the w2′ in the second size parameter (w2′, h2′) represents the width of the second area, and h2′ represents the height of the second area.

8. The method according to claim 2, characterized in that: The method further comprises: Determining a depth of field ratio according to a first depth of field corresponding to the captured image and a second depth of field of the virtual background source image, and determining a target width of the virtual background source image according to a product of the depth of field ratio and a width of the captured image; Determining a target height of the virtual background source image according to a product of the depth of field ratio and the height of the captured image; The width of the virtual background source image is adjusted to the target width, and the height of the virtual background source image is adjusted to the target height.

9. The method according to claim 8, characterized in that The first depth of field corresponding to the photographed picture is between the third depth of field and the fourth depth of field, wherein the third depth of field is the maximum value of the depth of field of the photographed picture; The fourth depth of field is the depth of field of the foreground object in the captured image.

10. The method according to any one of claims 2 to 9, characterized in that The first projection mode of the virtual background source image is adjusted to the second projection mode of the shooting picture, wherein the first projection mode and the second projection mode include orthogonal projection or perspective projection.

11. The method according to any one of claims 2 to 10, characterized in that The resolution of the virtual background source image is the same as the resolution of the captured image.

12. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the display method described in any one of claims 1 to 11.

13. A computer readable medium, characterized in that The readable medium stores instructions, which, when executed on a computer, enable the computer to execute the display method according to any one of claims 1 to 11.

Images