Live broadcast picture display method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a live broadcast picture display method and device, equipment and a storage medium, which are used for creating an augmented reality image by using a real scene image acquired by a terminal and a virtual decoration object, and taking the augmented reality image as a live broadcast background of an anchor object. Therefore, the live broadcast background of the anchor object interacts with the real environment, and the live broadcast background is enriched. Comprising the steps that a first augmented reality live broadcast picture is displayed, and the first augmented reality live broadcast picture comprises a first augmented reality image and an anchor object; and in response to conversion from a first reality scene image currently acquired by the first terminal to a second reality scene image, displaying a second augmented reality live broadcast image, the second augmented reality live broadcast image including the second augmented reality image and the anchor object. The technical scheme provided by the invention can be applied to the fields of artificial intelligence, cloud technology, computer vision and the like.

Description

Technical Field

[0001] The present application relates to the field of interface display technology, and in particular to a live screen display method, device, equipment and storage medium. Background Art

[0002] Augmented Reality (AR) technology is a technology that calculates the position and angle of camera images in real time and adds corresponding images, videos, and 3D models. The goal of this technology is to overlay the virtual world on the real world on the screen and interact with it, thereby applying virtual information to the real world and being perceived by human senses, thereby achieving a sensory experience beyond reality.

[0003] With the development of live broadcasting business, more and more virtual objects are starting to broadcast live. The live broadcast room of such virtual objects is a virtual environment, and the live broadcast background cannot be changed directly, and the live broadcast background is too monotonous. Summary of the invention

[0004] The embodiments of the present application provide a live screen display method, apparatus, device and storage medium for creating an augmented reality image using a real scene image acquired by a terminal and a virtual decoration object, and using the augmented reality image as the live broadcast background of the anchor object, thereby enabling the live broadcast background of the anchor object to interact with the real environment and enrich the live broadcast background.

[0005] In view of this, on the one hand, the present application provides a live broadcast screen display method, including: displaying a first augmented reality live broadcast screen, the first augmented reality live broadcast screen including a first augmented reality image and a host object, the first augmented reality image being created based on a first reality scene image currently acquired by the first terminal and a first virtual decoration object; in response to the first reality scene image currently acquired by the first terminal being transformed into a second reality scene image, displaying a second augmented reality live broadcast screen, the second augmented reality live broadcast screen including a second augmented reality image and the host object, the second augmented reality image being created based on the second reality scene image and the second virtual decoration object, the first reality scene image and the second reality scene image being obtained by the first terminal shooting the target reality scene.

[0006] On the other hand, the present application provides a live screen display device, including: a display module, used to display a first augmented reality live screen, the first augmented reality live screen including a first augmented reality image and a host object, the first augmented reality image is created based on a first reality scene image currently acquired by the first terminal and a first virtual decoration object;

[0007] A processing module is used to determine whether the first real scene image currently acquired by the first terminal is transformed into a second real scene image

[0008] The display module is used to display a second augmented reality live screen in response to the first reality scene image currently obtained by the first terminal being transformed into a second reality scene image. The second augmented reality live screen includes a second augmented reality image and the anchor object. The second augmented reality image is created based on the second reality scene image and the second virtual decoration object. The first reality scene image and the second reality scene image are obtained by the first terminal shooting the target reality scene.

[0009] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0010] A processing module, used for identifying a first scene object in the first real scene image;

[0011] A display module, used for displaying a decoration prompt control of the first scene object;

[0012] The processing module is used to decorate the first scene object with the first virtual decoration object in response to the operation of the decoration prompt control of the first scene object; and create the first augmented reality image based on the first virtual decoration object and the first real scene image.

[0013] In one possible design, in another implementation of another aspect of the embodiment of the present application, the device further includes:

[0014] a processing module, configured to identify a second scene object in the second real scene image; if the second scene object is the same as the first scene object, retain the first virtual decoration object and use the first virtual decoration object as the second virtual decoration object; if the second scene object is not the same as the first scene object, delete the first virtual decoration object;

[0015] A display module, used for displaying a decoration prompt control of the second scene object;

[0016] The processing module is used to decorate the second scene object with a second virtual decoration object in response to an operation on a decoration prompt control of the second scene object; and to create the first augmented reality image based on the second virtual decoration object and the second real scene image.

[0017] In one possible design, in another implementation of another aspect of the embodiment of the present application, the device further includes:

[0018] A processing module, configured to, in response to a decoration operation on any position in the first real scene image, decorate the first real scene image using the first virtual decoration object;

[0019] The first augmented reality image is created based on the first virtual decoration object and the first real scene image.

[0020] In one possible design, in another implementation of another aspect of the embodiment of the present application, the device further includes: a processing module, configured to obtain a moving distance and a moving direction of the second real scene image relative to the first real scene image;

[0021] The position of the first virtual decoration object in the second real scene image is moved according to the moving distance in the opposite direction of the moving direction.

[0022] In a possible design, in another implementation of another aspect of the embodiment of the present application, the first live broadcast interface is also used to display a live broadcast mode control, and the live broadcast mode control is in a selected state.

[0023] The display module is also used to display a third live broadcast interface in response to an operation on the live broadcast mode control. The third live broadcast interface is used to display a first virtual scene. The first virtual scene includes the virtual object and a first background image. The first background image is a fixed live broadcast background image.

[0024] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0025] The first augmented reality live broadcast screen is also used to display a decoration program entry control, and the decoration program entry control is used to indicate the function of starting the decoration interface to virtually decorate the first real scene image;

[0026] The display module is used to display a first decoration interface in response to an operation on the decoration function control, wherein the first decoration interface is used to display the first real scene image in a first area and to display a virtual decoration object library in a second area, and the first area and the second area are included in the first decoration interface; in response to an operation on the controls corresponding to each virtual decoration object in the virtual decoration object library, a control corresponding to a first target virtual decoration object is displayed in the first area, and the first target virtual decoration object is a virtual decoration object selected from the virtual decoration object library.

[0027] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0028] The processing module is further configured to change at least one of the position, size, color or direction of the first target virtual decoration object in the first area in response to an operation on a control corresponding to the first target virtual decoration object;

[0029] The display module is used to display the changed first target virtual decoration object in the first area.

[0030] In one possible design, in another implementation of another aspect of the embodiment of the present application, the first decoration interface is also used to display an upload control;

[0031] The device also includes a transceiver module, which is used to upload and save the first real scene image and the first target virtual decoration object displayed in the first area to the server in response to the operation of the upload control.

[0032] In a possible design, in another implementation of another aspect of the embodiment of the present application, the transceiver module is further used to receive a third augmented reality image sent by the server, where the third augmented reality image is created based on the first reality scene image and the target virtual decoration object;

[0033] The display module is also used to display a third augmented reality live broadcast screen, which includes the third augmented reality image and the anchor object.

[0034] In one possible design, in another implementation of another aspect of an embodiment of the present application, the transceiver module is also used to receive a background picture sent by the server, and the background picture is generated based on the first real scene image and the first target virtual decoration object, or the background picture is generated based on the first real scene image, the first target virtual decoration object and the anchor object.

[0035] In one possible design, in another implementation of another aspect of an embodiment of the present application, the first augmented reality live broadcast screen is also used to display a mini-program entry control, which is used to indicate the start of a function to obtain a second target virtual decoration object.

[0036] In one possible design, in another implementation of another aspect of an embodiment of the present application, the mini-program entry control is a QR code or a barcode or an icon control.

[0037] In one possible design, in another implementation of another aspect of the embodiment of the present application, the display module is further used to display the operation interface of the mini-program in response to an operation on the mini-program entry control;

[0038] After completing the game task or activity task corresponding to the mini program, the transceiver module is used to receive the second target virtual decoration object, and the display module is used to display the second target virtual decoration object.

[0039] In one possible design, in another implementation of another aspect of an embodiment of the present application, the first augmented reality live broadcast screen is also used to display a live broadcast mode control, and the live broadcast mode control is in a selected state, and the display module is used to display a fourth augmented reality live broadcast screen in response to an operation on the live broadcast mode control, and the fourth augmented reality live broadcast screen includes a fourth augmented reality image and the anchor object, and the fourth augmented reality image is an augmented reality image provided by a second terminal, and the second terminal is a device different from the first terminal.

[0040] In one possible design, in another implementation of another aspect of the embodiment of the present application,

[0041] The processing module is also used to take a screenshot and save the target image frame of the first augmented reality live broadcast screen in response to an operation on the screen saving control.

[0042] Another aspect of the present application provides a computer device, including: a memory, a processor, and a bus system;

[0043] Wherein, the memory is used to store programs;

[0044] The processor is used to execute the program in the memory, and the processor is used to execute the above-mentioned methods according to the instructions in the program code;

[0045] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0046] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0047] Another aspect of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

[0048] It can be seen from the above technical scheme that the embodiments of the present application have the following advantages: when a user uses a terminal to watch the live broadcast of the anchor object, the real scene image obtained in real time by the terminal and the virtual decoration object of the real scene image can be processed based on augmented reality technology to obtain an augmented reality image, and the augmented reality image can be displayed on the screen of the terminal as the live broadcast background of the anchor object. In this way, the live broadcast background of the anchor object is updated according to the user's real scene, thereby enriching the live broadcast background of the anchor object, and further enhancing the interaction between the user and the live broadcast of the anchor object. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the virtual environment for the host object to broadcast live;

[0050] Figure 2 A schematic diagram of the architecture of a live screen display system in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of an embodiment of a method for displaying a live screen in an embodiment of the present application;

[0052] Figure 3a A schematic diagram of prompting camera permissions in a live broadcast screen in an embodiment of the present application;

[0053] Figure 3b A schematic diagram of displaying an augmented reality virtual scene in a live broadcast screen in an embodiment of the present application;

[0054] Figure 3c A schematic diagram showing a live broadcast mode control in a live broadcast screen in an embodiment of the present application;

[0055] Figure 3d A schematic diagram of a prompt decoration control in a live broadcast screen in an embodiment of the present application;

[0056] Figure 3e A schematic diagram of a decoration screen in an embodiment of the present application;

[0057] Figure 3f This is another schematic diagram of the decoration screen in the embodiment of the present application;

[0058] Figure 3g Another schematic diagram of the decoration screen display upload control in the embodiment of the present application;

[0059] Figure 3h A schematic diagram of displaying a background image in a live broadcast screen in an embodiment of the present application;

[0060] Figure 3i Another schematic diagram of displaying a background image in a live broadcast screen in an embodiment of the present application;

[0061] Figure 3j A schematic diagram of a mini-program entry control in a live broadcast screen in an embodiment of the present application;

[0062] Figure 3k A schematic diagram of the positions of scene objects and virtual decoration objects in a real scene image in an embodiment of the present application;

[0063] Figure 3l Another schematic diagram of the positions of scene objects and virtual decoration objects in a real scene image in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of an application scenario flow of a live screen display method in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of an embodiment of a method for displaying a live screen in an embodiment of the present application;

[0066] Figure 6 This is a schematic diagram of another embodiment of the live screen display device in the embodiment of the present application;

[0067] Figure 6a A schematic diagram of another embodiment of the live screen display device in the embodiment of the present application;

[0068] Figure 7 This is a schematic diagram of another embodiment of the live screen display device in the embodiment of the present application;

[0069] Figure 8 Schematic diagram of another embodiment of the live screen display device in the embodiment of the present application. DETAILED DESCRIPTION

[0070] The embodiments of the present application provide a live screen display method, apparatus, device and storage medium for creating an augmented reality image using a real scene image acquired by a terminal and a virtual decoration object, and using the augmented reality image as the live broadcast background of the anchor object, thereby enabling the live broadcast background of the anchor object to interact with the real environment and enrich the live broadcast background.

[0071] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0072] Augmented Reality (AR) technology is a technology that calculates the position and angle of a camera image in real time and adds corresponding images, videos, and 3D models. The goal of this technology is to overlay the virtual world on the real world on the screen and interact with it, thereby applying virtual information to the real world and being perceived by human senses, thereby achieving a sensory experience beyond reality. With the development of live broadcast services, more and more virtual objects are now being broadcast live. The live broadcast room for such virtual objects is a virtual environment, and the live broadcast background cannot be directly changed because the live broadcast background is too monotonous. In an exemplary solution, the live broadcast room for the virtual object's live broadcast can be as follows: Figure 1 As shown, the anchor is a virtual object, and the live broadcast environment of the live broadcast room is also a virtual environment.

[0073] In order to solve the above-mentioned technical problems, the present application provides the following technical solutions: displaying a first augmented reality live broadcast screen, the first augmented reality live broadcast screen including a first augmented reality image and a host object, the first augmented reality image being created based on a first real scene image currently acquired by the first terminal and a first virtual decoration object; in response to the first real scene image currently acquired by the first terminal being transformed into a second real scene image, displaying a second augmented reality live broadcast screen, the second augmented reality live broadcast screen including a second augmented reality image and the host object, the second augmented reality image being created based on the second real scene image and the second virtual decoration object, the first real scene image and the second real scene image being obtained by the first terminal shooting the target real scene.

[0074] The live screen display method of each optional embodiment of the present application is implemented based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0075] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0076] The present application also involves cloud technology, which refers to a hosting technology that unifies system resources such as hardware, software, and network in a wide area network or a local area network to achieve data computing, storage, processing, and sharing.

[0077] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of Internet behavior, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing. The cloud technology involved in this application mainly refers to the image recognition between terminals or servers through the "cloud", etc.

[0078] The present application provides a method for displaying a live broadcast screen, which can be applied to a scene where a host object is broadcasting live. In this embodiment, the host object can be a virtual object or a real person, and the specific details are not limited here. Figure 2 The live screen display system shown in FIG. Figure 2As shown, in one case, the live screen display system includes a server and a terminal, and the client is deployed on the terminal, wherein the client can be run on the terminal in the form of a browser, or in the form of an independent application (application, APP), etc. The specific display form of the client is not limited here. The following will take the live screen display system including a server and a terminal as an example for introduction, however, this should not be understood as a limitation of the present application. The server involved in the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop, a PDA, a personal computer, a smart TV, a smart watch, a vehicle-mounted device, a wearable device, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited here by the present application. The number of servers and terminals is also not limited. The solution provided in this application can be completed independently by the terminal, can also be completed independently by the server, or can be completed by the terminal and the server in cooperation. This application does not make any specific limitations on this.

[0079] It is understandable that in the specific implementation of the present application, related data such as real-life scene images are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0080] based on Figure 2 The live screen display system shown in the figure is completed by the terminal and the server in cooperation with each other to illustrate the live screen display method in the embodiment of the present application. Figure 3 As shown, an embodiment of the live screen display method of the present application includes:

[0081] 301. Display a first augmented reality live broadcast screen, where the first augmented reality live broadcast screen includes a first augmented reality image and a host object, where the first augmented reality image is created based on a first reality scene image currently acquired by the first terminal and a first virtual decoration object.

[0082] When the terminal enters a live broadcast room of an anchor object in response to the user's operation, the real-time live broadcast interface of the live broadcast room will be directly displayed on the display screen of the terminal. In this embodiment, the anchor object can be a virtual object or a real person.

[0083] In an exemplary solution, the anchor object is described as a virtual object. The live broadcast room of the virtual object can use an augmented reality live broadcast screen by default and display it through the display screen of the terminal. The augmented reality live broadcast screen includes an augmented reality image and the virtual object, wherein the augmented reality image is created based on the augmented reality technology using the real scene image captured by the camera of the terminal and the virtual decorative object in the real scene image. It is understandable that when using this mode, the live broadcast application needs to have the permission to call the camera of the terminal and the permission to call the AR function. That is, when the live broadcast application is used, a pop-up window can be used to prompt the user that the camera permission needs to be enabled. For example, Figure 3a As shown, when the terminal is preparing to display the live broadcast room of the virtual object, a pop-up window will be displayed in its live broadcast interface to instruct the user whether to enable camera permissions. After authorizing the live broadcast application to use the camera, the terminal will display the augmented reality live broadcast screen after the augmented reality live broadcast screen is created. It is understandable that the first augmented reality live broadcast screen may also include a live broadcast mode control, and the live broadcast mode of the virtual object may be switched by clicking the live broadcast mode control, that is, whether to use a fixed virtual environment or an augmented reality virtual environment. As shown in FIG. Figure 3b As shown, the live mode control can be a point-and-click control to select a traditional mode or an augmented reality mode. At the same time, it can be understood that the live mode control can also be a mode switch selection control, which is not specifically limited here.

[0084] In another exemplary solution, the live broadcast room of the virtual object may use a fixed virtual environment by default, and then by selecting the live broadcast mode control, when the live broadcast mode of the live broadcast room is switched to using an augmented reality live broadcast screen, the first augmented reality live broadcast screen will display the augmented reality image and the virtual object, and the augmented reality image is created based on the augmented reality technology using the real scene image and the virtual decoration object captured by the camera of the terminal. Figure 3c The live broadcast mode control (such as Figure 3c Then, click the live mode control to determine the background environment of the live broadcast room. Click "Traditional Mode" and the following will appear: Figure 3c The background environment is shown in the figure below. Click "Augmented Reality Mode" and the following will appear: Figure 3c The background environment shown in the upper middle picture.

[0085] In an optional solution, if the terminal does not select to use the augmented reality virtual scene, the live broadcast background of the virtual object can also be replaced with an augmented reality image uploaded by other terminals. The way other terminals generate the augmented reality image is the same as that of this terminal, and the details are not repeated here.

[0086] Optionally, before displaying the first augmented reality live screen, the host object and the terminal may first display a real scene image obtained by shooting the target real scene; then identify the scene object in the real scene image, and display a decoration control for prompting that the scene object can be decorated, and then the user may directly click on the decoration control to enter the decoration program of the scene object. Alternatively, before displaying the first augmented reality live screen, the host object and the terminal may first display a real scene image obtained by shooting the target real scene; then display a decoration control for prompting that the real scene image can be decorated at any position, and then the user may directly click on the decoration control to enter the decoration program of the real scene image. It can be understood that the decoration program can be operated directly in the live screen, or it can jump to the decoration interface, which is not limited here.

[0087] In an exemplary solution, the decoration control is a decoration program entry control in the first augmented reality live screen, and the decoration program entry control is used to indicate whether the real scene image needs to be decorated. In an exemplary solution, the first augmented reality live screen can be as follows: Figure 3d As shown, the first augmented reality live screen includes the decoration program entry control (in the example Figure 3d In the example, the "Decoration" icon is used as the entry control of the decoration program), and then clicking on "Decoration" will display the first decoration interface.

[0088] Optionally, the first decoration interface is used to display the first real scene image in the first area and to display the virtual decoration object library in the second area. In an exemplary embodiment, the first decoration interface can be as follows: Figure 3e As shown, the upper part of the decoration interface is used to preview the first real scene image, and the lower part is used to display each virtual decoration object in the virtual decoration object library (such as Figure 3e In this embodiment, the terminal can preview the first real scene image through canvas, or use other methods to preview the first real scene image, which is not limited here.

[0089] It is understandable that the first decoration interface can also be as follows Figure 3fAs shown, the function bar at the bottom divides the first decoration interface into three pages, one page is used to select a corresponding virtual decoration object from the virtual decoration object library on the display line (that is, including all virtual decoration objects) and load it to the terminal locally (such as Figure 3f As shown in the middle figure of the figure, click the corresponding virtual decoration object in the virtual decoration object library, and click "Add to Local" to load the selected virtual decoration object to the terminal local storage); a page is used to decorate and preview the decoration effect (which is equivalent to the above Figure 3e , as Figure 3f The other page is used to display the live broadcast screen (as shown in the figure above); Figure 3f ). The models in the virtual decoration object library can be dynamic models or static models. The model types can include balloons, bouquets, or banners, etc. It is understandable that the virtual decoration object library can be stored in the server in the form of a database. The database (Database), in short, can be regarded as an electronic filing cabinet-a place for storing electronic files, where users can add, query, update, delete, and other operations on the data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program. Database Management System (DBMS) is a computer software system designed for managing databases, generally with basic functions such as storage, interception, security, and backup. Database management systems can be classified by the database model they support, such as relational, Extensible Markup Language (XML); or by the type of computer they support, such as server clusters, mobile phones; or by the query language used, such as Structured Query Language (SQL), XQuery; or by performance focus, such as maximum scale, maximum speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​at the same time.

[0090] In an exemplary solution, if the user is not satisfied with the current scene image after selecting the virtual decoration object, the user can further change and adjust it. Figure 3eAfter selecting and displaying the first target virtual decoration object from the multiple virtual decoration objects shown, the user operation for the multiple controls can be continuously received to select a new target virtual decoration object from the multiple virtual decoration objects to replace the first target virtual decoration object in the real scene image or to add a virtual decoration object in the real scene image. For example, when the user clicks on a control different from the last click, the virtual decoration object corresponding to the control is determined to be the new target virtual decoration object, and then the new target virtual decoration object can be displayed in the first area. At this time, the first target virtual decoration object can be retained or the first target virtual decoration object can be deleted by receiving a corresponding operation.

[0091] In an optional solution, after the first target virtual decoration object is loaded in the first area, the first target virtual decoration object can be further adjusted to increase the fun and richness of the decoration scene in the present application. Specifically, the terminal can receive the user's operation on the control corresponding to the first target virtual decoration object, and then adjust the first target virtual decoration object. After the user clicks the control of the first target virtual decoration object, multiple adjustment methods (such as adjusting color, adjusting position, adjusting size and adjusting direction) may appear. For example, after the user clicks the color adjustment button in the adjustment method, a plurality of preset colors for selection are displayed on the screen, and the user clicks one of the colors again to complete the replacement of the color of the first target virtual decoration object. For example, the user can enlarge or reduce the first target virtual decoration object by dragging with two fingers to complete the adjustment of the size of the first target virtual decoration object. For example, by adjusting the direction control on the box of the first target virtual decoration object, the first target virtual decoration object is placed to complete the adjustment of the direction of the first target virtual decoration object. For example, by long pressing the first target virtual decoration object and sliding on the screen, the first target virtual decoration object is changed in position according to the direction of finger sliding, so as to complete the adjustment of the position of the first target virtual decoration object. Through the above adjustment, the user can adjust the color, size and orientation of the first target virtual decoration object, making the augmented reality image more diverse.

[0092] Optionally, the first decoration interface is also used to display an upload control. By operating the upload control, the first real scene image and the first target virtual decoration object displayed in the first area can be uploaded to the server and saved. It is understandable that the upload control can be in various forms. For example, the upload control can be a "save" control, or a "complete" control; it can also be an "upload" control, etc. The specific form is not limited here, as long as it can trigger the first real scene image and the first target virtual decoration object displayed in the first area to be uploaded to the server and saved. For example, Figure 3g As shown, the "Save" control in the first decoration interface is used as the upload control. Clicking the "Save" control can send the image and virtual decoration object in the first area of ​​the first decoration interface to the server, so that the server can achieve the fusion of the real scene image and the virtual decoration object based on augmented reality technology to generate a new augmented reality image. It is understandable that after clicking the "Save" control, the terminal can also fuse the real scene image and the virtual decoration object through the built-in AR applet or AR web page to obtain a new augmented reality image. The specifics are not limited here.

[0093] After the new augmented reality image is generated, the terminal obtains the new augmented reality image and updates the live screen of the virtual object.

[0094] In an optional solution, after uploading the first real scene image and the first target virtual decoration object displayed in the first area, a background image can be generated based on the first real scene image and the first target virtual decoration object displayed in the first area, or a background image can be generated based on the first real scene image, the first target virtual decoration object and the virtual object displayed in the first area, and the background image is displayed in the live broadcast room. Figure 3h As shown, the server Figure 3g The real scene image and the target decoration image are used to generate a background image, which is then displayed in the live broadcast room (i.e., the background image is displayed on the live broadcast screen). At the same time, the background image can also be displayed on the page corresponding to the decoration interface, and interactive functions such as likes can be provided. Figure 3i As shown, the server Figure 3g The real scene image and the target decoration image shown are combined with the virtual object to generate a background image, which is then displayed in the live broadcast room (i.e., the background image is displayed in the live broadcast screen). At the same time, the background image can also be displayed on the page corresponding to the decoration interface, and interactive functions such as likes can be provided.

[0095] Optionally, in this embodiment, the models in the preset virtual decoration object library can be divided into common models and special models, wherein the common models can be used free of charge, while the special models need to be paid or completed in the mini-program to unlock and use. If one of the special models can be unlocked through a game or activity in the mini-program, the first augmented reality live broadcast screen can also be used to display the mini-program entry control. At this time, the mini-program entry control can be a graphic code, a jump control, or a trigger control; then, in response to the operation of the mini-program entry control, enter the corresponding mini-program and display the mini-program operation interface; when the game task or activity task is executed in the operation interface and the unlocking condition is met, the special model is unlocked; or the special model is unlocked and displayed. Figure 3j As shown, the mini-program entry control in the first augmented reality live broadcast screen is an image code; then long press the image code and identify the image code; after the result is identified, the terminal device displays the operation interface of the mini-program (such as Figure 3j ); when the user completes the jigsaw puzzle in the operation interface, the user is prompted that the special model 1 has been unlocked and the special model 1 is displayed.

[0096] In an optional solution, the live broadcast screen may support a screenshot saving function, that is, it may support taking a screenshot of the virtual object in the augmented reality virtual scene.

[0097] In an embodiment of the present application, the AR function can be published and disseminated in the form of an AR web page. When the user needs to experience the live background replacement function based on AR in this application, the user can open the corresponding AR web page or AR applet in the terminal. Correspondingly, the terminal can receive a trigger signal to open the AR web page. After receiving the trigger signal, the browser in the terminal can obtain the URL of the AR web page or the URL of the AR applet. Among them, the AR web page refers to a web page that can realize the AR function; the AR applet refers to a program that realizes the AR function based on the applet framework. The applet framework is a set of code frameworks that follow the official specifications of the applet (components, application programming interface (API), etc.). The framework (engine) can render and display the applet according to the developer's design intent, and provide various terminal capabilities in the official specifications.

[0098] The trigger signal may be an input signal of inputting the web address of the AR webpage or AR applet in the address bar of the browser (i.e., after selecting the live mode control, the web address of the AR webpage or AR applet is input into the AR webpage or AR applet by default); or an identification signal of the terminal identifying the graphic code; or a received click signal of clicking the webpage link. The graphic code may be a QR code or a barcode, and the graphic code may carry the URL of the AR webpage or AR applet.

[0099] In one embodiment, when the graphic code is recognized, the terminal can call the browser through the application programming interface (API) of the browser or Internet application, and the called browser or Internet application can obtain the URL of the AR webpage according to the content in the recognized graphic code, wherein the above-mentioned Internet application can be an instant messaging application, etc. Optionally, the terminal can call the browser in the system through the API of the browser in the system, or when there is a built-in browser in the application used when recognizing the graphic code, the terminal can also call the built-in browser through the API of the built-in browser, and this embodiment does not limit this. For example, after receiving the QR code corresponding to the AR webpage, the user can also recognize the QR code by long pressing the operation, wherein the long press refers to the operation of pressing the QR code for a preset time threshold, and then the terminal can call the built-in browser through the API of the built-in browser of the instant messaging application.

[0100] In one embodiment, when a click signal of clicking an AR web page or an AR applet link is received, the terminal can call the browser through the browser's API, and the browser can receive the URL of the AR web page or AR applet reported by the application where the web link is located. Among them, the above-mentioned web link is not a web link displayed in the browser. In actual application, the terminal can call the browser in the system through the API of the browser in the system. Optionally, when the web link is a link propagated in a social application, if the social application has a built-in browser, the terminal can also call the built-in browser through the API of the built-in browser. Of course, it should be noted that when there is a built-in browser in a social application, the terminal can still call the browser in the system through the API of the browser in the system, and this embodiment does not limit the called browser.

[0101] After the browser obtains the URL, it opens the AR webpage or AR applet and creates a 3D scene in the webpage. In an exemplary embodiment, a web3D environment can be built in a webpage based on Threejs. Among them, Three represents 3D, js represents javascript, openGL is a cross-platform 3D / 2D drawing standard, and WebGL is an implementation of openGL on the browser. Web front-end developers can directly program with the WebGL interface, but WebGL is only a very basic drawing API. Threejs encapsulates WebGL, so using Threejs to build web3D in a webpage is more efficient. Further basic elements of Threejs include the following aspects: scene, camera, light, and object.

[0102] In one embodiment, before creating a 3D scene on a web page, it can be determined whether the current browser supports the 3D scene rendering function, wherein if it is determined that the current browser supports the 3D scene rendering function, a 3D scene is created on the web page. The browser is the browser used by the current web page, and if the browser used by the current web page supports the 3D scene rendering function, a 3D scene can be created in the web page, and if the browser used by the current web page does not support the 3D scene rendering function, the browser is replaced, such as replacing the currently used system built-in browser with a browser built into the application.

[0103] In one embodiment, creating a three-dimensional space corresponding to the real scene where the terminal is currently located includes: loading a preset 3D engine component library, rendering the 3D scene, initializing a 3D scene camera and adding the initialized 3D scene camera to the 3D scene, rendering the scene in the visible area of ​​the 3D scene camera to obtain the 3D scene, and synthesizing the 3D scene with the real scene where the terminal is currently located to obtain the three-dimensional space of the real scene. Among them, the above-mentioned preset 3D engine component library can be a threejs library and related component libraries, the 3D scene is rendered by the preset 3D engine component library, then the 3D scene camera is initialized, and finally the scene in the camera visible area in the 3D scene is rendered.

[0104] In one embodiment, creating a 3D scene on a web page specifically includes the following steps:

[0105] (1) Load the threejs library and related component libraries and render the scene. The code includes:

[0106] scene = new THREE.Scene();

[0107] (2) Initialize the 3D scene camera. The code includes:

[0108] camera=new THREE.Perspective Camera(90, camera_w / camera_h, 1, 1000);

[0109] (3) Render the scene in the visible area of ​​the camera. The code includes:

[0110] renderer=new THREE.WebGLRenderer({antialias:true,alpha:true});

[0111] renderer.render(scene,camera);

[0112] Furthermore, after creating a 3D scene on the web page, the terminal can synthesize the three-dimensional scene with the currently acquired real scene image to obtain the three-dimensional space of the real scene image, wherein the above-mentioned three-dimensional space, due to the addition of the 3D scene, can be understood as the addition of a spatial coordinate system in the real scene, so that the spatial coordinates of each position in the real scene image can be calculated.

[0113] 302. In response to the first reality scene image currently obtained by the first terminal being transformed into the second reality scene image, a second augmented reality live broadcast screen is displayed, wherein the second augmented reality live broadcast screen includes the second augmented reality image and the anchor object, and the second augmented reality image is created based on the second reality scene image and the second virtual decoration object, and the first reality scene image and the second reality scene image are obtained by the first terminal shooting the target reality scene.

[0114] In this embodiment, when the terminal moves, the real scene image acquired by the terminal will change. At this time, the augmented reality image based on the real scene image and the virtual decoration object will change, so the live broadcast screen will be updated to display the new augmented reality image.

[0115] In one implementation, the first target virtual decoration object is relatively fixed to the scene object in the first real scene image, that is, the first target virtual decoration object moves with the movement of the scene object. Figure 3k The dining table shown in FIG. 1 (the dining table is located in the middle of the display interface of the terminal), and the first target virtual decoration object is a balloon, then in the second real scene image, the position of the dining table is as follows: Figure 3lAs shown, when the display interface of the terminal is located to the right, the balloon also follows the position of the dining table and is located to the right of the display interface of the terminal. At the same time, if a new scene object can be identified in the second real scene image, the new scene image can be re-decorated in the manner of step 301 above, and the details are not repeated here.

[0116] In an optional embodiment, when the terminal moves, the positions of the target virtual decoration object and the anchor object will also have different position display situations. In an optional scheme, the positions of the first target virtual decoration object and the anchor object can move accordingly with the moving position of the terminal. At this time, the result displayed on the screen of the terminal is that as the terminal moves, the first target virtual decoration object and the anchor object are always displayed at a fixed position relative to the screen. In another optional scheme, as the terminal moves, the positions of the first target virtual decoration object and the anchor object remain unchanged. At this time, the result displayed on the screen of the terminal is that as the terminal moves, the augmented reality image displayed on the screen by the first target virtual decoration object and the anchor object will change with the movement of the terminal. In another optional scheme, one of the first target virtual decoration object and the anchor object can move accordingly with the moving position of the terminal, while the position of the other remains unchanged. It is not specifically limited here.

[0117] In an optional embodiment, during the movement of the terminal, the image of the first target virtual decoration object and the anchor object on the terminal screen can be displayed in real time by motion following technology. Specifically, ARkit on the iOS platform and ARcore on the Android platform can realize the motion following function. Taking ARcore as an example, the algorithm of the visual inertial odometry system (Visual Inertial Odometry, VIO) is used in ARcore to realize motion following. VIO combines the image features identified from the camera of the device with the internal motion sensor to follow the relative position and direction of the device. By following the orientation and position, the position of the terminal within the range of six degrees of freedom (six Degrees of Freedom, 6DoF) can be obtained, which is usually referred to as the posture of the terminal. Thus, the augmented reality image displayed on the terminal screen by the first target virtual decoration object and the anchor object is adjusted according to the position of the terminal. It should be noted that when the above-mentioned terminal uses the above-mentioned VIO algorithm, the fusion of sensors is required in hardware, so the terminal includes a camera and a gyroscope, wherein the camera is used to generate pictures, and the gyroscope is used to generate acceleration and angular velocity.

[0118] For example, when the terminal moves, the sizes of the anchor object and the first target virtual decoration object can be adjusted according to the position relationship, and the orientations of the anchor object and the first target virtual decoration object can also be adjusted according to the position relationship. At this time, the terminal determines the first spatial position of its real scene image, and determines that the anchor object and the first target virtual decoration object are in the second spatial position.

[0119] In an optional embodiment, in order to enhance the user's freedom to move the position of the first target virtual decoration object and the anchor object through the mobile terminal, a first operation of the user on the terminal screen can be received before the mobile terminal is moved. This operation is different from the above-mentioned user operation. After receiving the first operation, when the mobile terminal is moved, the first spatial position of the terminal is updated in real time, and the first target virtual decoration object and the anchor object are moved to the updated first spatial position.

[0120] Furthermore, in the process of moving the terminal and updating the position of the first target virtual decoration object and the anchor object in real time, the user's second operation on the terminal screen can be further received. The second operation is different from the above-mentioned first operation. After receiving the user's second operation, the current position of the first target virtual decoration object and the anchor object is determined to be the final position and fixed, and no longer moves, and the current position of the terminal in the real scene is no longer obtained. It can be understood that after the terminal receives the second operation and fixes the position of the first target virtual decoration object and the virtual object, if the terminal continues to move, the virtual image position of the above-mentioned target virtual decoration object and the virtual object displayed on the screen will change accordingly, and will no longer be fixed at the center of the terminal screen.

[0121] For example, the second operation may be a long press. When the terminal screen is pressed, the first spatial position of the terminal in the real scene is updated in real time as the terminal moves. The positions of the first target virtual decoration object and the host object also move with the movement of the terminal. In this process, the user's third operation is received. The third operation may be releasing the screen, that is, ending the long press operation. At this time, the current positions of the first target virtual decoration object and the host object can be determined as the final positions and fixed, and no longer move.

[0122] The following uses the anchor object as a virtual object as an example to illustrate the live screen display process in a specific application scenario:

[0123] 1. Display the mini-program entrance for participating in the live broadcast background decoration in the live broadcast interface of the virtual object.

[0124] 2. In response to the operation of the mini program entrance, enter the activity of participating in the live background decoration. Check whether the user has participated in the activity. If so, display the corresponding work and prompt the user to participate in the live background decoration activity; if not, prompt the user to participate in the live background decoration activity.

[0125] 3. After deciding to participate in the live background decoration activity, jump to the decoration interface. At this time, the decoration interface may include a model selection page (i.e., a virtual decoration object selection page, where the virtual decoration object may also be referred to as a decoration model), a decoration page, and a live page. Among them, the model selection page can directly pull the default model from the background for display. After checking the model, the model is stored in the local background of the terminal. After entering the decoration page, the model stored in the local background of the terminal can be displayed. The live page can directly jump back to the live interface.

[0126] 4. The decoration page will be used to preview the real scene image and the model stored in the local background of the terminal. The real scene image can be displayed using the canvas tag. At the same time, the AR image algorithm can be called to load and transform the model using the 3D rendering engine threejs-miniprogram of the mini program.

[0127] 5. Select the model displayed in the decoration page and call the 3D rendering engine to load the model into the real scene image. At the same time, you can also call the transformation interface of the 3D rendering engine to adjust the position, size, color and direction of the model.

[0128] 6. After the editing is completed, the edited model data, model location information, and real scene image information are uploaded to the server; the server generates an augmented reality image or background image based on the above information. The server can save the above augmented reality image and background image.

[0129] 7. Use the augmented reality image to replace the live background of the virtual object or use the background image to display in the mini program. The mini program also provides the function of commenting and liking the background image.

[0130] The above descriptions are all about the product side to explain the live screen display method. The following is an explanation of the live screen display method from the technical side. Figure 5 As shown, an embodiment of the live screen display method in the present application includes:

[0131] 501. The terminal obtains a current real scene image.

[0132] In this embodiment, the terminal device can capture the real scene image of the real environment in which it is currently located through its own camera, or can receive the real scene image sent by other terminals. There is no specific limitation here, as long as it is a real scene image.

[0133] It is understandable that when the terminal moves, the terminal will reacquire the real scene image, thereby achieving real-time updating of the real scene image.

[0134] 502. The terminal sends the real scene image to the server.

[0135] The terminal sends the real scene image to the server via wired or wireless means. It is understandable that the server should have an AR image processing function.

[0136] 503. The server generates an augmented reality image based on the real scene image and the anchor object using augmented reality technology.

[0137] The server loads a preset 3D engine component library, renders the 3D scene, initializes the 3D scene camera and adds the initialized 3D scene camera to the 3D scene, renders the scene in the visible area of ​​the 3D scene camera to obtain the 3D scene, and synthesizes the 3D scene with the real scene where the terminal is currently located to obtain the three-dimensional space of the real scene. Among them, the above-mentioned preset 3D engine component library can be a threejs library and related component libraries. The 3D scene is rendered by the preset 3D engine component library, and then the 3D scene camera is initialized. Finally, the scene in the visible area of ​​the camera in the 3D scene is rendered to obtain the three-dimensional space of the real scene image. Then, the 3D rendering engine is used to merge the virtual decoration object with the three-dimensional space to obtain the augmented reality image.

[0138] 504. The server sends the augmented reality image to the terminal.

[0139] The server sends the augmented reality image to the terminal via wired or wireless means.

[0140] 505. The terminal replaces the live broadcast background of the anchor object with the augmented reality image.

[0141] The terminal calls the augmented reality image to replace the live broadcast background of the virtual object, thereby realizing the integration of the live broadcast environment of the virtual object and the real scene where the user watching the live broadcast is located.

[0142] The following is a detailed description of the live screen display device in this application. Figure 6 , Figure 6 This is a schematic diagram of an embodiment of a live screen display device in an embodiment of the present application. The live screen display device 20 includes:

[0143] The display module 201 is used to display a first augmented reality live screen, where the first augmented reality live screen includes a first augmented reality image and a host object, where the first augmented reality image is created based on a first reality scene image currently acquired by the first terminal and a first virtual decoration object;

[0144] Processing module 202, used to determine the first real scene image currently acquired by the first terminal to be transformed into the second real scene image

[0145] The display module 201 is used to display a second augmented reality live screen in response to the first reality scene image currently obtained by the first terminal being transformed into a second reality scene image. The second augmented reality live screen includes a second augmented reality image and the anchor object. The second augmented reality image is created based on the second reality scene image and the second virtual decoration object. The first reality scene image and the second reality scene image are obtained by the first terminal shooting the target reality scene.

[0146] In an embodiment of the present application, a live screen display device is provided. With the above device, when a user uses a terminal to watch a live broadcast of a virtual object, the real scene image acquired by the terminal in real time and the virtual object can be processed based on augmented reality technology to obtain an augmented reality virtual scene, and the augmented reality virtual scene can be displayed on the screen of the terminal, so that the live broadcast background of the virtual object is updated according to the user's real scene, thereby enriching the live broadcast background of the virtual object, and further enhancing the interaction between the user and the live broadcast of the virtual object.

[0147] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0148] A processing module 202 is used to identify a first scene object in the first real scene image;

[0149] Display module 201, used for displaying the decoration prompt control of the first scene object;

[0150] The processing module 202 is used to decorate the first scene object with the first virtual decoration object in response to the operation of the decoration prompt control of the first scene object; and create the first augmented reality image based on the first virtual decoration object and the first real scene image.

[0151] In an embodiment of the present application, a live screen display device is provided. With the above device, when a user uses a terminal to watch the live broadcast of an anchor object, the real scene image acquired by the terminal in real time and the virtual decoration object of the real scene image can be processed based on augmented reality technology to obtain an augmented reality image, and the augmented reality image can be displayed on the screen of the terminal as the live broadcast background of the anchor object, so that the live broadcast background of the anchor object is updated according to the user's real scene, thereby enriching the live broadcast background of the anchor object and enhancing the interaction between the user and the live broadcast of the anchor object. At the same time, some scene objects are provided for decoration, which can limit the scope of decoration and make the augmented reality image more unified.

[0152] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application, the processing module 202 is used to identify the second scene object in the second real scene image; if the second scene object is the same as the first scene object, retain the first virtual decoration object and use the first virtual decoration object as the second virtual decoration object; if the second scene object is not the same as the first scene object, delete the first virtual decoration object;

[0153] Display module 201, used for displaying the decoration prompt control of the second scene object;

[0154] The processing module 202 is used to decorate the second scene object with a second virtual decoration object in response to an operation on a decoration prompt control of the second scene object; and to create the first augmented reality image based on the second virtual decoration object and the second real scene image.

[0155] In an embodiment of the present application, a live screen display device is provided. With the above device, when a user uses a terminal to watch the live broadcast of an anchor object, the real scene image acquired by the terminal in real time and the virtual decoration object of the real scene image can be processed based on augmented reality technology to obtain an augmented reality image, and the augmented reality image can be displayed on the screen of the terminal as the live broadcast background of the anchor object, so that the live broadcast background of the anchor object is updated according to the user's real scene, thereby enriching the live broadcast background of the anchor object, and further enhancing the interaction between the user and the live broadcast of the anchor object. At the same time, the virtual decoration object is relatively fixed with the scene object. When the scene object does not exist in the real scene image, the virtual decoration object is deleted, so that the virtual decoration object can be updated according to the real scene image, thereby improving the richness of the live broadcast background.

[0156] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0157] The processing module 202 is used to decorate the first real scene image with the first virtual decoration object in response to a decoration operation on any position in the first real scene image;

[0158] The first augmented reality image is created based on the first virtual decoration object and the first real scene image.

[0159] In an embodiment of the present application, a live screen display device is provided. With the above device, when a user uses a terminal to watch the live broadcast of an anchor object, the real scene image acquired by the terminal in real time and the virtual decoration object of the real scene image can be processed based on augmented reality technology to obtain an augmented reality image, and the augmented reality image can be displayed on the screen of the terminal as the live broadcast background of the anchor object, so that the live broadcast background of the anchor object is updated according to the user's real scene, thereby enriching the live broadcast background of the anchor object, and further enhancing the interaction between the user and the live broadcast of the anchor object. At the same time, any position is provided for decoration, which can increase the diversity of decoration.

[0160] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0161] The processing module 202 is used to obtain the moving distance and moving direction of the second real scene image relative to the first real scene image;

[0162] The position of the first virtual decoration object in the second real scene image is moved according to the moving distance in the opposite direction of the moving direction.

[0163] In an embodiment of the present application, a live screen display device is provided. With the above device, when a user uses a terminal to watch the live broadcast of an anchor object, the real scene image acquired by the terminal in real time and the virtual decoration object of the real scene image can be processed based on augmented reality technology to obtain an augmented reality image, and the augmented reality image can be displayed on the screen of the terminal as the live broadcast background of the anchor object. In this way, the live broadcast background of the anchor object is updated according to the user's real scene, thereby enriching the live broadcast background of the anchor object and further enhancing the interaction between the user and the live broadcast of the anchor object. At the same time, the virtual decoration object is relatively fixed to the scene object. When the scene object does not exist in the real scene image, the virtual decoration object is deleted, so that the virtual decoration object can be updated according to the real scene image, thereby improving the richness of the live broadcast background. Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0164] The display module 201 is also used to display a third augmented reality live broadcast screen in response to an operation on the live broadcast mode control. The third augmented reality live broadcast screen is used to display a first virtual scene. The first virtual scene includes the virtual object and a first background image. The first background image is a fixed live broadcast background image.

[0165] In the embodiment of the present application, a live screen display device is provided. The above device is used to provide an optional solution, so that users can choose to watch live broadcasts in a traditional mode or based on augmented reality technology. This provides users with multiple solutions, which can meet the needs of different users and enhance the integrity of the solution.

[0166] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0167] The first augmented reality live broadcast screen is also used to display a decoration program entry control, and the decoration program entry control is used to indicate the function of starting the decoration interface to virtually decorate the first real scene image;

[0168] The display module 201 is used to display a first decoration interface in response to the operation of the decoration function control, and the first decoration interface is used to display the first real scene image in the first area and the virtual decoration object library in the second area, and the first area and the second area are included in the first decoration interface; in response to the operation of the controls corresponding to each virtual decoration object in the virtual decoration object library, the controls corresponding to the first target virtual decoration object are displayed in the first area, and the first target virtual decoration object is a virtual decoration object selected from the virtual decoration object library. In an embodiment of the present application, a live screen display device is provided. By using the above-mentioned device, in the case of providing augmented reality technology, the real scene image can also be model decorated, thereby enriching the personalized production of the real scene image, thereby improving the interaction between the user and the live broadcast room, and meeting the needs of different users.

[0169] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0170] The processing module 202 is further configured to change at least one of the position, size, color or direction of the first target virtual decoration object in the first area in response to an operation on a control corresponding to the first target virtual decoration object;

[0171] The display module 201 is used to display the changed first target virtual decoration object in the first area.

[0172] In an embodiment of the present application, a live broadcast screen display device is provided. With the above device, in the case of providing augmented reality technology, the real scene image can also be decorated with models, and a modification function can be provided for each model, thereby enriching the personalized production of the real scene image, thereby improving the interaction between the user and the live broadcast room and meeting the needs of different users.

[0173] Optionally, in the above Figure 6 Based on the corresponding embodiments, Figure 6a As shown, in another embodiment of the live screen display device 20 provided in the embodiment of the present application, the first decoration interface is also used to display an upload control;

[0174] The device also includes a transceiver module 203, which is used to upload and save the first real scene image and the first target virtual decoration object displayed in the first area to the server in response to the operation of the upload control.

[0175] In an embodiment of the present application, a live broadcast screen display device is provided. With the above device, in the case of providing augmented reality technology, after realizing the personalized production of the real scene image, the personalized scene is uploaded to the back end of the live broadcast room, so that the back end of the live broadcast room can select replacements from the above scenes, thereby displaying different production plans of different users, which can improve the interaction between users and the live broadcast room and between users, and meet the needs of different users.

[0176] Optionally, in the above Figure 6a On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application, the transceiver module 203 is further used to receive a third augmented reality image sent by the server, where the third augmented reality image is created based on the first reality scene image and the target virtual decoration object;

[0177] The display module 201 is also used to display a third augmented reality live broadcast screen, which includes the third augmented reality image and the anchor object.

[0178] In an embodiment of the present application, a live broadcast screen display device is provided. With the above device, in the case of providing augmented reality technology, the real scene image can also be modeled, thereby enriching the personalized production of the real scene image, thereby improving the interaction between the user and the live broadcast room and meeting the needs of different users.

[0179] Optionally, in the above Figure 6a On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0180] The transceiver module 203 is also used to receive a background image sent by the server, where the background image is generated based on the first real scene image and the first target virtual decoration object, or the background image is generated based on the first real scene image, the first target virtual decoration object and the virtual object.

[0181] In the embodiment of the present application, a live screen display device is provided. With the above device, in the case of providing augmented reality technology, the real scene image can also be modeled, thereby enriching the personalized production of the real scene image, and generating corresponding photos, which can provide users with the function of saving and showing to others, thereby improving the interaction between users and the live broadcast room and between users, and meeting the needs of different users.

[0182] Optionally, in the above Figure 6a On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0183] The first augmented reality live broadcast screen is also used to display a mini-program entry control, which is used to indicate the start of a function to obtain a second target virtual decoration object.

[0184] In an embodiment of the present application, a live broadcast screen display device is provided. The above device is used to provide users with an entrance to a mini-program game or activity, so that users can obtain special virtual decoration objects according to different games and activities, which can help increase the interaction between users and the live broadcast room.

[0185] Optionally, in the above Figure 6a On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0186] The mini program entry control is a QR code, barcode or icon control.

[0187] In an embodiment of the present application, a live screen display device is provided. The above device is used to provide users with an entrance to a mini-program game or activity, so that users can obtain special virtual decoration objects according to different games and activities, which can help increase the interaction between users and the live broadcast room. At the same time, a solution of providing multiple mini-program entrance controls can improve the feasibility and operability of the solution.

[0188] Optionally, in the above Figure 6a On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0189] The display module 201 is also used to display the operation interface of the mini-program in response to the operation of the mini-program entry control;

[0190] After completing the game task or activity task corresponding to the mini program, the transceiver module 203 is used to receive the second target virtual decoration object, and the display module 201 is used to display the second target virtual decoration object.

[0191] In an embodiment of the present application, a live broadcast screen display device is provided. The above device is used to provide users with an entrance to a mini-program game or activity, so that users can obtain special virtual decoration objects according to different games and activities, which can help increase the interaction between users and the live broadcast room.

[0192] Optionally, in the above Figure 6a On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application,

[0193] The first augmented reality live broadcast screen is also used to display a live broadcast mode control, and the live broadcast mode control is in a selected state. The display module 201 is used to display a fourth augmented reality live broadcast screen in response to an operation on the live broadcast mode control. The fourth augmented reality live broadcast screen includes a fourth augmented reality image and the anchor object. The fourth augmented reality image is an augmented reality image provided by a second terminal, and the second terminal is a device different from the first terminal.

[0194] In an embodiment of the present application, a live broadcast screen display device is provided. With the above device, when the user does not enable the function of creating an augmented reality virtual scene based on the real scene image in which he is located, the backend of the live broadcast room can replace the live broadcast background of the live broadcast room according to the augmented reality virtual scene created by other users, which can also enrich the live broadcast background of the live broadcast room and increase the interaction between the users in the live broadcast room.

[0195] Optionally, in the above Figure 6a On the basis of the corresponding embodiment, in another embodiment of the live screen display device 20 provided in the embodiment of the present application, the processing module 202 is also used to respond to the operation of the screen save control to screenshot and save the target image frame of the first augmented reality live screen.

[0196] In the embodiment of the present application, a live screen display device is provided. With the above device, in the case of providing augmented reality technology, the real scene image can also be modeled, thereby enriching the personalized production of the real scene image, and generating corresponding photos, which can provide users with the function of saving and showing to others, thereby improving the interaction between users and the live broadcast room and between users, and meeting the needs of different users.

[0197] The live screen display device provided in this application can be used for the server, please refer to Figure 7 , Figure 7: is a schematic diagram of a server structure provided in an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0198] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0199] The steps performed by the server in the above embodiment can be based on the Figure 7 The server structure shown.

[0200] The live screen display device provided in this application can be used in a terminal, see Figure 8 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. In the embodiments of the present application, the terminal is taken as an example of a smart phone:

[0201] Figure 8 FIG. 1 is a block diagram showing a partial structure of a smart phone related to a terminal provided in an embodiment of the present application. Figure 8 The smart phone includes: a radio frequency (RF) circuit 410, a memory 420, an input unit 430, a display unit 440, a sensor 450, an audio circuit 460, a wireless fidelity (WiFi) module 470, a processor 480, and a power supply 490. Those skilled in the art will appreciate that Figure 8The structure of the smartphone shown in the figure does not constitute a limitation of the smartphone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0202] Combine the following Figure 8 A detailed introduction to the various components of a smartphone:

[0203] The RF circuit 410 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 480 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 410 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 410 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to the global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0204] The memory 420 can be used to store software programs and modules. The processor 480 executes various functional applications and data processing of the smartphone by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, a phone book, etc.), etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0205] The input unit 430 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the smart phone. Specifically, the input unit 430 may include a touch panel 431 and other input devices 432. The touch panel 431, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 431 or near the touch panel 431 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 431 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 480, and can receive and execute commands sent by the processor 480. In addition, the touch panel 431 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 431, the input unit 430 may also include other input devices 432. Specifically, other input devices 432 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0206] The display unit 440 can be used to display information input by the user or information provided to the user and various menus of the smartphone. The display unit 440 may include a display panel 441. Optionally, the display panel 441 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, the touch panel 431 may cover the display panel 441. When the touch panel 431 detects a touch operation on or near it, it is transmitted to the processor 480 to determine the type of touch event. Subsequently, the processor 480 provides a corresponding visual output on the display panel 441 according to the type of touch event. Although in Figure 8 In the embodiment, the touch panel 431 and the display panel 441 are used as two independent components to implement the input and output functions of the smartphone, but in some embodiments, the touch panel 431 and the display panel 441 can be integrated to implement the input and output functions of the smartphone.

[0207] The smartphone may also include at least one sensor 450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 441 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 441 and / or the backlight when the smartphone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the smartphone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in smartphones, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0208] The audio circuit 460, the speaker 461, and the microphone 462 can provide an audio interface between the user and the smartphone. The audio circuit 460 can transmit the received audio data to the speaker 461 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 460 and converted into audio data, and then the audio data is output to the processor 480 for processing, and then sent to another smartphone through the RF circuit 410, or the audio data is output to the memory 420 for further processing.

[0209] WiFi is a short-range wireless transmission technology. Smartphones can help users send and receive emails, browse web pages, and access streaming media through WiFi module 470, which provides users with wireless broadband Internet access. Figure 8 A WiFi module 470 is shown, but it is understandable that it is not an essential component of the smartphone and can be omitted as needed without changing the essence of the invention.

[0210] Processor 480 is the control center of the smartphone, which uses various interfaces and lines to connect various parts of the entire smartphone, and executes various functions of the smartphone and processes data by running or executing software programs and / or modules stored in memory 420, and calling data stored in memory 420, thereby monitoring the smartphone as a whole. Optionally, processor 480 may include one or more processing units; optionally, processor 480 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into processor 480.

[0211] The smart phone also includes a power source 490 (such as a battery) for supplying power to various components. Optionally, the power source can be logically connected to the processor 480 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.

[0212] Although not shown, the smartphone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0213] The steps performed by the terminal in the above embodiment can be based on the Figure 8 The terminal structure shown.

[0214] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0215] The present application also provides a computer program product including a program, which, when executed on a computer, enables the computer to execute the methods described in the aforementioned embodiments.

[0216] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0217] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0218] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0220] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0221] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A live screen display method, applied to a first terminal, characterized in that: include: Displaying a first augmented reality live broadcast screen, wherein the first augmented reality live broadcast screen includes a first augmented reality image and a host object, wherein the first augmented reality image is created based on a first reality scene image currently acquired by the first terminal and a first virtual decoration object; In response to the first reality scene image currently acquired by the first terminal being transformed into a second reality scene image, a second augmented reality live broadcast screen is displayed, the second augmented reality live broadcast screen includes a second augmented reality image and the anchor object, the second augmented reality image is created based on the second reality scene image and the second virtual decoration object, and the first reality scene image and the second reality scene image are obtained by shooting the target reality scene by the first terminal.

2. The method according to claim 1, characterized in that: Before displaying the first augmented reality live picture, the method further includes: Identifying a first scene object in the first real scene image, and displaying a decoration prompt control of the first scene object; In response to an operation on a decoration prompt control of the first scene object, decorating the first scene object with the first virtual decoration object; The first augmented reality image is created based on the first virtual decoration object and the first real scene image.

3. The method according to claim 2, characterized in that Before displaying the second augmented reality live screen, the method further includes: Identifying a second scene object in the second real scene image; If the second scene object is the same as the first scene object, retaining the first virtual decoration object and using the first virtual decoration object as the second virtual decoration object; If the second scene object is different from the first scene object, deleting the first virtual decoration object and displaying a decoration prompt control of the second scene object; In response to an operation on a decoration prompt control of the second scene object, decorating the second scene object with a second virtual decoration object; The first augmented reality image is created based on the second virtual decoration object and the second real scene image.

4. The method according to claim 1, characterized in that: Before displaying the first augmented reality live picture, the method further includes: In response to a decoration operation on any position in the first real scene image, decorating the first real scene image using the first virtual decoration object; The first augmented reality image is created based on the first virtual decoration object and the first real scene image.

5. The method according to claim 4, characterized in that Before displaying the second augmented reality live screen, the method further includes: Acquire a moving distance and a moving direction of the second real scene image relative to the first real scene image; The position of the first virtual decoration object in the second real scene image is moved according to the moving distance in the opposite direction of the moving direction.

6. The method according to any one of claims 1 to 5, characterized in that The first augmented reality live broadcast screen is also used to display a decoration program entry control, and the decoration program entry control is used to indicate the function of starting a decoration interface to virtually decorate the first real scene image. The method also includes: In response to an operation on the decoration function control, a first decoration interface is displayed, wherein the first decoration interface is used to display the first real scene image in a first area and to display a virtual decoration object library in a second area, wherein the first area and the second area are included in the first decoration interface; In response to an operation on controls corresponding to each virtual decoration object in the virtual decoration object library, controls corresponding to a first target virtual decoration object are displayed in the first area, where the first target virtual decoration object is a virtual decoration object selected from the virtual decoration object library.

7. The method according to claim 6, characterized in that The method further comprises: In response to an operation on a control corresponding to the first target virtual decoration object, at least one of the position, size, color or direction of the first target virtual decoration object in the first area is changed, and the changed target virtual decoration object is displayed in the first area.

8. The method according to claim 6, characterized in that The first decoration interface is also used to display an upload control, and the method further includes: In response to the operation on the upload control, the first real scene image and the first target virtual decoration object displayed in the first area are uploaded to the server and saved.

9. The method according to claim 8, characterized in that The method further comprises: Receiving a third augmented reality image sent by the server, where the third augmented reality image is created based on the first reality scene image and the first target virtual decoration object; A third augmented reality live broadcast screen is displayed, wherein the third augmented reality live broadcast screen includes the third augmented reality image and the anchor object.

10. The method according to claim 8, characterized in that The method further comprises: Receive a background picture sent by the server, where the background picture is generated based on the first real scene image and the first target virtual decoration object, or the background picture is generated based on the first real scene image, the first target virtual decoration object and the virtual object.

11. The method according to any one of claims 1 to 5 and 7 to 10, characterized in that The first augmented reality live broadcast screen is also used to display a mini-program entry control, and the mini-program entry control is used to indicate to start a function of acquiring a second target virtual decoration object; The method further comprises: In response to an operation on the mini-program entry control, displaying an operation interface of the mini-program; When a game task or activity task corresponding to the mini-program is executed in the operation interface and a preset condition is met, the second target virtual decoration object is obtained, or the second target virtual decoration object is displayed.

12. The method according to any one of claims 1 to 5 and 7 to 10, characterized in that The first augmented reality live broadcast screen is also used to display a live broadcast mode control, and the live broadcast mode control is in a selected state, and the method further includes: In response to the operation of the live broadcast mode control, a fourth augmented reality live broadcast screen is displayed, and the fourth augmented reality live broadcast screen includes a fourth augmented reality image and the anchor object. The fourth augmented reality image is an augmented reality image provided by the second terminal, and the second terminal is a device different from the first terminal.

13. The method according to any one of claims 1 to 5, 7 to 10, characterized in that: The method further comprises: In response to an operation on a screen saving control, a screenshot is taken to save a target image frame of the first augmented reality live screen.

14. A live screen display device, characterized in that: include: A display module, configured to display a first augmented reality live broadcast screen, wherein the first augmented reality live broadcast screen includes a first augmented reality image and a host object, wherein the first augmented reality image is created based on a first reality scene image currently acquired by the first terminal and a first virtual decoration object; A processing module, used for determining that a first real scene image currently acquired by the first terminal is transformed into a second real scene image; The display module is used to transform the first reality scene image currently obtained by the first terminal into a second reality scene image, and display a second augmented reality live screen. The second augmented reality live screen includes a second augmented reality image and the anchor object. The second augmented reality image is created based on the second reality scene image and the second virtual decoration object. The first reality scene image and the second reality scene image are obtained by shooting the target reality scene by the first terminal.

15. A computer device, characterized in that: include: Memory, processor, and bus system; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, and the processor is used to execute the method according to any one of claims 1 to 13 according to the instructions in the program code; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

16. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 13.

Images