Data processing method and apparatus, readable storage medium, and device

Abstract

Embodiments of the present application disclose a data processing method and device, a storage medium and equipment. The method comprises: obtaining a broadcast control data packet through a mixed reality glasses; if operation instruction data in the broadcast control data packet matches preset first instruction data in the mixed reality glasses, triggering a target first event in the mixed reality glasses associated with the preset first instruction data; using a corresponding augmented reality virtual asset in the mixed reality glasses through the target first event; and displaying a use effect of the corresponding augmented reality virtual asset through the mixed reality glasses. In this way, the actual live information can be watched through the mixed reality glasses, and the use effect of the augmented reality virtual asset can also be seen, so that the live audience can watch the use effect of the augmented reality virtual asset in a first perspective, and the watching experience of the live audience is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method, apparatus, computer-readable storage medium, and device. Background Technology

[0002] With the maturation and development of virtual technology, Augmented Reality (AR) technology is increasingly being applied to live streaming, such as esports broadcasts. Thanks to AR, online esports broadcasts bring in-game characters or cool virtual effects to life on stage, greatly enriching the live stream and making it more game-like. However, because this method only caters to online viewers, in-person viewers cannot experience the virtual effects firsthand, resulting in a less immersive viewing experience. Summary of the Invention

[0003] This application provides a data processing method, apparatus, computer-readable storage medium, and device that enable on-site audiences to experience the use of augmented reality virtual assets from a first-person perspective, thereby enhancing their viewing experience.

[0004] This application provides a data processing method applied to mixed reality glasses, wherein the mixed reality glasses include at least one augmented reality virtual asset, and the method includes:

[0005] Acquire broadcast control data packets and obtain operation instruction data from the broadcast control data packets; if the operation instruction data matches preset first instruction data in the mixed reality glasses, trigger a target first event associated with the preset first instruction data in the mixed reality glasses, use the corresponding augmented reality virtual asset in the mixed reality glasses through the target first event; display the usage effect of the corresponding augmented reality virtual asset through the mixed reality glasses.

[0006] This application also provides a data processing method, including:

[0007] A playback control panel is obtained, the playback control panel including at least one playback control button; when a target trigger event of a target playback control button is triggered, a playback control data packet corresponding to the target playback control button is generated according to the target trigger event; the playback control data packet is sent to the mixed reality glasses, so that the mixed reality glasses trigger a target first event associated with the target trigger event of the target playback control button according to the playback control data packet.

[0008] This application embodiment also provides a data processing method applied to a second server, the second server including at least one augmented reality virtual resource, the method comprising:

[0009] The system acquires live footage captured by a real camera and a broadcast control data packet. When the operation instruction data in the broadcast control data packet matches the preset second instruction data in the second server, a target second event associated with the preset second instruction data is triggered in the second server. The corresponding augmented reality virtual asset in the second server is used through the target second event. The virtual image corresponding to the use effect of the corresponding augmented reality virtual asset is combined with the live footage to obtain the live broadcast image.

[0010] This application also provides a data processing apparatus, including:

[0011] The first acquisition module is used to acquire broadcast control data packets and acquire operation instruction data in the broadcast control data packets; the first trigger module is used to trigger a target first event associated with the preset first instruction data in the mixed reality glasses if the operation instruction data matches the preset first instruction data in the mixed reality glasses, and use the corresponding augmented reality virtual asset in the mixed reality glasses through the target first event; the display module is used to display the usage effect of the corresponding augmented reality virtual asset through the mixed reality glasses.

[0012] This application also provides a data processing apparatus, including:

[0013] The second acquisition module is used to acquire a broadcast control panel, which includes at least one broadcast control button; the generation module is used to generate a broadcast control data packet corresponding to the target broadcast control button when the target trigger event of the target broadcast control button among the at least one broadcast control button is triggered; the sending module is used to send the broadcast control data packet to the mixed reality glasses, so that the mixed reality glasses trigger a target first event associated with the target trigger event of the target broadcast control button according to the broadcast control data packet.

[0014] This application also provides a data processing apparatus, including:

[0015] The third acquisition module is used to acquire the live scene footage captured by a real camera and to acquire the broadcast control data packet; the second triggering module is used to trigger a target second event associated with the preset second instruction data in the second server when the operation instruction data in the broadcast control data packet matches the preset second instruction data in the second server, and to use the corresponding augmented reality virtual asset in the second server through the target second event; the compositing module is used to composit the virtual image corresponding to the use effect of the corresponding augmented reality virtual asset with the live scene footage to obtain the live broadcast image.

[0016] This application also provides a computer-readable storage medium storing a computer program adapted for loading by a processor to perform the steps of the data processing method as described in any of the above embodiments.

[0017] This application also provides an apparatus, which includes a memory and a processor. The memory stores a computer program, and the processor executes the steps of the data processing method described in any of the above embodiments by calling the computer program stored in the memory.

[0018] The data processing method, apparatus, computer-readable storage medium, and device provided in this application embodiment, by providing mixed reality glasses, acquires broadcast control data packets through the mixed reality glasses. If the operation instruction data in the broadcast control data packet matches the preset first instruction data in the mixed reality glasses, a target first event associated with the preset first instruction data is triggered in the mixed reality glasses. The corresponding augmented reality virtual asset in the mixed reality glasses is then used through the target first event. Unlike the prior art where the target first event is triggered by directly clicking a button in the mixed reality glasses, this application requires acquiring the broadcast control data packet, and the target first event is only triggered when the operation instruction data in the broadcast control data packet matches the preset first instruction data in the mixed reality glasses. After the target first event is triggered, the usage effect of the corresponding augmented reality virtual asset is displayed in the mixed reality glasses through the target first event. In this way, the mixed reality glasses not only allow viewers to see the actual on-site information, but also the usage effect of the augmented reality virtual asset, enabling on-site viewers to view the usage effect of the augmented reality virtual asset from a first-person perspective, thus improving the viewing experience of the on-site viewers. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating the application scenarios provided in the embodiments of this application.

[0021] Figure 2 This is a schematic flowchart illustrating a data processing method applied to hybrid display glasses, as provided in an embodiment of this application.

[0022] Figure 3 This is a flowchart illustrating a data processing method applied to hybrid display glasses, as provided in an embodiment of this application.

[0023] Figure 4 This is another schematic flowchart illustrating a data processing method for hybrid display glasses provided in an embodiment of this application.

[0024] Figure 5 This is a flowchart illustrating a data processing method applied to a first server, as provided in an embodiment of this application.

[0025] Figure 6 This is another schematic flowchart illustrating a data processing method applied to a first server, as provided in an embodiment of this application.

[0026] Figure 7 This is a flowchart illustrating a data processing method applied to a second server, as provided in an embodiment of this application.

[0027] Figure 8 This is a flowchart illustrating a data processing method applied to a second server, as provided in an embodiment of this application.

[0028] Figure 9 This is a schematic diagram of a live screen provided in an embodiment of this application.

[0029] Figure 10 This is a schematic diagram of the data processing device integrated into hybrid display glasses, as provided in an embodiment of this application.

[0030] Figure 11 This is a schematic diagram of the structure of a data processing device integrated in a first server, as provided in an embodiment of this application.

[0031] Figure 12 This is a schematic diagram of the structure of a data processing device integrated in a second server, provided in an embodiment of this application.

[0032] Figure 13 This is a schematic diagram of the device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] This application provides a data processing method, apparatus, computer-readable storage medium, and device. Specifically, the data processing method of this application can be executed by a device, which can be a mixed reality glasses device or a server, etc., as will be described in detail below.

[0035] The mixed reality glasses integrate the application environment required for the execution of mixed reality applications. The server includes a first server (broadcast control server), a second server (virtual reality server), and a third server (mixed reality server). The first server, the second server, and the third server can all be independent physical servers, or a server cluster composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services and cloud databases.

[0036] The live stream footage is created by overlaying virtual special effects onto the live footage captured by a real camera, ultimately presenting a third-person perspective. This allows the virtual special effects to be displayed on stage, satisfying the viewing experience of online viewers. However, in a live event, the live stream can only be displayed on a large screen, preventing the on-site audience from experiencing the virtual special effects firsthand, resulting in a poor viewing experience for them.

[0037] To address the technical problems in this application, the technical solution presented herein is proposed. This technical solution can be applied to any mixed reality viewing scenario. This application uses a live streaming scenario as an example for illustration, and further uses a virtual game live streaming scenario as an example for illustrative purposes.

[0038] In this application, "blueprint" refers to a visual programming language / visual programming script, which can be a complete game script that does not require code. Its working principle is to use a node-based interface to connect events, functions, variables, etc., to achieve game functions and effects. Blueprints enable level designers and game developers to quickly create and iterate virtual games.

[0039] The playback control engine in this application is a script / module / program / code library that facilitates playback control. By referencing the playback control engine, playback control can be quickly implemented.

[0040] The virtual AR engine in this application refers to scripts / modules / programs / code libraries that facilitate the implementation of augmented reality effects. It typically includes, but is not limited to, rendering technologies, audio and video compression and transmission technologies, computer vision and auditory technologies, camera-related technologies, image blending, and interactive technologies. Through the virtual AR engine, it is possible to quickly synthesize live footage captured by a real camera with virtual images corresponding to augmented reality virtual assets. The functionality of the virtual AR engine extends far beyond these features. Virtual AR engines include Unreal Engine and, for example, the UE4 engine.

[0041] In this application, the mixed reality development engine refers to a script / module / program / code library that bridges the virtual and real worlds, including but not limited to entirely new mixed reality scenes, film and animation, computer vision, computer graphics, and interactive technologies. The mixed reality development engine can be any type of mixed reality development engine, such as Magics Engine from Magic Technology.

[0042] The following describes a data processing method, apparatus, computer-readable storage medium, and device according to this application. It should be noted that the order described in the following embodiments is not intended to limit the preferred order of the embodiments.

[0043] Figure 1 This is a schematic diagram of an application scenario provided in the embodiments of this application. Figure 1 The system provides a first server (broadcast control server), a second server (virtual reality server), a third server (mixed reality server), and mixed reality glasses.

[0044] The broadcast control server provides a broadcast control engine, and broadcast control engine projects or broadcast control engine engineering projects developed based on the broadcast control engine; the virtual reality server provides a virtual augmented reality engine, namely a virtual AR engine or a virtual AR tracking engine, and virtual AR live streaming engine engineering projects or virtual AR live streaming engine projects developed based on the virtual AR engine; the mixed reality server provides a mixed reality development engine, and mixed reality development engine engineering projects or mixed reality development engine projects developed based on the mixed reality development engine.

[0045] exist Figure 1 In this setup, the virtual reality server, broadcast control server, and mixed reality server required for live streaming are connected to the same switch, and the router's LAN port is connected to the switch. The router's bandwidth should be as high as possible, such as gigabit or higher. The mixed reality glasses are connected to the wireless signal emitted by the router. This ensures that the broadcast control server, virtual reality server, mixed reality server, and mixed reality glasses are all on the same local area network. The broadcast control server, virtual reality server, mixed reality server, and mixed reality glasses are set to different IP addresses within the same network segment.

[0046] The virtual reality server is a server cluster. Figure 1 The diagram shows three virtual reality servers, each equipped with a virtual AR engine. To distinguish the virtual AR engines in different virtual reality servers, the virtual AR engines are numbered as Virtual AR Engine 1, Virtual AR Engine 2, and Virtual AR Engine 3, respectively.

[0047] for Figure 1 For a virtual reality server, blueprints can be written to acquire live footage captured by a real camera. Blueprints can also be used to create transmission control protocols, such as a transmission control protocol client, to receive broadcast control data packets sent by the broadcast control server (acting as the transmission control protocol server). Blueprints can then be used to control the display of virtual assets (including at least one augmented reality virtual asset and / or model information corresponding to at least one virtual character involved in the live stream) within the virtual reality server based on the broadcast control data packets. The virtual images corresponding to the virtual assets and the live footage are then composited to obtain the live stream image, which is then sent to a live streaming platform or rebroadcasting platform. These functionalities can be integrated into the virtual AR live streaming engine project of the virtual reality server. The composited live stream image is used for online live streaming, meaning online live streaming can be achieved through the virtual reality server.

[0048] for Figure 1 For the mixed reality server in the system, it can perform lightweight processing on the model information corresponding to at least one augmented reality virtual asset and / or at least one virtual character involved in the live broadcast to reduce performance consumption. According to the engineering development requirements of the mixed reality glasses, the corresponding plugins and settings can be enabled, such as enabling the OpenXR plugin, setting the "start in vr" option, etc., and setting the position information of the object to be controlled (playerPawn) in the mixed reality development engine project according to the position relationship between the position of the audience and the virtual assets (such as the model information corresponding to the augmented reality virtual assets and / or virtual characters). The blueprint is used to create a transmission control protocol, such as a transmission control protocol client, to receive broadcast control data packets sent by the broadcast control server, which acts as the transmission control protocol server. The blueprint is also used to control the display of virtual assets (including at least one augmented reality virtual asset and / or model information corresponding to at least one virtual character involved in the live broadcast) in the mixed reality server based on the broadcast control data packets. Interactive content is designed according to the live broadcast requirements, such as a one-click pop-up live broadcast panel, and a user interface panel that can move with the location information of the object to be controlled. In addition, a first event needs to be set for each button on the live broadcast panel. For example, a blueprint can be used to set the first event, so that the buttons on the user interface panel can be triggered by gestures.

[0049] The functionalities implemented in the mixed reality server described above can be integrated into the mixed reality development engine project. After the mixed reality development engine project is completed, it will be packaged into a program package that supports mixed reality glasses. The program package will then be installed in the mixed reality glasses, enabling them to implement the functionalities implemented in the mixed reality server. It should be noted that the mixed reality glasses can be worn by on-site audiences, allowing them to experience the use of augmented reality virtual assets and / or the display of virtual character model information from a first-person perspective.

[0050] for Figure 1 For the broadcast control server, it can obtain real-time data interfaces from virtual game live streams, such as e-sports event live streams. This real-time data interface can be a link, such as a JSON link. A blueprint is written, and JSON is used to parse the blueprint nodes, extracting the game data to be presented from the JSON link and storing it as variables for later use. This game data includes the game's ban / pick (BP) data, the sects and formations chosen by the spectators, and the sects and formations chosen by the opposing team. Based on the broadcast control server's IP and port (which is independent and unused), a blueprint is written to create a transmission control protocol, such as setting up a transmission control protocol server. A graphical user interface, such as a UMG interface or broadcast control panel, is written. The broadcast control panel includes at least one broadcast control button. A blueprint is written to set trigger events for each broadcast control button. Each broadcast control button's trigger event is associated with a first event of the corresponding button provided in the mixed reality glasses and a second event of the corresponding button provided in the virtual reality server, so that when the broadcast control button's trigger event is triggered, both the first event of the corresponding button in the mixed reality glasses and the second event of the corresponding button in the virtual reality server are triggered simultaneously.

[0051] When the broadcast control server detects that at least one target broadcast control button has been triggered, a transmission control protocol data packet, such as a broadcast control data packet, is generated based on the data operation instruction corresponding to the target trigger event of the target broadcast control button. In some scenarios, such as when it is possible to see which hero each virtual character in a virtual game live stream has selected on a stage, the acquired live stream variable data can also be used to generate a broadcast control data packet. The broadcast control data packet is simultaneously sent to the virtual reality server and the mixed reality server via the transmission control protocol server, so that when the virtual reality server and the mixed reality server receive the broadcast control data packet, they can perform corresponding processing on the model information corresponding to at least one augmented reality virtual asset and / or at least one virtual character involved in the live stream.

[0052] The functions implemented by the broadcast control server described above can be integrated into the broadcast control engine project. The broadcast control server simultaneously sends broadcast control data packets to the virtual reality server and the mixed reality server (mixed reality glasses), so that the virtual assets seen in the mixed reality glasses (including model information of augmented reality virtual assets and virtual characters, etc.) appear at the same time as the virtual assets seen in the live broadcast, thus ensuring that the effect seen by the on-site audience from the first-person perspective is the same as the effect seen from the third-person perspective in the live broadcast.

[0053] The following text will be in the format of Figure 1 The application scenarios shown illustrate the functions implemented by the broadcast control server, virtual reality server, mixed reality server, and mixed reality glasses, respectively.

[0054] It is important to note that Figure 1 The application scenarios described are merely illustrative examples. Figure 1 The control server, virtual reality server, mixed reality server, and mixed reality glasses are all on the same local area network. However, they can also be on different networks or not all on the same local area network, as long as they can connect to each other and communicate accordingly.

[0055] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The data processing method is applied to mixed reality glasses and includes the following steps.

[0056] 101. Obtain the broadcast control data packet and the operation instruction data in the broadcast control data packet.

[0057] The broadcast control data packet is sent from other devices, such as a broadcast control server, and received by the mixed reality glasses. This broadcast control data packet includes operation command data. The operation command data can be pre-set in the broadcast control server. For example, controlling the playback of a specific augmented reality virtual asset corresponds to an operation command, such as HXZ; correspondingly, the operation command data in the broadcast control data packet includes HXZ. For example, controlling the release of the formation animation of the first virtual character on the left corresponds to an operation command, such as ZFL; simultaneously controlling the playback of the formation animation content, such as controlling the playback of the Tiger Roar Formation, corresponds to an operation command, such as HXZ; correspondingly, the operation command data in the broadcast control data packet includes ZFL-HXZ, etc.

[0058] Before the mixed reality glasses acquire broadcast control data packets, a package corresponding to the mixed reality development engine project is received from the mixed reality server, and this package is installed in the mixed reality glasses. Since transmission control protocols, such as transmission control protocol clients, can be created in the mixed reality development engine project, a transmission control protocol client is automatically created in the mixed reality glasses when the package is installed. That is, before acquiring broadcast control data packets, the method further includes: creating a transmission control protocol, such as a transmission control protocol client, in the mixed reality glasses and establishing a communication connection with the broadcast control server.

[0059] In one embodiment, the step of creating a transmission control protocol in mixed reality glasses includes: obtaining the glasses' IP address and port; and creating a transmission control protocol in the mixed reality glasses based on the glasses' IP address and port, such as creating a transmission control protocol client. Here, the glasses' IP address refers to the IP address of the mixed reality glasses, and the glasses' port refers to a port in the mixed reality glasses that is independent and unused.

[0060] In one embodiment, the step of establishing a communication connection with a broadcast control server includes: obtaining a first IP address and a first port of the broadcast control server that has a transmission control protocol; and establishing a first communication connection with the broadcast control server via the transmission control protocol based on the glasses' IP address, glasses port, first IP address, and first port. The transmission control protocol, such as a transmission control protocol server, can be pre-created in the broadcast control server. The mixed reality glasses obtain the first IP address and first port of the transmission control protocol server and establish a first communication connection with the broadcast control server via the transmission control protocol based on the corresponding IP address and port.

[0061] It's important to note that even though the mixed reality glasses and the broadcast control server are on the same local area network, they cannot communicate with each other. Therefore, creating a Transmission Control Protocol (TCP) is essential to enable them to communicate with each other based on the TCP. It's also important to note that although a TCP server is created in the broadcast control server and a TCP client is created in the mixed reality glasses, they all use the same TCP. Furthermore, as mentioned later, a TCP client is also created in the virtual reality server, all using the same TCP.

[0062] Correspondingly, the step of obtaining broadcast control data packets includes: receiving broadcast control data packets sent by the broadcast control server through a first communication connection.

[0063] 102. If the operation instruction data matches the preset first instruction data in the mixed reality glasses, a target first event associated with the preset first instruction data is triggered in the mixed reality glasses, and the corresponding augmented reality virtual asset in the mixed reality glasses is used through the target first event.

[0064] In the mixed reality glasses, multiple preset first instruction data are pre-set, and these preset first instruction data correspond one-to-one with the operation instruction data in the broadcast control server. These multiple first preset data can be exactly the same as the corresponding operation instruction data, and in some cases they can be different, but they need to correspond one-to-one. This application takes the example of being exactly the same. When they are different, further parsing is required for matching.

[0065] In mixed reality glasses, multiple preset first command data are associated with target first events of buttons on the user interface panel of the mixed reality glasses. For example, to control the playback of a certain augmented reality virtual asset, a preset first command data, such as HXZ, is set, and this preset first command data is associated with the target first event of the corresponding button. One preset first command data corresponds to one target first event.

[0066] In one embodiment, the target first event of the button can be a button press event, a button touch event, or a gesture control event of the button, etc., and is not specifically limited. The button press event, touch event, and gesture event all include a function corresponding to the event, which implements the function matching the target first event. The function corresponding to each different event implements different functions.

[0067] The mixed reality glasses include virtual assets, including at least one augmented reality (AR) virtual asset designed for augmented reality effects. This AR virtual asset can be an AR virtual image, AR virtual animation, or AR virtual video composed of AR virtual animation. The AR virtual asset also includes location information, which is pre-set based on the scene conditions.

[0068] In one embodiment, the function of the target first event of the button mentioned above implements the function of using the corresponding augmented reality virtual asset in the mixed reality glasses. For example, in the function of the press event of a button A, the augmented reality virtual asset A in the mixed reality glasses is displayed, or the corresponding augmented reality virtual asset A in the mixed reality glasses is played; in the function of the press event of a button B, the augmented reality virtual asset B in the mixed reality glasses is displayed, or the corresponding augmented reality virtual asset B in the mixed reality glasses is played.

[0069] In one embodiment, the virtual assets in the mixed reality glasses can be obtained from the virtual reality server by the mixed reality server. Because the virtual assets in the virtual reality server will have their positions adjusted according to the actual situation before the corresponding project is implemented, when the virtual assets in the mixed reality server are imported from the virtual reality server, no adjustments need to be made to the virtual assets in the mixed reality server, so that the online live broadcast and the on-site audience see the same effect.

[0070] When the operation command data in the broadcast control data matches the preset first command data in the mixed reality glasses, a target first event associated with the preset first command data is triggered in the mixed reality glasses to execute the target first event. For example, the function of the target first event is called and executed to use the corresponding augmented reality virtual asset in the mixed reality glasses through the target first event, such as playing the corresponding augmented reality virtual asset, such as playing the tiger animation corresponding to the Tiger Roar Array.

[0071] It should be noted that although virtual assets exist in the mixed reality glasses, the use of virtual assets in this embodiment is controlled by the broadcast control server, specifically by sending broadcast control data packets. Therefore, unlike the prior art where the target first event is triggered by clicking a button in the mixed reality glasses, this application requires obtaining the broadcast control data packet, and the target first event is only triggered when the operation instruction data of the broadcast control data packet matches the preset first instruction data in the mixed reality glasses.

[0072] 103. Display the usage effect of corresponding augmented reality virtual assets through mixed reality glasses.

[0073] Mixed reality glasses consist of a frame and lenses that match the frame. Each lens can be a display screen, through which the user can see live information from the event (similar to the function of regular glasses) and the effects of using corresponding augmented reality virtual assets—that is, information about the virtual assets. Specifically, the display screen shows the effects of using the corresponding augmented reality virtual assets.

[0074] In this embodiment, the mixed reality glasses not only allow viewers to see the actual on-site information, but also to see the effects of using augmented reality virtual assets. This allows on-site viewers to experience the effects of using augmented reality virtual assets from a first-person perspective, enhancing their viewing experience.

[0075] In one embodiment, such as Figure 3 As shown, the data processing method applied to mixed reality glasses before acquiring the broadcast control data packet also includes the following steps.

[0076] 201. Obtain the location information of the object to be controlled. The location information of the object to be controlled is determined based on the location information of the audience on site.

[0077] The object to be controlled is represented by playerPawn, which is an object of the Pawn class, which inherits from the Actor class. The position information of the object to be controlled is determined based on the position information of the audience at the venue. Specifically, it is pre-set / determined based on the positional relationship between the offline audience and at least one augmented reality virtual asset to be presented. The position information of the object to be controlled can be understood as the positional information of the audience's viewpoint after wearing mixed reality glasses.

[0078] For example, if the virtual asset's position in the virtual scene is (x=0, y=0, z=0), and the virtual asset's position in the live broadcast is 10m vertically, 2m horizontally, and 3m high above the position of the audience, then the playerPawn's position can be set to (x=10, y=2, z=3).

[0079] Obtain the position information of the object to be controlled, i.e., obtain the position information of the playerPawn. Since each audience member is in a different location, the position information of the object to be controlled will also be different in the mixed reality glasses worn by each audience member.

[0080] 202. Set the user interface panel in the mixed reality glasses to a subclass of the class to which the object to be controlled belongs.

[0081] Mixed reality glasses provide a User Interface (UI) panel. This UI panel can be set to a subclass of the class of the object being controlled; for example, the UI panel can be bound to a child actor of `playerPawn`. Setting the UI panel to a subclass of the object being controlled allows it to also have position information and the ability to change that position.

[0082] 203. Bind the position information of the object to be controlled to the position information of the user interface panel so that the user interface panel changes with the position information of the object to be controlled.

[0083] The position information of the object to be controlled is bound to the position information of the user interface panel. When the audience wears mixed reality glasses, regardless of whether the audience is lying down or sitting, the player Pawn's position information will change as the audience moves back and forth. If the audience moves back and forth, the UI panel will also change accordingly. If the UI panel includes multiple buttons, these buttons will also move with the player Pawn's position, allowing users to easily click multiple buttons at any time.

[0084] This embodiment implements a portion of the interaction in the mixed reality glasses, namely, the UI panel changes according to the position information of the object to be controlled.

[0085] In one embodiment, the UI panel also provides at least one button. By triggering the button (e.g., pressing the button), a target first event associated with the button is executed. This target first event allows the use of a corresponding augmented reality virtual asset or the implementation of other functions. The button can be triggered directly or through operation command data in a broadcast control data packet. In this embodiment, it is further specified that the corresponding augmented reality virtual asset can be used through the target first event associated with the button on the mixed reality glasses, or other functions can be implemented to achieve multi-functional interaction within the mixed reality glasses.

[0086] It is important to note that whether the UI panel changes according to the position information of the object being controlled, or other functions are implemented, the interactions in these mixed reality glasses are all conducted from the first-person perspective of the audience.

[0087] In one embodiment, the mixed reality glasses also provide a first control object, and the data processing method applied to the mixed reality glasses further includes: when the first control object is detected to be triggered, popping up a live broadcast panel; obtaining a live broadcast address and obtaining a real-time live broadcast image from the live broadcast address; and displaying the real-time live broadcast image in the live broadcast panel.

[0088] The first control object can be placed on the UI panel or exist independently. This first control object can be a button, an icon, a thumbnail, or other forms of display. For example, there might be a small holographic panel directly below the mixed reality glasses; this small holographic panel is the first control object. When this holographic panel is triggered, the first control object is activated. The activation method for the first control object can be a click or other methods, with no specific limitations.

[0089] When the first control object is triggered, a live stream panel pops up. This live stream panel can be a holographic live stream panel. The mixed reality glasses obtain the live stream address, which is an address supported by the RTP protocol or other protocols, to retrieve the live stream from the network. After obtaining the live stream, the real-time live stream is displayed in the live stream panel.

[0090] This embodiment can be used in scenarios where a short video, such as a VCR, or a game scene is cut in from the live stream, or it can be adapted to scenarios where live viewers want to watch a live stream. For example, in the scenario where a short video or game scene is cut in from the live stream, live viewers wearing mixed reality glasses may see an empty stage without any video or game footage. If live viewers want to watch the video or game footage, they need to look up at the large live stream screen, which may be far away from the viewers, making the information displayed on the large screen small and inconvenient to view. The solution in this embodiment allows users to switch to the live stream at any time, improving the viewing experience for live viewers.

[0091] In one embodiment, such as Figure 4 As shown, another flowchart of a data processing method is provided, which is applied to mixed reality glasses and includes the following steps.

[0092] 301, retrieve the broadcast control data packet, and retrieve the operation instruction data and live broadcast variable data in the broadcast control data packet.

[0093] For example, on the broadcast control server, the broadcast control personnel first pull real-time BP data by pressing a button, and then issue commands to trigger characters and formations by clicking separate buttons. For example, clicking a button triggers the formation animation of the first virtual character on the left, which corresponds to one operation command data. Clicking another button triggers the same formation animation content, which corresponds to another operation command data. In addition, the live broadcast variable data also needs to be combined to generate a broadcast control data packet and sent to the mixed reality glasses.

[0094] The live stream variable data is obtained by storing the game data to be presented in the virtual game live stream, which is acquired by the broadcast control server, as variables. Alternatively, it can be understood as storing the game data to be presented. For example, in the BP (Ban / Pick) phase of a virtual game live stream, this phase may involve BP data such as the sect, character, and formation chosen by the spectator team and the sect, character, and formation chosen by the opposing team. In this case, the sect, character, and formation chosen by both sides can be displayed on the live stream screen of the virtual game live stream, and correspondingly, the sect, character, and formation chosen by both sides can also be displayed in the mixed reality glasses.

[0095] 302. If the operation instruction data matches the preset first instruction data in the mixed reality glasses and the live variable data matches the preset first variable data in the mixed reality glasses, obtain the target model information corresponding to the target virtual character corresponding to the preset first variable data, and trigger the target first event in the mixed reality glasses associated with the preset first instruction data, and use the corresponding augmented reality virtual asset in the mixed reality glasses through the target first event.

[0096] In the mixed reality glasses, multiple preset first variable data are pre-set, and these preset first variable data correspond one-to-one with the live broadcast variable data in the broadcast control server. Within the mixed reality glasses, a correspondence is pre-established between these preset first variable data and the model information corresponding to the virtual character. For example, when the live broadcast variable data is "aaa", the corresponding target model information is model A. The model information can be a static model effect or a model animation, etc., without specific limitations.

[0097] When the live variable data in the broadcast control data packet matches one of the preset first variable data among multiple preset first variable data, the target model information corresponding to the target virtual character corresponding to the preset first variable data is obtained. This target model information can be the model information corresponding to the currently selected sect character and / or formation.

[0098] In some embodiments, the matching of operation instruction data with preset first instruction data in the mixed reality glasses, and the matching of live variable data with preset first variable data in the mixed reality glasses, are not necessarily in an AND relationship: as long as the operation instruction data matches the preset first instruction data in the mixed reality glasses, a target first event associated with the preset first instruction data in the mixed reality glasses is triggered; as long as the live variable data matches the preset first variable data in the mixed reality glasses, the step of obtaining the target model information corresponding to the target virtual character corresponding to the preset first variable data is triggered.

[0099] 303, Display target model information and the corresponding augmented reality virtual asset usage effect through mixed reality glasses.

[0100] The display position of the target model information is predetermined, such as displaying it in the center of the mixed reality glasses. The target model information is then displayed using the mixed reality glasses.

[0101] For any parts not described in detail in this embodiment, please refer to the corresponding descriptions above.

[0102] This embodiment integrates virtual assets displayed in mixed reality glasses with virtual game live streaming, allowing viewers to see relevant content from the virtual game live stream through the mixed reality glasses. This enables on-site viewers to see the relevant content from a first-person perspective, improving their viewing experience.

[0103] In one embodiment, before acquiring the broadcast control data packet, the method further includes: lightweighting the model information corresponding to virtual assets, such as at least one augmented reality virtual asset or at least one virtual character. This lightweighting process includes model polygon reduction and / or special effects adjustment. Since the performance of mixed reality glasses is lower than that of a server, the virtual assets in the mixed reality glasses are lightweighted. For example, the models in the virtual assets are polygon-reduced to decrease the number of faces and reduce performance consumption. Additionally, the special effects generated in the virtual assets can be adjusted to reduce the performance consumed by the effects.

[0104] like Figure 5 The present application provides a flowchart of a data processing method applied to a broadcast control server. The data processing method applied to the broadcast control server specifically includes the following steps.

[0105] 401. Obtain the playback control panel, which includes at least one playback control button.

[0106] The broadcast control panel is built on a broadcast control engine. Live broadcast control personnel operate the broadcast control panel of the broadcast control engine according to the live broadcast process. The broadcast control panel includes at least one broadcast control button, and usually includes multiple broadcast control buttons. Each play button corresponds to an operation command data, and the operation command data corresponding to each different play button is different.

[0107] For example, taking the BP (Ban / Pick) phase in a virtual game live stream as an example, the live stream controller first presses a button (this button can be set on the control panel or not) to retrieve real-time BP data. Then, by triggering two control buttons on the control panel, they issue commands to activate the character and formation respectively, obtaining two different operation command data based on these commands. For example, the first control button triggers the formation animation of the first virtual character on the left, corresponding to one operation command data, such as ZFL; the second control button triggers the animation content of that formation, such as the Tiger Roar Formation, corresponding to one operation command data, such as HXZ.

[0108] In one embodiment, such as Figure 5 As shown, before step 402, the data processing method further includes step 402a.

[0109] 402a, set a trigger event for each play control button, associate the trigger event with the target first event of the corresponding button provided in the mixed reality glasses, and associate the trigger event with the second event of the corresponding button provided in the virtual reality server.

[0110] The triggering event can be a button press or other operation event; taking a button press as an example, the triggering event of the button is associated with a first target event and a second event. This ensures that pressing the button automatically triggers the first target event in the mixed reality glasses, allowing the first target event to be executed. For example, the first target event could be used to access augmented reality virtual assets in the glasses and display the effects of those assets. Simultaneously, the second event can be automatically triggered on the virtual reality server, allowing the second event to be executed, such as retrieving model information of a corresponding virtual character.

[0111] 402. When the target trigger event of at least one of the target control buttons is triggered, generate the corresponding broadcast control data packet based on the target trigger event.

[0112] When at least one target play control button is triggered, such as when the target play control button is pressed, the target trigger event corresponding to the target play control button is triggered. Based on the function implemented by the target trigger event / the triggered instruction, the operation instruction data corresponding to the corresponding function / instruction is obtained, and the play control data packet corresponding to the target play control button is generated based on the operation instruction data.

[0113] For example, when two target play control buttons are triggered as described above, the corresponding operation instruction data are ZFL and HXZ, respectively. Play control data packets for the two target play control buttons are generated based on the operation instruction data, and the content of the play control data packets includes ZFL-HXZ, etc.

[0114] 403, Send a broadcast control data packet to the mixed reality glasses so that the mixed reality glasses trigger the target first event associated with the target trigger event of the target broadcast control button according to the broadcast control data packet.

[0115] In one embodiment, prior to the step of sending broadcast control data packets to the mixed reality glasses, the data processing method further includes: obtaining a first IP address and a first port of a first server; creating a transmission control protocol in the first server based on the first IP address and the first port, such as creating a transmission control protocol server, and establishing a first communication connection with the mixed reality glasses through the transmission control protocol. In this embodiment, it is assumed that a transmission control protocol has already been created in the mixed reality glasses, such as creating a transmission control protocol client. Correspondingly, the step of sending broadcast control data packets to the mixed reality glasses includes: sending the broadcast control data packets to the mixed reality glasses through the first communication connection.

[0116] The broadcast control data packet triggers a target first event associated with the target trigger event of the target broadcast control button to execute the target first event. Further, the operation instruction data in the broadcast control data packet triggers the target first event associated with the target trigger event of the target broadcast control button; for more details, please refer to the data processing method applied to mixed reality glasses described above.

[0117] In this embodiment, the execution of the target first event associated with the triggering event in the mixed reality glasses is automatically triggered by the triggering event on the broadcast control server, so as to automatically use the corresponding augmented reality virtual assets in the mixed reality environment and automatically display the use effect of the augmented reality virtual assets in the mixed reality glasses, so that the audience can watch the use effect of the augmented reality virtual assets from a first-person perspective.

[0118] In one embodiment, while performing step 403, the data processing method also performs step 404.

[0119] 404, Send a broadcast control data packet to the second server so that the second server triggers the target second event associated with the target trigger event of the target broadcast control button based on the broadcast control data packet.

[0120] The second server is the virtual reality server. The virtual reality server includes the same virtual assets as those in the mixed reality glasses, such as at least one augmented reality virtual asset. By using the corresponding augmented reality virtual asset in the virtual reality server to capture live footage from a real camera through the target second event, the virtual reality server synthesizes the live footage with the corresponding virtual image of the augmented reality virtual asset to obtain a live broadcast image, and then sends the live broadcast image to the live broadcast platform.

[0121] The second event can also be a press event, a touch event, or a gesture event. Press events, touch events, and gesture events all include functions corresponding to the events. These functions implement the functionality that matches the second event. The functions corresponding to each different event have different functionalities.

[0122] Before sending the broadcast control data packet to the second server, the data processing method further includes: establishing a second communication connection with the virtual reality server through the transmission control protocol of the broadcast control server. In this embodiment, it is assumed that a transmission control protocol has already been created in the virtual reality server, such as by creating a transmission control protocol client. Specifically, a second communication connection is established with the virtual reality server through the transmission control protocol. Correspondingly, the step of sending the broadcast control data packet to the second server, i.e., the virtual reality server, through the second communication connection.

[0123] The second server triggers a second target event associated with the target trigger event of the target broadcast control button based on the broadcast control data packet. Furthermore, it triggers a second target event associated with the target trigger event of the target broadcast control button based on the operation instruction data in the broadcast control data packet. Further details can be found in the data processing method applied to the second server below.

[0124] Steps 403 and 404 can be executed simultaneously. If executed simultaneously, the broadcast control server sends the broadcast control data packet to the mixed reality glasses and the virtual reality server at the same time. This ensures that the time when the mixed reality glasses and the virtual reality server receive the broadcast control data packet is kept as synchronized as possible, and the time when the corresponding augmented reality virtual asset usage effect is displayed is also kept as synchronized as possible. This ensures that the display of the augmented reality virtual asset usage effect of the mixed reality glasses (live scene) and the display of the augmented reality virtual asset in the live broadcast of the virtual reality server (live scene) are kept synchronized, avoiding delays.

[0125] In some embodiments, in addition to controlling the display of augmented reality virtual assets, the broadcast control server also needs to control the display of data involved in the virtual game live stream. This is closely integrated with the virtual game live stream scenario; for details, please refer to [link / reference needed]. Figure 6 Furthermore, a data processing method is provided, which is applied in a broadcast control server and includes the following steps.

[0126] 501, Obtain the broadcast control panel and the data interface in the virtual game live stream, the broadcast control panel including at least one broadcast control button.

[0127] The data interface for virtual game live streaming can be provided by the game developer. This interface can be a link, such as a JSON link or other format, containing the current real-time virtual game data. A JSON link refers to a link that stores data using JSON format, or in other words, the data within that link is stored in JSON format. It's important to note that JSON format is just one example; other formats can also be used for data storage.

[0128] In one embodiment, the data interface in the virtual game live stream can be obtained by triggering a corresponding button in the broadcast control server. For example, when the corresponding button is clicked, the data interface in the virtual game live stream is automatically obtained.

[0129] 502, Parse the data interface to determine the real-time game data to be presented, and save the game data to be presented as live stream variable data.

[0130] The virtual game data in the JSON link includes a lot of information, such as the health and combat data of the first player on the left. However, during virtual game live streaming, not all of the virtual game data is needed, such as health and combat data. Only some important data that needs to be presented is required, i.e., the game data to be presented. Taking the BP (Ban / Pick) phase as an example, the game data to be presented includes the virtual character data selected by the current player, or the sect and formation selected by the spectator and the sect and formation selected by the opponent.

[0131] Therefore, the data interface is parsed to obtain the virtual game data within it. The data to be presented, such as virtual character data, is then extracted from this virtual game data and saved as live stream variable data. This process allows for rapid retrieval of the game data to be presented through variables set within the virtual game.

[0132] 503. When the target trigger event of at least one of the target control buttons is triggered, generate the corresponding broadcast control data packet for the target broadcast control button based on the target trigger event and live broadcast variable data.

[0133] After obtaining the operation instruction data corresponding to the target trigger event, the operation instruction data and live variable data are combined to generate the broadcast control data packet corresponding to the target broadcast control button. This can also be understood as the broadcast control data packet including operation instruction data and live variable data. Obtaining the operation instruction data corresponding to the target trigger event includes: when the target trigger event corresponding to the target broadcast control button is triggered, obtaining the operation instruction data corresponding to the corresponding function / instruction based on the function implemented by the target trigger event / the triggered instruction.

[0134] 504, Send a broadcast control data packet to the mixed reality glasses so that the mixed reality glasses trigger the target first event associated with the target trigger event of the target broadcast control button according to the broadcast control data packet.

[0135] 505, Send a broadcast control data packet to the second server so that the second server triggers the target second event associated with the target trigger event of the target broadcast control button based on the broadcast control data packet.

[0136] Steps 503 and 504 can be executed simultaneously, as described above.

[0137] For steps not described in detail in this embodiment, please refer to the corresponding descriptions above, and they will not be repeated here.

[0138] In this embodiment, the broadcast control server acquires virtual game data from virtual game live streams, such as e-sports event live streams, and parses the virtual game data to obtain the game data to be presented. Based on the game data to be presented and operation instruction data, it generates a broadcast control data packet and sends it to the mixed display glasses and virtual reality server. This allows the on-site audience wearing the mixed display glasses to view the game data to be presented from a first-person perspective, making the data displayed in the mixed display glasses closely related to the virtual game data, and also ensuring that the live stream also includes the game data to be presented.

[0139] Figure 7 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method is applied to a second server, namely a virtual reality server, and includes the following steps.

[0140] 601, to obtain live footage from a real camera and to obtain broadcast control data packets.

[0141] like Figure 1 As shown, the real camera will capture scene information of the live broadcast and send the captured scene to the virtual AR engine of the virtual reality server. The virtual AR engine of the virtual reality server will receive the scene captured by the real camera to obtain the scene.

[0142] Prior to the step of the virtual reality server acquiring the broadcast control data packet, the method further includes: creating a transmission control protocol, such as a transmission control protocol client, in the virtual reality server, and establishing a communication connection with the broadcast control server.

[0143] In one embodiment, the step of creating a transmission control protocol in a virtual reality server includes: obtaining a second IP address and a second port of the virtual reality server; and creating a transmission control protocol in the virtual reality server based on the second IP address and the second port, such as creating a transmission control protocol client. The second port is a separate, unused port in the virtual reality server.

[0144] In one embodiment, the step of establishing a communication connection with a broadcast control server includes: obtaining a first IP address and a first port of the broadcast control server that has a transmission control protocol; and establishing a second communication connection with the broadcast control server through the transmission control protocol based on the second IP address, the second port, the first IP address, and the first port.

[0145] Correspondingly, the steps for obtaining broadcast control data packets include: receiving broadcast control data packets sent by the broadcast control server through a second communication connection. The necessity of creating a transmission control protocol will not be elaborated further.

[0146] 602. When the operation instruction data in the broadcast control data packet matches the preset second instruction data in the second server, a target second event associated with the preset second instruction data is triggered in the second server, and the corresponding augmented reality virtual asset in the second server is used through the target second event.

[0147] In the virtual reality server, multiple preset second instruction data are pre-set, and these multiple preset second instruction data correspond one-to-one with the operation instruction data in the broadcast control server.

[0148] In a virtual reality server, multiple preset second instruction data are associated with second events in the virtual reality server. For example, to control the playback of a certain augmented reality virtual asset, a preset second instruction data, such as HXZ, is set, and the preset second instruction data is associated with the corresponding second event. One preset second instruction data corresponds to one second event.

[0149] The virtual reality server includes virtual assets, which include at least one augmented reality virtual asset, and each augmented reality virtual asset also includes location information that is pre-set according to the site conditions.

[0150] When the operation command data in the broadcast control data matches the preset second command data in the virtual reality server, the target second event associated with the preset second command data in the virtual reality server is triggered to execute the target second event, such as calling and executing the function of the target second event, and using the corresponding augmented reality virtual asset in the virtual reality server through the target second event, such as playing the corresponding augmented reality virtual asset, such as playing the tiger animation corresponding to the Tiger Roar Formation.

[0151] It should be noted that although virtual assets exist in the virtual reality server, the use of virtual assets in this embodiment is controlled by the broadcast control server, specifically by sending broadcast control data packets. Therefore, unlike the prior art where the target second event is triggered by clicking a button in the virtual reality server, this application requires obtaining the broadcast control data packet, and the target second event is only triggered when the operation instruction data of the broadcast control data packet matches the preset second instruction data in the virtual reality server.

[0152] 603, the virtual images corresponding to the usage effects of the corresponding augmented reality virtual assets are combined with the on-site images to obtain the live broadcast images.

[0153] If the real camera information of the actual camera is fixed, there is no need to adjust the real camera information throughout the live broadcast. Instead, the real camera information is set to be consistent with the virtual camera information. The virtual camera is then used to project and map the effects of the corresponding augmented reality virtual assets to obtain a corresponding virtual image. This virtual image is then combined with the live image to create the live broadcast. The resulting live broadcast is then sent to a live streaming platform or a rebroadcasting platform.

[0154] The real camera information includes the location and lens information of the real camera, such as zoom and pan information. The virtual camera information includes the same information. Setting the real and virtual camera information to be consistent ensures that the phases of the real and virtual cameras are the same, allowing for proportional scaling and panning of the virtual and live images, resulting in a more harmonious composite live stream.

[0155] If the real camera information of the real camera is adjusted during the live broadcast, the virtual reality server also needs to obtain the real camera information of the real camera in real time, set the virtual camera information of the virtual camera according to the real camera information, so that the real camera information is consistent with the set virtual camera information, and then use the virtual camera information of the virtual camera to perform projection mapping processing on virtual assets such as at least one augmented reality virtual asset. The virtual image corresponding to the usage effect of the augmented reality virtual asset after the corresponding projection mapping processing is combined with the live image to obtain the live broadcast image.

[0156] This embodiment further specifies the acquisition of broadcast control data packets. The corresponding augmented reality virtual assets are only triggered when the operation instruction data in the broadcast control data packets matches the preset second instruction data in the virtual reality server. This is different from the prior art, which is different from directly clicking a button in the second server to trigger the use of the corresponding augmented reality virtual assets. In this application, the augmented reality virtual assets are triggered through broadcast control data packets.

[0157] like Figure 8 As shown, a flowchart of a data processing method applied in a virtual reality server is provided, which includes the following steps.

[0158] 701, acquire the live footage captured by the real camera, the real camera information, and the broadcast control data packet, which includes operation instruction data and live broadcast variable data.

[0159] 702, Set the virtual camera information of the virtual camera according to the real camera information so that the real camera information is consistent with the set virtual camera information, and use the virtual camera information of the virtual camera to perform projection mapping processing on at least one augmented reality virtual asset.

[0160] The real camera information is consistent with the configured virtual camera information, including: the position information of the real camera is consistent with the position information of the virtual camera, and if the real camera zooms in and out, the virtual camera also zooms in and out in the same way. Using the configured virtual camera information, at least one augmented reality virtual asset in the virtual scene is projected and mapped to make the phase of the virtual camera the same as that of the real camera, resulting in a more harmonious composite live broadcast.

[0161] 703. If the operation instruction data in the broadcast control data packet matches the preset second instruction data in the second server and the live variable data matches the preset second variable data in the second server, obtain the target model information corresponding to the target virtual character corresponding to the preset second variable data, and trigger the target second event in the second server associated with the preset second instruction data, and use the corresponding augmented reality virtual asset in the second server through the target second event.

[0162] The virtual reality server also includes model information for at least one virtual character involved in the virtual game live stream. It's important to note that the virtual assets in the virtual reality server are the same as those in the mixed reality glasses, but the virtual assets in the mixed reality glasses may have been lightweighted.

[0163] In the virtual reality server, multiple preset second variable data are pre-set, and these preset second variable data correspond one-to-one with the live broadcast variable data in the broadcast control server. In the mixed reality glasses, a correspondence is pre-set between the multiple preset second variable data and the model information corresponding to the virtual character.

[0164] When the live variable data in the broadcast control data packet matches one of the preset second variable data among multiple preset second variable data, the target model information corresponding to the target virtual character corresponding to the preset second variable data is obtained. This target model information may be the model information corresponding to the currently selected sect character and / or formation, etc.

[0165] In some embodiments, the matching of operation instruction data with preset second instruction data in the virtual reality server, and the matching of live stream variable data with preset second variable data in the virtual reality server, are not necessarily an AND relationship, meaning that both conditions do not necessarily have to be met simultaneously. Specifically, as long as the operation instruction data matches the preset second instruction data in the virtual reality server, a target second event associated with the preset second instruction data is triggered in the virtual reality server; and as long as the live stream variable data matches the preset second variable data in the virtual reality server, the step of obtaining the target model information corresponding to the target virtual character corresponding to the preset second variable data is triggered.

[0166] 704. The virtual screen corresponding to the usage effect of the augmented reality virtual asset and the display effect of the target model information after the corresponding projection mapping processing is combined with the on-site screen to obtain the live broadcast screen.

[0167] This embodiment displays the game data to be presented in the virtual game live stream on the live stream screen.

[0168] like Figure 9 The image shown is a schematic diagram of a live stream provided by a virtual reality server during a virtual game live stream. The upper half of the stream includes footage captured by a real camera. In the lower left corner of the upper half, a virtual tiger is displayed, corresponding to the tiger in the Tiger Roar formation. In the upper right corner, there is a virtual phoenix. The lower half of the stream displays the game data to be presented, including model information for the virtual characters selected by both sides.

[0169] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0170] To facilitate better implementation of the data processing method of the embodiments of this application, the embodiments of this application also provide a data processing apparatus.

[0171] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application, which is applied in mixed reality glasses. The data processing device 800 may include a first acquisition module 801, a first triggering module 802, and a display module 803.

[0172] The first acquisition module 801 is used to acquire broadcast control data packets and acquire operation instruction data in the broadcast control data packets.

[0173] The first triggering module 802 is used to trigger a target first event associated with the preset first instruction data in the mixed reality glasses if the operation instruction data matches the preset first instruction data in the mixed reality glasses, and to use the corresponding augmented reality virtual asset in the mixed reality glasses through the target first event.

[0174] Display module 803 is used to display the effect of using the corresponding augmented reality virtual assets through the mixed reality glasses.

[0175] In one embodiment, the data processing device 800 further includes a location acquisition module 804 and a binding setting module 805. The location acquisition module 804 is used to acquire the location information of the object to be controlled, which is determined based on the location information of the audience. The binding setting module 805 is used to set the user interface panel in the mixed reality glasses to a subclass of the class to which the object to be controlled belongs; and to bind the location information of the object to be controlled to the location information of the user interface panel, so that the user interface panel changes according to the change in the location information of the object to be controlled.

[0176] In one embodiment, the mixed reality glasses also provide a first control object, and the data processing device 800 further includes a live streaming acquisition module 806. The live streaming acquisition module 806 is configured to pop up a live streaming screen panel when the first control object is detected to be triggered; acquire a live streaming address; and acquire a real-time live streaming image from the live streaming address. Correspondingly, the display module 803 is further configured to display the real-time live streaming image in the live streaming screen panel.

[0177] In one embodiment, the data processing device 800 further includes a first creation and establishment module 807. The first creation and establishment module 807 is used to acquire the glasses IP and glasses port of the mixed reality glasses, and the first IP and first port of a first server with a transmission control protocol; based on the glasses IP and glasses port, it creates a transmission control protocol in the mixed reality glasses, and establishes a first communication connection with the first server through the transmission control protocol based on the glasses IP, glasses port, the first IP, and the first port. Correspondingly, the first acquisition module 801 is used to receive broadcast control data packets sent by the first server through the first communication connection.

[0178] In one embodiment, the mixed reality glasses further include model information corresponding to at least one virtual character involved in the virtual game live stream, and the broadcast control data packet further includes live stream variable data; correspondingly, the first trigger module 802 is further configured to obtain target model information corresponding to the target virtual character corresponding to the preset first variable data if the live stream variable data matches the preset first variable data in the mixed reality glasses; the display module 803 is further configured to display the target model information through the mixed reality glasses.

[0179] Please see Figure 11 , Figure 11 This is a schematic diagram of a data processing device provided in an embodiment of this application. The data processing device is applied in a first server, namely a broadcast control server. The data processing device 900 may include a second acquisition module 901, a generation module 902, and a sending module 903.

[0180] The second acquisition module 901 is used to acquire the broadcast control panel, which includes at least one broadcast control button.

[0181] The generation module 902 is used to generate a broadcast control data packet corresponding to the target broadcast control button according to the target trigger event when the target trigger event of the target broadcast control button among at least one broadcast control button is triggered.

[0182] The sending module 903 is used to send the broadcast control data packet to the mixed reality glasses, so that the mixed reality glasses trigger a target first event associated with the target trigger event of the target broadcast control button according to the broadcast control data packet.

[0183] In one embodiment, the sending module 903 is further configured to send the broadcast control data packet to the second server, so that the second server triggers a target second event associated with the target trigger event of the target broadcast control button according to the broadcast control data packet; wherein, the second server includes at least one augmented reality virtual asset, and by using the corresponding augmented reality virtual asset in the second server through the target second event, the second server acquires the live footage captured by the real camera, and synthesizes the live footage with the virtual footage corresponding to the corresponding augmented reality virtual asset to obtain a live broadcast.

[0184] In one embodiment, the data processing device 900 further includes a second creation and establishment module 904. The second creation and establishment module 904 is used to obtain the first IP address and first port of the first server; create a transmission control protocol in the first server based on the first IP address and first port, and establish a first communication connection with the mixed reality glasses through the transmission control protocol; correspondingly, the second acquisition module 901 is specifically used to send the broadcast control data packet to the mixed reality glasses through the first communication connection.

[0185] In one embodiment, the data processing device 900 further includes a game data processing module 905. The game data processing module 905 is used to acquire data interfaces in the virtual game live stream; parse the data interfaces to determine the game data to be presented in real time, and save the game data to be presented as live stream variable data; correspondingly, the generation module 902 is specifically used to generate a broadcast control data packet corresponding to the target broadcast control button based on the target trigger event and the live stream variable data.

[0186] Please see Figure 12 , Figure 12 This is another schematic diagram of the data processing device provided in an embodiment of this application. The data processing device is applied in a second server, namely a virtual reality server. The data processing device 1000 may include a third acquisition module 1001, a second triggering module 1002, and a synthesis module 1003.

[0187] The third acquisition module 1001 is used to acquire live footage captured by a real camera and to acquire broadcast control data packets.

[0188] The second triggering module 1002 is used to trigger a target second event associated with the preset second instruction data in the second server when the operation instruction data in the broadcast control data packet matches the preset second instruction data in the second server, and to use the corresponding augmented reality virtual asset in the second server through the target second event.

[0189] The compositing module 1003 is used to compose the virtual image corresponding to the usage effect of the corresponding augmented reality virtual asset with the on-site image to obtain a live broadcast image.

[0190] In one embodiment, the second server further includes model information corresponding to at least one virtual character involved in the virtual game live stream, and the broadcast control data packet further includes live stream variable data. Correspondingly, the second triggering module 1002 is also used to obtain target model information corresponding to the target virtual character corresponding to the preset second variable data when the live stream variable data matches preset second variable data in the second server. Correspondingly, the compositing module 1003 is specifically used to composite the virtual screen corresponding to the usage effect of the corresponding augmented reality virtual asset and the display effect of the target model information with the live screen to obtain the live stream screen.

[0191] In one embodiment, the data processing device 1000 may further include a virtual camera processing module 1004. The virtual camera processing module 1004 is used to acquire real camera information from a real camera, and set virtual camera information for a virtual camera based on the real camera information, so that the real camera information is consistent with the set virtual camera information; using the virtual camera information, it performs projection mapping processing on at least one augmented reality virtual asset. Correspondingly, the compositing module 1003 is specifically used to composite the virtual image corresponding to the usage effect of the augmented reality virtual asset after the corresponding projection mapping processing with the live image to obtain a live broadcast image.

[0192] In one embodiment, the data processing device 1000 may further include a third creation and establishment module 1005. The third creation and establishment module 1005 is used to obtain the second IP address and second port of the second server; based on the second IP address and second port, it creates a transmission control protocol in the second server and establishes a second communication connection with the first server through the transmission control protocol. Correspondingly, the third acquisition module 1001 is specifically used to receive broadcast control data packets sent by the first server through the second communication connection.

[0193] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0194] Accordingly, embodiments of this application also provide an electronic device, which can be mixed reality glasses or a server. For example... Figure 13 As shown, Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1100 includes a processor 1101 with one or more processing cores, a memory 1102 with one or more computer-readable storage media, and a computer program stored in the memory 1102 and executable on the processor. The processor 1101 and the memory 1102 are electrically connected.

[0195] The processor 1101 is the control center of the electronic device 1100. It connects various parts of the electronic device 1100 through various interfaces and lines. By running or loading software programs (computer programs) and / or modules stored in the memory 1102, and calling data stored in the memory 1102, it performs various functions of the electronic device 1100 and processes data, thereby monitoring the electronic device 1100 as a whole.

[0196] In this embodiment, the processor 1101 in the electronic device 1100 loads instructions corresponding to the processes of one or more applications into the memory 1102 according to the following steps, and the processor 1101 runs the applications stored in the memory 1102 to achieve various functions, such as implementing the data processing method of any embodiment applied to mixed reality glasses, or implementing the data processing method of any embodiment applied to the first server, or implementing the data processing method of any embodiment applied to the second server. For example:

[0197] Acquire broadcast control data packets and obtain operation instruction data from the broadcast control data packets; if the operation instruction data matches preset first instruction data in the mixed reality glasses, trigger a target first event associated with the preset first instruction data in the mixed reality glasses, and use the corresponding augmented reality virtual asset in the mixed reality glasses through the target first event; display the usage effect of the corresponding augmented reality virtual asset through the mixed reality glasses; or...

[0198] A playback control panel is obtained, the playback control panel including at least one playback control button; when a target trigger event of a target playback control button is triggered, a playback control data packet corresponding to the target playback control button is generated according to the target trigger event; the playback control data packet is sent to the mixed reality glasses, so that the mixed reality glasses trigger a target first event associated with the target trigger event of the target playback control button according to the playback control data packet; or...

[0199] The system acquires live footage captured by a real camera and a broadcast control data packet. When the operation instruction data in the broadcast control data packet matches the preset second instruction data in the second server, a target second event associated with the preset second instruction data is triggered in the second server. The corresponding augmented reality virtual asset in the second server is used through the target second event. The virtual image corresponding to the use effect of the corresponding augmented reality virtual asset is combined with the live footage to obtain the live broadcast image.

[0200] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0201] Optional, such as Figure 13 As shown, the electronic device 1100 also includes: a touch display screen 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch display screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107. Those skilled in the art will understand that... Figure 12 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0202] The touch display screen 1103 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 1103 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), or other similar display technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program. The touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the data to the processor 1101 to determine the type of touch event. Subsequently, the processor 1101 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and display panel can be integrated into the touch display screen 1103 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 1103 can be used as two independent components to achieve input and output functions. That is, the touch display screen 1103 can also be used as part of the input unit 1106 to achieve input functions.

[0203] In this embodiment, the touch display screen 1103 is used to present a graphical user interface and receive operation commands generated by the user interacting with the graphical user interface.

[0204] The radio frequency circuit 1104 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other devices, and to transmit and receive signals with network devices or other devices.

[0205] Audio circuitry 1105 can be used to provide an audio interface between a user and a device via a speaker and a microphone. Audio circuitry 1105 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 1105, converted back into audio data, and then processed by processor 1101 before being transmitted via radio frequency circuitry 1104 to, for example, another device, or output to memory 1102 for further processing. Audio circuitry 1105 may also include an earphone jack to provide communication between peripheral headphones and the device.

[0206] The input unit 1106 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0207] Power supply 1107 is used to supply power to various components of electronic device 1100. Optionally, power supply 1107 can be logically connected to processor 1101 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 1107 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0208] although Figure 13 As not shown in the diagram, the electronic device 1100 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0209] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0210] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0211] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute steps in any of the data processing methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0212] Acquire broadcast control data packets and obtain operation instruction data from the broadcast control data packets; if the operation instruction data matches preset first instruction data in the mixed reality glasses, trigger a target first event associated with the preset first instruction data in the mixed reality glasses, and use the corresponding augmented reality virtual asset in the mixed reality glasses through the target first event; display the usage effect of the corresponding augmented reality virtual asset through the mixed reality glasses; or...

[0213] A playback control panel is obtained, the playback control panel including at least one playback control button; when a target trigger event of a target playback control button is triggered, a playback control data packet corresponding to the target playback control button is generated according to the target trigger event; the playback control data packet is sent to the mixed reality glasses, so that the mixed reality glasses trigger a target first event associated with the target trigger event of the target playback control button according to the playback control data packet; or...

[0214] The system acquires live footage captured by a real camera and a broadcast control data packet. When the operation instruction data in the broadcast control data packet matches the preset second instruction data in the second server, a target second event associated with the preset second instruction data is triggered in the second server. The corresponding augmented reality virtual asset in the second server is used through the target second event. The virtual image corresponding to the use effect of the corresponding augmented reality virtual asset is combined with the live footage to obtain the live broadcast image.

[0215] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0216] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0217] Since the computer program stored in the storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0218] The data processing method, apparatus, storage medium, and device provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, Applied to mixed reality glasses, the mixed reality glasses including at least one augmented reality virtual asset, the method includes: Acquire broadcast control data packets and acquire operation instruction data from the broadcast control data packets; If the operation instruction data matches the preset first instruction data in the mixed reality glasses, a target first event associated with the preset first instruction data is triggered in the mixed reality glasses, and the corresponding augmented reality virtual asset in the mixed reality glasses is used through the target first event; The mixed reality glasses display the usage effects of corresponding augmented reality virtual assets. The mixed reality glasses also include a user interface panel, which provides at least one button. By triggering the button, a target first event associated with the button is executed, and the corresponding augmented reality virtual asset is used through the target first event.

2. The method according to claim 1, characterized in that, Before acquiring the broadcast control data packet, the method further includes: The location information of the object to be controlled is obtained, and the location information of the object to be controlled is determined based on the location information of the audience on site. The method further includes: Set the user interface panel to a subclass of the class to which the object to be controlled belongs; The position information of the object to be controlled is bound to the position information of the user interface panel, so that the user interface panel changes according to the position information of the object to be controlled.

3. The method according to claim 1, characterized in that, The mixed reality glasses also provide a first control object, and the method further includes: When the first controlled object is detected to be triggered, a live broadcast panel will pop up; Obtain the live stream address and retrieve the live stream video from the live stream address; The live stream is displayed in the live stream panel.

4. The method according to claim 1, characterized in that, Before acquiring the broadcast control data packet, the method further includes: Obtain the IP address and port of the mixed reality glasses, and create the first IP address and port of the first server with the transmission control protocol; Based on the glasses IP and the glasses port, a transmission control protocol is created in the mixed reality glasses, and a first communication connection with the first server is established through the transmission control protocol based on the glasses IP, the glasses port, the first IP and the first port; The step of obtaining the broadcast control data packet includes: receiving the broadcast control data packet sent by the first server through the first communication connection.

5. The method according to claim 1, characterized in that, The mixed reality glasses also include model information corresponding to at least one virtual character involved in the virtual game live stream, and the broadcast control data packet also includes live stream variable data. The method further includes: If the live stream variable data matches the preset first variable data in the mixed reality glasses, obtain the target model information corresponding to the target virtual character corresponding to the preset first variable data; The target model information is displayed through the mixed reality glasses.

6. A data processing method, characterized in that, include: Obtain a broadcast control panel, wherein the broadcast control panel includes at least one broadcast control button; When a target trigger event of a target broadcast control button is triggered in at least one broadcast control button, a broadcast control data packet corresponding to the target broadcast control button is generated according to the target trigger event; The playback control data packet is sent to the mixed reality glasses so that the mixed reality glasses trigger a target first event associated with the target trigger event of the target playback control button according to the playback control data packet. The mixed reality glasses include a user interface panel, and the user interface panel also provides at least one button. By triggering the button, the target first event associated with the button is executed, and the corresponding augmented reality virtual asset is used through the target first event.

7. The method according to claim 6, characterized in that, Also includes: The broadcast control data packet is sent to the second server so that the second server triggers a target second event associated with the target trigger event of the target broadcast control button based on the broadcast control data packet; The second server includes at least one augmented reality virtual asset. By using the corresponding augmented reality virtual asset in the second server through the target second event, the second server can acquire the live footage captured by a real camera and synthesize the live footage with the virtual footage corresponding to the augmented reality virtual asset to obtain a live broadcast.

8. The method according to claim 6, characterized in that, Before sending the broadcast control data packet to the mixed reality glasses, the method further includes: Obtain the first IP address and first port of the first server; Based on the first IP address and the first port, a transmission control protocol is created in the first server, and a first communication connection with the mixed reality glasses is established through the transmission control protocol; The step of sending the broadcast control data packet to the mixed reality glasses includes: sending the broadcast control data packet to the mixed reality glasses through the first communication connection.

9. The method according to claim 6, characterized in that, Also includes: Obtain data interfaces from virtual game live streams; The data interface is parsed to determine the game data to be presented in real time, and the game data to be presented is saved as live stream variable data; The step of generating the broadcast control data packet corresponding to the target broadcast control button based on the target trigger event includes: generating the broadcast control data packet corresponding to the target broadcast control button based on the target trigger event and the live broadcast variable data.

10. A data processing method, characterized in that, The method is applied to a second server, which includes at least one augmented reality virtual resource and model information corresponding to at least one virtual character involved in the virtual game live stream. The method includes: The system acquires live footage captured by a real camera and acquires broadcast control data packets, wherein the broadcast control data packets include operation instruction data and live broadcast variable data. When the operation instruction data in the broadcast control data packet matches the preset second instruction data in the second server, a target second event associated with the preset second instruction data is triggered in the second server, and the corresponding augmented reality virtual asset in the second server is used through the target second event; The virtual images corresponding to the usage effects of the corresponding augmented reality virtual assets are combined with the on-site images to obtain the live broadcast images; When the live stream variable data matches the preset second variable data in the second server, the target model information corresponding to the target virtual character corresponding to the preset second variable data is obtained; The step of combining the virtual images corresponding to the usage effects of the corresponding augmented reality virtual assets with the on-site images to obtain a live broadcast image includes: combining the virtual images corresponding to the usage effects of the corresponding augmented reality virtual assets and the display effects of the target model information with the on-site images to obtain a live broadcast image.

11. The method according to claim 10, characterized in that, Also includes: Obtain the real camera information of a real camera, and set the virtual camera information of a virtual camera based on the real camera information, so that the real camera information and the set virtual camera information are consistent; Using the virtual camera information of the virtual camera, at least one augmented reality virtual asset is subjected to projection mapping processing; The step of combining the virtual image corresponding to the usage effect of the corresponding augmented reality virtual asset with the on-site image to obtain a live broadcast image includes: combining the virtual image corresponding to the usage effect of the corresponding augmented reality virtual asset after projection mapping processing with the on-site image to obtain a live broadcast image.

12. The method according to claim 10, characterized in that, Also includes: Obtain the second IP address and second port of the second server; Based on the second IP address and the second port, a transmission control protocol is created in the second server, and a second communication connection with the first server is established through the transmission control protocol; The step of obtaining the broadcast control data packet includes: receiving the broadcast control data packet sent by the first server through the second communication connection.

13. A data processing apparatus, characterized in that, Applied to mixed reality glasses, the mixed reality glasses including at least one augmented reality virtual asset, the device includes: The first acquisition module is used to acquire broadcast control data packets and acquire operation instruction data in the broadcast control data packets; A first triggering module is configured to, if the operation instruction data matches preset first instruction data in the mixed reality glasses, trigger a target first event associated with the preset first instruction data in the mixed reality glasses, and use the corresponding augmented reality virtual asset in the mixed reality glasses through the target first event. The mixed reality glasses also include a user interface panel, which provides at least one button. By triggering the button, the target first event associated with the button is executed, and the corresponding augmented reality virtual asset is used through the target first event. The display module is used to display the effect of using the corresponding augmented reality virtual assets through the mixed reality glasses.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the steps of the data processing method as described in any one of claims 1-12.

15. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the steps of the data processing method as described in any one of claims 1-12 by calling the computer program stored in the memory.

Images