Extended reality display method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides an extended reality display method, device, electronic equipment and storage medium. The extended reality display method comprises: in response to a trigger operation of multi-user sharing a picture, obtaining a view image corresponding to each user; generating an extended reality spherical model according to the view image corresponding to each user, the extended reality spherical model being composed of the view image corresponding to each user; and displaying a picture corresponding to the extended reality spherical model on a head-mounted terminal of a user. The method of the present disclosure can display an extended reality scene of multiple users on a head-mounted terminal of an extended reality device.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to an extended reality display method, apparatus, electronic device, and storage medium. Background Technology

[0002] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating a virtual environment that allows for human-computer interaction. However, in related technologies, single-player XR games cannot provide a "game companionship" function for multiple users; that is, while the current user is playing an XR game, they cannot switch viewpoints to observe other users' gameplay from a "first-person perspective." Summary of the Invention

[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] This disclosure provides an extended reality display method, apparatus, electronic device, and storage medium.

[0005] The following technical solution is adopted in this disclosure.

[0006] In some embodiments, this disclosure provides an extended reality display method, including:

[0007] In response to a multi-user screen sharing trigger, obtain the view image corresponding to each user;

[0008] An extended reality spherical model is generated based on the view images corresponding to each user. The extended reality spherical model is composed of the view images corresponding to each user.

[0009] The corresponding screen of the extended reality spherical model is displayed on the user's head-mounted display.

[0010] In some embodiments, this disclosure provides an extended reality display device, comprising:

[0011] The acquisition module is used to acquire the view image corresponding to each user in response to the trigger operation of multi-user screen sharing;

[0012] The processing module is used to generate an extended reality spherical model based on the view images corresponding to each user. The extended reality spherical model is composed of the view images corresponding to each user.

[0013] The display module is used to display the corresponding image of the extended reality spherical model on the user's head-mounted device.

[0014] In some embodiments, this disclosure provides an electronic device, including: at least one memory and at least one processor;

[0015] The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above method.

[0016] In some embodiments, this disclosure provides a computer-readable storage medium for storing program code that, when run by a processor, causes the processor to perform the methods described above.

[0017] The extended reality display method provided in this disclosure responds to a trigger operation of multi-user shared screen by acquiring view images corresponding to each user, and generating an extended reality spherical model based on the view images corresponding to each user. The extended reality spherical model is composed of stitched view images corresponding to each user. Finally, the corresponding screen of the extended reality spherical model is displayed on the head-mounted display of the user. The method of this disclosure can realize the display of extended reality scenes of multiple users on the head-mounted display of an extended reality device. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0019] Figure 1 This is one of the flowcharts of the extended reality display method according to an embodiment of this disclosure.

[0020] Figure 2 This is a schematic diagram of the synthesis of an extended reality spherical model according to an embodiment of this disclosure.

[0021] Figure 3 This is a schematic diagram of a multi-user shared data processing flow according to an embodiment of this disclosure.

[0022] Figure 4 This is a schematic diagram of the local archive data processing flow according to an embodiment of this disclosure.

[0023] Figure 5 This is the second flowchart of the extended reality display method according to an embodiment of this disclosure.

[0024] Figure 6 This is a rendering of an extended reality display based on an embodiment of this disclosure.

[0025] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the various steps described in the method embodiments of this disclosure can be performed in sequence and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. The term "in response to" and related terms refer to a signal or event being affected to some extent by another signal or event, but not necessarily completely or directly. If event x occurs "in response to" event y, then x may be directly or indirectly responsive to y. For example, the occurrence of y may ultimately lead to the occurrence of x, but there may be other intermediate events and / or conditions. In other cases, y may not necessarily lead to the occurrence of x, and x may occur even if y has not yet occurred. Furthermore, the term "in response to" can also mean "at least partially responsive to".

[0029] The term "determine" broadly encompasses a wide variety of actions, including acquisition, calculation, computation, processing, derivation, investigation, search (e.g., searching in a table, database, or other data structure), discovery, and similar actions; it may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), parsing, selecting, choosing, building, and similar actions, etc. Definitions for other terms will be provided below.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0031] It should be noted that the use of the word "a" in this disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as "one or more" unless otherwise expressly indicated in the context.

[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0033] The solutions provided by the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, Figure 1 This is a flowchart of an extended reality display method according to an embodiment of the present disclosure, which includes the following steps.

[0035] Step S01: In response to the trigger operation of multi-user screen sharing, obtain the view image corresponding to each user;

[0036] In some embodiments, typical extended reality videos, such as virtual reality (VR) videos, are spherical models. Users are essentially viewing the sphere from its center. Due to the limited field of view of the human eye, users can only see a 360-degree portion of the sphere at any given time. They can only see images of other spherical views when they rotate their viewpoint. The extended reality spherical model is obtained by rendering panoramic images. Therefore, when a cloud server responds to a multi-user shared view trigger, it can obtain the spherical model corresponding to each user and, based on that model, acquire the corresponding panoramic image of the extended reality scene. For example, the spherical model can be segmented, distorted, stretched, and stitched sequentially to obtain the panoramic image. The spherical model for each user is obtained by the cloud server rendering game images using their respective game engines based on sensor data uploaded by each user, and then compressing and distorting those images.

[0037] In some embodiments, the triggering operation of multi-user screen sharing can be understood as the cloud server receiving a screen sharing request signal initiated by the main user and a screen sharing request signal determined by the user.

[0038] In some embodiments, this disclosure applies to game scenarios where the game screen has a single orientation or does not require the user to rotate the headset significantly (in related technologies, the game screen perspective switching follows the control buttons, joysticks, etc., rather than the user spinning in place or frequently turning their head). It should be noted that, to achieve the "game companionship" function in multiplayer games, related technologies employ a screen partitioning display method. This method is often used in 2D game scenarios that are not virtual reality games, or for projecting desktop recordings onto a virtual reality headset. However, regardless of the display method, in a virtual reality scenario, the game screen can only be displayed through desktop projection, and the projection position is in the user's field of view, obstructing the user's own game screen and resulting in a poor user experience.

[0039] In some embodiments, the main user initiates a "game companionship" request. After the other two users agree to the request, the cloud server extracts images of portions of the spherical models from each of the three users' respective spherical models at a specified field of view. Specifically, the panoramic images corresponding to the spherical models of the three users are cropped at a specified field of view to obtain the view images of the three users respectively.

[0040] Step S02: Generate an extended reality spherical model based on the view images corresponding to each user. The extended reality spherical model is composed of the view images corresponding to each user.

[0041] In some embodiments, the cut view images are stitched together on a cloud server to form a new spherical model.

[0042] Step S03: Display the corresponding screen of the extended reality spherical model on the user's head-mounted display.

[0043] In some embodiments, based on sensor data uploaded by the main user, an image corresponding to the main user's "viewport" is extracted from the new spherical model and downloaded to the head-mounted display on the main user's end for display. It should be noted that each user can switch to "Game Companion" mode to become the "main user" on their own local end.

[0044] The extended reality display method provided in this disclosure responds to a trigger operation of multi-user shared screen by acquiring view images corresponding to each user, and generating an extended reality spherical model based on the view images corresponding to each user. The extended reality spherical model is composed of stitched view images corresponding to each user. Finally, the corresponding screen of the extended reality spherical model is displayed on the head-mounted display of the user. The method of this disclosure can realize the display of extended reality scenes of multiple users on the head-mounted display of an extended reality device.

[0045] In some embodiments, the step of obtaining the view image corresponding to each user in response to a multi-user shared screen triggering operation includes:

[0046] In response to a multi-user shared screen trigger, acquire panoramic images of each user in their extended reality scene;

[0047] The panoramic images of each user are segmented to obtain the view images corresponding to each user.

[0048] In some embodiments, acquiring panoramic images of each user in their extended reality scene includes:

[0049] The extended reality spherical model corresponding to each user is segmented to obtain a panoramic image of each user in their extended reality scene.

[0050] In some embodiments, the triggering operation in response to multi-user screen sharing includes:

[0051] Triggering operations in response to a shared screen request initiated by the main user, and triggering operations in response to a user accepting the shared screen request.

[0052] In some embodiments, the view image is the user's current perspective view in the extended reality scene.

[0053] In some embodiments, the step of segmenting the panoramic images of each user to obtain view images corresponding to each user includes:

[0054] The panoramic images of each user are segmented according to a preset field of view allocation rule to obtain the view images corresponding to each user.

[0055] In some embodiments, the step of segmenting the panoramic images of each user according to a preset field-of-view allocation rule includes:

[0056] Based on the preset field of view of the master user and the number of slave users, the panoramic images of the master user and the slave users are segmented respectively.

[0057] In some embodiments, the field of view (FOV) in a display system refers to the angle between the edge of the display and the line connecting the viewing point (the user's eye). For head-mounted displays, the optimal FOV is 120 degrees. This is because, under normal conditions, the horizontal width of the human eye when easily scanning left and right is 120 degrees, with a limit close to 180 degrees. The image presented by an extended reality head-mounted display must conform to human anatomy and behavioral habits to ensure immersion. It should be noted that the "field of view" mentioned in this disclosure is not a hardware parameter of the user's extended reality head-mounted device, but a concept used for panoramic image segmentation.

[0058] In some embodiments, such as Figure 2 As shown, to achieve the "companionship" or "interaction" function in multiplayer games within extended reality scenarios, this disclosure provides an extended reality display method. Taking a three-user multiplayer game as an example, with each user's field of view at 120 degrees (achieving a good VR immersive experience), this disclosure stitches the game screens of the three players together into a spherical model. Specifically, after user A initiates a "game companionship" request, the game screens of multiple players, including user A, are then synchronously displayed in partitions on user A's headset, within the newly stitched extended display spherical model. Furthermore, for scenarios with more than three users, besides the current main user's extended reality scene (ensuring the current user's virtual reality immersive experience, their field of view should remain at 120°), the extraction of view images (field of view images) from other users can be adjusted based on the segmentation. That is, the extracted field of view images of other "game companion" users need to be segmented (outside of 360°-120°). This segmentation can be done evenly or according to other custom proportions, without specific limitations. Furthermore, for a game involving two people, a 180° + 180° combination can be used to create an extended reality spherical model. Essentially, the front hemisphere represents the main user's game view, while the back hemisphere represents the companion's game view.

[0059] In some embodiments, displaying the corresponding image of the extended reality spherical model on the user's head-mounted display includes:

[0060] Obtain the sensor data from the head-mounted device uploaded by the main user;

[0061] Based on the sensor data, determine the view image corresponding to the current perspective of the main user in the extended reality spherical model;

[0062] The view image is displayed on the head-mounted display of the main user.

[0063] In some embodiments, it also includes:

[0064] In response to a local read operation, acquire panoramic video images archived locally on the extended reality device;

[0065] The at least one panoramic video image is segmented according to a preset field of view allocation rule to obtain at least one view image;

[0066] The at least one view image is stitched together to generate an extended reality spherical model;

[0067] The corresponding image of the extended reality spherical model is displayed on the head-mounted display of the extended reality device.

[0068] In some embodiments, the extended reality display method provided in this disclosure can be a multi-player online cloud game scene splicing display, or a single-player game scene local multiple save backup synchronous playback.

[0069] In some embodiments, local save data can be used in single-device or multi-user scenarios. Due to limitations on the number of devices, multiple users can only take turns using one device. Under these conditions, if a game's "companionship" function is desired, it can be simulated by reading game save data. It is understandable that by retrieving save data, users can compare their own performance in the same game scenario, which is suitable for scenarios such as game training.

[0070] In some embodiments, such as Figure 5 As shown, the extended reality display method provided in this disclosure includes:

[0071] Step a: Obtain touch information from the headset and controllers;

[0072] In some embodiments, the game engine needs to render game visuals based on sensor data from the headset and controller uploaded by the user.

[0073] Step b: The VR game engine outputs a rendered image corresponding to the game model coordinates based on the input 6DoF data;

[0074] In some embodiments, the output here is a panoramic image, which is rendered onto a VR spherical model.

[0075] Step c: The panoramic image output by the game engine is rendered into the spherical model;

[0076] In some embodiments, the panoramic image output by the game is rendered into a spherical model of VR. Then, the position of the "viewport" is determined based on the user's head-mounted sensor data. The image corresponding to the "viewport" position is then determined as the current frame and sent back to the head-mounted device for decoding and display.

[0077] Step d: Divide the spherical model according to the preset field of view of the current user;

[0078] In some embodiments, when segmenting the spherical model, the panoramic image can actually be segmented directly. To ensure the current user's immersive experience, the current main user's field of view must remain within a 120° horizontal range, regardless of how the field of view of other users is segmented. Taking a multiplayer (N-player) game scenario as an example, if the current main user's field of view is 120°, then the field of view of the remaining (N-1) users is allocated from (360° to 120°). The allocation principle can be equal division or division according to a custom rule.

[0079] Step e1: Acquire panoramic images of other users and segment the images from their panoramic images according to the set field of view;

[0080] In some embodiments, such as Figure 3 As shown in the embodiments of this disclosure, multi-player online cloud gaming scene splicing display can be performed.

[0081] Step e2: Based on the locally backed-up panoramic video images, segment the sub-images to be stitched according to the set field of view;

[0082] In some embodiments, the current primary user's images are stitched together with their previously locally archived backups 2 and 3 after being cropped from the field of view. It should be noted that the archived backup data must be panoramic video data.

[0083] In some embodiments, such as Figure 4 As shown in the embodiments of this disclosure, multiple local save backups of a single-player game scene can be played synchronously.

[0084] Step f: Reassemble the acquired segmented images into a new panoramic image;

[0085] In some embodiments, multiple acquired segmented images are re-stitched into a new panoramic image, for example, the field-of-view segmented images acquired from user 1, user 2 (or backup 2) and user 3 (or backup 3) are stitched together into a new panoramic image.

[0086] Step g: Render the stitched panoramic image onto a new spherical model;

[0087] In some embodiments, after acquiring a new panoramic image, it needs to be rendered onto a new VR spherical model, thus completing the construction of the new spherical model.

[0088] Step h: Determine the current true "viewport" position on the new spherical model based on the head-mounted sensor data uploaded by the user;

[0089] Step i: The viewport image is sent back to the user's head-mounted display for display.

[0090] In some embodiments, the solutions provided by related technologies, whether cloud gaming or local gaming, can only display the user's own game scene on the VR headset. However, the extended reality display method provided in this disclosure, through the segmentation of different "field of view" angles of the panoramic image, re-stitching the panoramic view, and reconstruction of the VR spherical model, enables multiple "game companions" to watch their games simultaneously within a single user scenario. This embodiment is similar to 2D projection playback of desktop screen recording, representing a true switching between different user game scenes in virtual reality. This embodiment does not require logical modifications to existing VR game single-player modes, and the segmentation of the "field of view" for participating users other than the main user can be either equal or a custom allocation ratio can be set. The locally backed-up panoramic game video in this embodiment can be added as a special user to the "game companions" for purposes such as player self-comparison.

[0091] In some embodiments, when a user activates the "companion" mode, the following can be achieved: Figure 6 The scene effect shown. (As shown) Figure 6 As shown, when User 1 turns their head, for example to the right, they can see User 2's game scene (playing a virtual reality version of CS); when User 1 turns their head to the left, they can see User 3 watching a VR live basketball game.

[0092] In some embodiments, this disclosure also provides an extended reality display device, including:

[0093] The acquisition module is used to acquire the view image corresponding to each user in response to the trigger operation of multi-user screen sharing;

[0094] The processing module is used to generate an extended reality spherical model based on the view images corresponding to each user. The extended reality spherical model is composed of the view images corresponding to each user.

[0095] The display module is used to display the corresponding image of the extended reality spherical model on the user's head-mounted device.

[0096] In some embodiments, the acquisition module is specifically used for:

[0097] In response to a multi-user shared screen trigger, acquire panoramic images of each user in their extended reality scene;

[0098] The panoramic images of each user are segmented to obtain the view images corresponding to each user.

[0099] In some embodiments, the acquisition module is specifically used for:

[0100] The extended reality spherical model corresponding to each user is segmented to obtain a panoramic image of each user in their extended reality scene.

[0101] In some embodiments, the acquisition module is further specifically used for:

[0102] Triggering operations in response to a shared screen request initiated by the main user, and triggering operations in response to a user accepting the shared screen request.

[0103] In some embodiments, the view image is the user's current perspective view in the extended reality scene.

[0104] In some embodiments, the first processing module is specifically used for:

[0105] The panoramic images of each user are segmented according to a preset field of view allocation rule to obtain the view images corresponding to each user.

[0106] In some embodiments, the first processing module is further specifically used for:

[0107] Based on the preset field of view of the master user and the number of slave users, the panoramic images of the master user and the slave users are segmented respectively.

[0108] In some embodiments, the display module is specifically used for:

[0109] Obtain the sensor data from the head-mounted device uploaded by the main user;

[0110] Based on the sensor data, determine the view image corresponding to the current perspective of the main user in the extended reality spherical model;

[0111] The view image is displayed on the head-mounted display of the main user.

[0112] In some embodiments, the acquisition module is further specifically used for:

[0113] In response to a trigger operation that reads a local file, acquire panoramic video images archived locally on the extended reality device;

[0114] Accordingly, the first processing module is also specifically used for:

[0115] The at least one panoramic video image is segmented according to a preset field of view allocation rule to obtain at least one view image;

[0116] Accordingly, the second processing module is also specifically used for:

[0117] The at least one view image is stitched together to generate an extended reality spherical model;

[0118] Accordingly, the display module is also specifically used for:

[0119] The corresponding image of the extended reality spherical model is displayed on the head-mounted display of the extended reality device.

[0120] For embodiments of the apparatus, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments. The apparatus embodiments described above are merely illustrative, and the modules described as separate modules may or may not be separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] The methods and apparatus of this disclosure have been described above based on embodiments and application examples. Furthermore, this disclosure also provides an electronic device and a computer-readable storage medium, which are described below.

[0122] The following is for reference. Figure 7 The figure illustrates a structural schematic of an electronic device (e.g., a terminal device or server) 800 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in the figure is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.

[0123] Electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from storage device 808 into random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0124] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although an electronic device 800 with various devices is shown in the figure, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0125] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0126] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0127] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0128] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0129] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods of the present disclosure.

[0130] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0132] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0133] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] According to one or more embodiments of this disclosure, an extended reality display method is provided, comprising:

[0136] In response to a multi-user screen sharing trigger, obtain the view image corresponding to each user;

[0137] An extended reality spherical model is generated based on the view images corresponding to each user. The extended reality spherical model is composed of the view images corresponding to each user.

[0138] The corresponding screen of the extended reality spherical model is displayed on the user's head-mounted display.

[0139] According to one or more embodiments of this disclosure, a method is provided in which, in response to a triggering operation of multi-user shared screen, view images corresponding to each user are acquired, including:

[0140] In response to a multi-user shared screen trigger, acquire panoramic images of each user in their extended reality scene;

[0141] The panoramic images of each user are segmented to obtain the view images corresponding to each user.

[0142] According to one or more embodiments of this disclosure, a method is provided for acquiring panoramic images of each user in their extended reality scene, including:

[0143] The extended reality spherical model corresponding to each user is segmented to obtain a panoramic image of each user in their extended reality scene.

[0144] According to one or more embodiments of this disclosure, a method is provided in which the triggering operation in response to multi-user screen sharing includes:

[0145] Triggering operations in response to a shared screen request initiated by the main user, and triggering operations in response to a user accepting the shared screen request.

[0146] According to one or more embodiments of this disclosure, a method is provided in which the view image is a user's current perspective view in an extended reality scene.

[0147] According to one or more embodiments of this disclosure, a method is provided in which panoramic images of each user are segmented to obtain view images corresponding to each user, comprising:

[0148] The panoramic images of each user are segmented according to a preset field of view allocation rule to obtain the view images corresponding to each user.

[0149] According to one or more embodiments of this disclosure, a method is provided in which the panoramic image of each user is segmented according to a preset field-of-view allocation rule, comprising:

[0150] Based on the preset field of view of the master user and the number of slave users, the panoramic images of the master user and the slave users are segmented respectively.

[0151] According to one or more embodiments of this disclosure, a method is provided for displaying a screen corresponding to the extended reality spherical model on a user's head-mounted display, comprising:

[0152] Obtain the sensor data from the head-mounted device uploaded by the main user;

[0153] Based on the sensor data, determine the view image corresponding to the current perspective of the main user in the extended reality spherical model;

[0154] The view image is displayed on the head-mounted display of the main user.

[0155] According to one or more embodiments of this disclosure, a method is provided, further comprising:

[0156] In response to a trigger operation that reads a local file, acquire panoramic video images archived locally on the extended reality device;

[0157] The at least one panoramic video image is segmented according to a preset field of view allocation rule to obtain at least one view image;

[0158] The at least one view image is stitched together to generate an extended reality spherical model;

[0159] The corresponding image of the extended reality spherical model is displayed on the head-mounted display of the extended reality device.

[0160] According to one or more embodiments of the present disclosure, an extended reality display device is provided, comprising:

[0161] The acquisition module is used to acquire the view image corresponding to each user in response to the trigger operation of multi-user screen sharing;

[0162] The processing module is used to generate an extended reality spherical model based on the view images corresponding to each user. The extended reality spherical model is composed of the view images corresponding to each user.

[0163] The display module is used to display the corresponding image of the extended reality spherical model on the user's head-mounted device.

[0164] According to one or more embodiments of the present disclosure, an electronic device is provided, including: at least one memory and at least one processor;

[0165] The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method described in any one of the above.

[0166] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided for storing program code that, when executed by a processor, causes the processor to perform the methods described above.

[0167] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0168] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0169] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An extended reality display method, characterized in that, include: In response to a multi-user shared screen trigger, acquire panoramic images of each user in their extended reality scene; The panoramic images of each user are segmented to obtain the view images corresponding to each user. An extended reality spherical model is generated based on the view images corresponding to each user. The extended reality spherical model is composed of the view images corresponding to each user. The corresponding screen of the extended reality spherical model is displayed on the user's head-mounted display.

2. The method according to claim 1, characterized in that, The acquisition of panoramic images of each user in their extended reality scene includes: The extended reality spherical model corresponding to each user is segmented to obtain a panoramic image of each user in their extended reality scene.

3. The method according to claim 1, characterized in that, The trigger operation in response to multi-user screen sharing includes: Triggering operations in response to a shared screen request initiated by the main user, and triggering operations in response to a user accepting the shared screen request.

4. The method according to claim 1, characterized in that, The view image is the user's current perspective in the extended reality scene.

5. The method according to claim 1, characterized in that, The step of segmenting the panoramic images of each user to obtain the view images corresponding to each user includes: The panoramic images of each user are segmented according to a preset field of view allocation rule to obtain the view images corresponding to each user.

6. The method according to claim 5, characterized in that, The step of segmenting the panoramic images of each user according to a preset field-of-view allocation rule includes: Based on the preset field of view of the master user and the number of slave users, the panoramic images of the master user and the slave users are segmented respectively.

7. The method according to claim 3, characterized in that, The display of the extended reality spherical model on the user's head-mounted display includes: Obtain the sensor data from the head-mounted device uploaded by the main user; Based on the sensor data, determine the view image corresponding to the current perspective of the main user in the extended reality spherical model; The view image is displayed on the head-mounted display of the main user.

8. The method according to claim 1, characterized in that, Also includes: In response to a trigger operation that reads a local file, acquire panoramic video images archived locally on the extended reality device; At least one panoramic video image is cut according to a preset field of view allocation rule to obtain at least one view image; The at least one view image is stitched together to generate an extended reality spherical model; The corresponding image of the extended reality spherical model is displayed on the head-mounted display of the extended reality device.

9. An extended reality display device, characterized in that, include: The acquisition module is used to respond to the trigger operation of multi-user shared screen, acquire panoramic images of each user in their extended reality scene, and cut the panoramic images of each user to obtain the view images corresponding to each user. The processing module is used to generate an extended reality spherical model based on the view images corresponding to each user. The extended reality spherical model is composed of the view images corresponding to each user. The display module is used to display the corresponding image of the extended reality spherical model on the user's head-mounted device.

10. An electronic device, comprising: At least one memory and at least one processor; The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method of any one of claims 1 to 8.

11. A computer-readable storage medium for storing program code that, when executed by a computer device, causes the computer device to perform the method of any one of claims 1 to 8.

Images