Visual field sharing method and device based on XR, equipment and storage medium

Abstract

The invention discloses a visual field sharing method and device based on XR, equipment and a storage medium, and relates to the field of XR. The method comprises the following steps: sending a shared video stream to second XR glasses, wherein the shared video stream is a video stream shared between first XR glasses and the second XR glasses; receiving mark information sent by the second XR glasses, wherein the mark information is associated with a target element in the shared video stream; and displaying the mark information associated with the target element in the shared video stream. According to the method, bidirectional view sharing in an XR scene can be realized.

Description

Technical Field

[0001] The present application relates to the field of extended reality (XR), and in particular to a field of view sharing method, apparatus, device and storage medium based on XR. Background Art

[0002] The field of view sharing based on XR glasses in the related technology is one-way, which only supports sharing the field of view of user A with user B, but does not support transmitting the operations of user B on the images in the field of view back to user A, resulting in the inability of the two parties to form efficient interaction.

[0003] How to use XR technology to enable two-way vision sharing between users is a problem that needs to be solved. Summary of the invention

[0004] The embodiment of the present application provides a method, apparatus, device and storage medium for sharing vision based on XR, and the technical solution is as follows:

[0005] According to one aspect of the present application, a method for sharing a field of view based on XR is provided, the method being performed by a first pair of XR glasses, the method comprising:

[0006] Sending a shared video stream to a second XR glasses, where the shared video stream is a video stream shared between the first XR glasses and the second XR glasses;

[0007] receiving marking information sent by the second XR glasses, where the marking information is associated with a target element in the shared video stream;

[0008] The marking information associated with the target element is displayed in the shared video stream.

[0009] According to one aspect of the present application, a method for sharing a field of view based on XR is provided, the method being performed by a second pair of XR glasses, the method comprising:

[0010] receiving a shared video stream sent by the first XR glasses, where the shared video stream is a video stream shared between the first XR glasses and the second XR glasses;

[0011] Sending marking information to the first XR glasses, wherein the marking information is associated with a target element in the shared video stream;

[0012] The marking information associated with the target element is displayed in the shared video stream.

[0013] According to one aspect of the present application, a field of view sharing device based on XR is provided, the device comprising:

[0014] A first sending module, configured to send a shared video stream to a second pair of XR glasses, where the shared video stream is a video stream shared between the device and the second pair of XR glasses;

[0015] A first receiving module, configured to receive marking information sent by the second XR glasses, where the marking information is associated with a target element in the shared video stream;

[0016] The first processing module is used to display the marking information associated with the target element in the shared video stream.

[0017] According to one aspect of the present application, a field of view sharing device based on XR is provided, the device comprising:

[0018] A second receiving module, configured to receive a shared video stream sent by the first XR glasses, where the shared video stream is a video stream shared between the first XR glasses and the device;

[0019] A second sending module, configured to send marking information to the first XR glasses, where the marking information is associated with a target element in the shared video stream;

[0020] The second processing module is used to display the marking information associated with the target element in the shared video stream.

[0021] According to one aspect of the present application, an electronic device is provided, comprising: a processor; a transceiver connected to the processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to load and execute the executable instructions to implement the XR-based field of view sharing method as described in the above aspects.

[0022] According to one aspect of the present application, a computer-readable storage medium is provided, in which executable instructions are stored. The executable instructions are loaded and executed by a processor to implement the XR-based field of view sharing method as described in the above aspects.

[0023] According to one aspect of the present application, a computer program product is provided, which includes computer instructions, the computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the XR-based field of view sharing method as described in the above aspects.

[0024] According to one aspect of the present application, a chip is provided, which includes a programmable logic circuit or a program, and the chip is used to implement the XR-based field of view sharing method as described in the above aspect.

[0025] According to one aspect of the present application, a computer program is provided, which includes computer instructions. A processor of a computer device executes the computer instructions, so that the computer device executes the XR-based field of view sharing method as described in the above aspects.

[0026] The technical solution provided by the embodiments of the present application includes at least the following beneficial effects:

[0027] By transmitting the shared video stream and marker information between the first XR glasses and the second XR glasses, efficient interaction is formed between the first XR glasses and the second XR glasses, and two-way field of view sharing is achieved. The design of associating the marker information with the target element is conducive to improving the accuracy of displaying the marker information and the precision of field of view sharing, further improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A structural block diagram of a computer system provided by an exemplary embodiment of the present application is shown;

[0030] Figure 2 A schematic diagram of a flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown;

[0031] Figure 3 A schematic diagram of a flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown;

[0032] Figure 4 A schematic diagram of a flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown;

[0033] Figure 5 A schematic diagram of a flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown;

[0034] Figure 6 A schematic diagram of a flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown;

[0035] Figure 7 A schematic diagram of a shared video stream provided by an exemplary embodiment of the present application is shown;

[0036] Figure 8A schematic diagram of a flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown;

[0037] Fig. 9 A schematic diagram of a flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown;

[0038] Fig.10 A schematic diagram of a flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown;

[0039] Fig.11 A schematic diagram of an XR-based field of view sharing method provided by an exemplary embodiment of the present application is shown;

[0040] Fig.12 A structural block diagram of an XR-based field of view sharing device provided by an exemplary embodiment of the present application is shown;

[0041] Fig.13 A structural block diagram of an XR-based field of view sharing device provided by an exemplary embodiment of the present application is shown;

[0042] Fig.14 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings. The exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation modes described in the following exemplary embodiments do not represent all implementation modes consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0044] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0045] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0046] XR Glasses:

[0047] XR glasses refer to glasses that support XR technology, which includes at least one of the technologies based on the fusion of virtual and real, such as augmented reality (AR), mixed reality (MR), virtual reality (VR), baffle reality (BR), cinematic reality (CR), and deceived reality (DR).

[0048] Vision sharing based on XR glasses:

[0049] In the related technologies, the field of view sharing technology based on XR glasses can be mainly divided into the following two categories.

[0050] Category 1: XR glasses-based field of view sharing for teaching. XR glasses A worn by user A receive the information to be displayed. XR glasses A, as the host, searches the local database and projects the information to be displayed into the field of view corresponding to XR glasses A. At the same time, XR glasses A merges the image in the field of view with the information to be displayed and uploads it to the main server. XR glasses B worn by user B are connected to the main server, thereby realizing the field of view sharing between user A and user B.

[0051] Category 2: Ordinary field of view sharing. User A wears XR glasses A, and the camera of XR glasses A collects images. XR glasses A transmit the collected images to the smart device used by user B (such as a smartphone, XR glasses, etc.) through the network. User B observes the image in the field of view of XR glasses A through the smart device, thereby realizing the field of view sharing between user A and user B.

[0052] However, the field of view sharing based on XR glasses in the above-mentioned related technologies is one-way, that is, it only supports sharing the field of view of user A with user B, but does not support transmitting user B's operations on the image in the field of view back to user A. The two parties cannot form an efficient interaction, nor can they achieve two-way field of view sharing.

[0053] Figure 1 The computer system 100 includes a first terminal 110 and a second terminal 120 , and may or may not include a server 130 .

[0054] The first terminal 110 has a camera and has an application that supports XR-based field of view sharing installed and running. The first terminal 110 is an XR terminal used by a first user.

[0055] The second terminal 120 has a camera and has an application that supports XR-based field of view sharing installed and running. The second terminal 120 is an XR terminal used by a second user.

[0056] Optionally, the first user and the second user are in the same scene. Optionally, the first user and the second user may belong to the same team, or the same organization, or have a friend relationship or have temporary communication permissions.

[0057] Optionally, the applications installed on the first terminal 110 and the second terminal 120 are the same, or the applications installed on the two terminals are the same type of applications on different control system platforms. The first terminal 110 may generally refer to one of a plurality of terminals, and the second terminal 120 may generally refer to one of a plurality of terminals. The embodiments of the present application only take the first terminal 110 and the second terminal 120 as examples. The device types of the first terminal 110 and the second terminal 120 are the same or different, and the device types include at least one of: XR glasses, smart phones, tablet computers, e-book readers, laptop computers, desktop computers, and XR watches. The following embodiments are described by taking the first terminal and the second terminal as XR glasses as an example, but do not mean to limit the device types of the first terminal and the second terminal.

[0058] In some embodiments, the first terminal 110 is connected to the second terminal 120 via a wireless network or a wired network.

[0059] In some embodiments, the first terminal 110 is connected to the server 130 via a wireless network or a wired network.

[0060] In some embodiments, the second terminal 120 is connected to the server 130 via a wireless network or a wired network.

[0061] In some embodiments, the server 130 includes one server, multiple servers, a cloud computing platform, and a virtualization center. Schematically, the server 130 includes a processor 131 and a memory 132, and the memory 132 includes a receiving module 1321, a display module 1322, and a control module 1323. The server 130 is used to provide background services for applications that support XR-based field of view sharing. Optionally, the server 130 undertakes the main computing work, and the first terminal 110 and the second terminal 120 undertake the secondary computing work; or, the server 130 undertakes the secondary computing work, and the first terminal 110 and the second terminal 120 undertake the main computing work; or, the server 130, the first terminal 110, and the second terminal 120 adopt a distributed computing architecture for collaborative computing; or, the first terminal 110 undertakes the main computing work, and the second terminal 120 undertakes the secondary computing work; or, the first terminal 110 undertakes all computing work; or, the first terminal 110 undertakes the secondary computing work, and the second terminal 120 undertakes the main computing work.

[0062] Those skilled in the art will appreciate that the number of the above terminals or users may be more or less. For example, the above terminal or user may be only one, or the above terminals or users may be dozens or hundreds, or more. The embodiments of the present application do not limit the number of terminals or users and the device types.

[0063] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the shared video streams involved in this application are obtained with full authorization, and the terminal (such as XR glasses) and the server only cache the shared video stream during program execution, and will not solidify the storage and reuse the relevant data of the shared video stream.

[0064] Figure 2 A flow chart of an XR-based field of view sharing method provided by an exemplary embodiment of the present application is shown. Taking the method performed by the first XR glasses as an example, the method includes at least some of the following steps:

[0065] Step 210: Sending a shared video stream to a second pair of XR glasses;

[0066] The shared video stream is a video stream shared between the first XR glasses and the second XR glasses.

[0067] The first XR glasses belong to Figure 1The first terminal shown. An application supporting XR-based vision sharing is installed and run in the first XR glasses. The first XR glasses are equipped with a camera. The first XR glasses include one or more XR glasses.

[0068] The second XR glasses belong to Figure 1 The second terminal shown. An application supporting XR-based vision sharing is installed and run in the second XR glasses. The second XR glasses are equipped with a camera. The second XR glasses include one or more XR glasses.

[0069] In some embodiments, the first XR glasses share the shared video stream with the second XR glasses via a wireless network, or the first XR glasses share the shared video stream with the second XR glasses via a wired network. That is, the first XR glasses send the shared video stream to the second XR glasses via a wireless network, or the first XR glasses send the shared video stream to the second XR glasses via a wired network.

[0070] In some embodiments, the shared video stream includes a plurality of video frames.

[0071] Step 230: receiving marking information sent by the second XR glasses;

[0072] The marking information is associated with the target element in the shared video stream. Alternatively, it can be understood that the marking information is marking information for the target element in the shared video stream.

[0073] The marking information is sent by the second XR glasses, or generated and sent by the second XR glasses.

[0074] Step 250: Displaying the tag information associated with the target element in the shared video stream.

[0075] After receiving the marking information from the second XR glasses, the first XR glasses display the marking information in the shared video stream. In other words, both the first XR glasses and the second XR glasses can display the marking information through the shared video stream.

[0076] In summary, the method provided in the embodiment of the present application forms efficient interaction between the first XR glasses and the second XR glasses by transmitting a shared video stream and marking information between the first XR glasses and the second XR glasses, thereby realizing two-way field of view sharing. The design of associating the marking information with the target element is conducive to improving the accuracy of displaying the marking information and the precision of field of view sharing, further improving the user experience.

[0077] Figure 3 A flow chart of an XR-based field of view sharing method provided by an exemplary embodiment of the present application is shown. Taking the method performed by the first XR glasses as an example, the method includes at least some of the following steps:

[0078] Step 310: Sending a shared video stream to a second pair of XR glasses;

[0079] For related content, please refer to step 210, which will not be repeated here.

[0080] Step 320: locally cache the most recently displayed N frames of video corresponding to the shared video stream;

[0081] Wherein, N is a positive integer, and the displayed video frame refers to the video frame in the shared video stream that has been displayed in the field of view of the first XR glasses.

[0082] In some embodiments, the first XR glasses determine the number of frames N to be locally cached based on the network delay value.

[0083] In some embodiments, the size of the frame number N is positively correlated with the network delay value, or the size of the frame number N is correlated with the numerical range to which the network delay value belongs.

[0084] Step 330: Receive the marking information and the frame number corresponding to the marking information sent by the second XR glasses;

[0085] The marking information is associated with the target element in the shared video stream. Alternatively, it can be understood that the marking information is marking information for the target element in the shared video stream.

[0086] The marking information is sent by the second XR glasses, or generated and sent by the second XR glasses.

[0087] The shared video stream includes multiple video frames. The frame number corresponding to the marking information is, that is, the frame number of the video frame associated with the marking moment of the marking information. The video frame associated with the marking moment of the marking information may be understood as the video frame corresponding to the marking information, or the video frame where the marking information is located, or the video frame associated with the marking information, or the video frame displayed by the second XR glasses when the marking information is generated.

[0088] Step 340: Determine, in the first video frame in the shared video stream, a target element associated with the marking information;

[0089] In some embodiments, the first XR glasses determine a first video frame in the shared video stream corresponding to a frame number corresponding to the marking information, the first video frame being a video frame associated with a marking moment of the marking information.

[0090] In some embodiments, the first XR glasses determine a target element associated with the marker information in the first video frame.

[0091] The target element is an element marked by the marking information. The type of the target element includes at least one of the following: coordinates, moving targets, pixel blocks, display elements. The display elements include at least one of the following: text, pictures, symbols, controls, and color blocks.

[0092] In some embodiments, the type of the target element includes coordinates, and the target coordinates are also coordinates marked by the marking information. The first XR glasses determine the first coordinates associated with the marking information in the first video frame based on the coordinate information of the first video frame, and the first coordinates are the target element in the first video frame.

[0093] In some embodiments, the type of target element includes a moving target, and the moving target is also a moving target marked by the marking information. The first XR glasses determine the moving target associated with the marking information in the first video frame.

[0094] In some embodiments, the type of the target element includes a pixel block. The first XR glasses divide the first video frame into M1 pixel blocks, where M1 is a positive integer; a first pixel block associated with the marker information is determined from the M1 pixel blocks, and the first pixel block is the target element in the first video frame.

[0095] In some embodiments, the type of the target element includes a display element, and the target element is also a display element marked with the marking information. The first XR glasses determine a first display element associated with the marking information in the first video frame based on the display elements in the first video frame.

[0096] Step 350: Tracking a target element in a second video frame in the shared video stream;

[0097] The second video frame is a video frame located after the first video frame in the shared video stream. That is, the first XR glasses display the second video frame after displaying the first video frame.

[0098] In some embodiments, the target element is tracked in the second video frame in the shared video stream, or understood as, the target element is determined in the second video frame in the shared video stream.

[0099] In some embodiments, the type of the target element includes coordinates, and the target coordinates are coordinates marked by the marking information. The first XR glasses determine a second coordinate corresponding to the first coordinate in the second video frame based on the coordinate information of the first video frame and the coordinate information of the second video frame, and the second coordinate is the target element in the second video frame;

[0100] The display element corresponding to the first coordinate in the first video frame is the same as the display element corresponding to the second coordinate in the second video frame.

[0101] In some embodiments, the type of target element includes a moving target, which is also a moving target marked by the marking information. The first XR glasses track the motion coordinates in the second video frame based on the motion tracking algorithm; and determine the target element in the second video frame based on the tracking result of the moving target.

[0102] In some embodiments, the type of the target element includes a pixel block. The first XR glasses divide the second video frame into M2 pixel blocks, where M2 is a positive integer; the first XR glasses determine a second pixel block matching the first pixel block in the second video frame based on a pixel block matching algorithm, and the second pixel block is the target element in the second video frame. M1 is the same as or different from M2.

[0103] In some embodiments, the type of the target element includes a display element, and the target element is also a display element marked with marking information. The first XR glasses determine a second display element corresponding to the first display element in the second video frame based on the display element in the first video frame and the display element in the second video frame, and the second display element is the target element in the second video frame.

[0104] Step 360: Displaying the marking information associated with the target element in the second video frame.

[0105] The marking information is displayed in the second video frame, and the marking information in the second video frame is associated with the target element in the second video frame, that is, the marking information in the second video frame corresponds to the target element in the second video frame.

[0106] It should be understood that the execution of the above steps can be adjusted according to actual conditions. For example, step 320 is executed before step 310. For example, step 310 and step 320 are combined into one step. For example, step 330 and step 340 are combined into one step. For example, step 340 and step 350 are combined into one step. For example, step 330 is combined with step 340 and step 350 into one step, and so on.

[0107] In summary, the method provided in the embodiment of the present application, by transmitting a shared video stream and marking information between the first XR glasses and the second XR glasses, enables efficient interaction between the first XR glasses and the second XR glasses, realizes two-way field of view sharing, and is beneficial to improving the efficiency and convenience of communication between users. The design of associating the marking information with the frame number and the target element is beneficial to improving the accuracy of the displayed marking information and the accuracy of the field of view sharing, further improving the user experience. Moreover, since the target element supports multiple types, the flexibility of displaying the marking information is effectively improved, making the method provided in the embodiment of the present application applicable to a variety of field of view sharing scenarios. In addition, compared with the interaction between traditional mobile phones and computers, the interaction between the two XR glasses can bring users a more immersive and rich interactive experience and communication experience.

[0108] Figure 4 A flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown. Taking the method performed by the second XR glasses as an example, the method includes at least some of the following steps:

[0109] Step 410: Receive a shared video stream sent by the first XR glasses;

[0110] The shared video stream is a video stream shared between the first XR glasses and the second XR glasses.

[0111] The first XR glasses belong to Figure 1 The first terminal shown. An application supporting XR-based vision sharing is installed and run in the first XR glasses. The first XR glasses are equipped with a camera. The first XR glasses include one or more XR glasses.

[0112] The second XR glasses belong to Figure 1 The second terminal shown. An application supporting XR-based vision sharing is installed and run in the second XR glasses. The second XR glasses are equipped with a camera. The second XR glasses include one or more XR glasses.

[0113] In some embodiments, the first XR glasses share the shared video stream with the second XR glasses via a wireless network, or the first XR glasses share the shared video stream with the second XR glasses via a wired network. That is, the second XR glasses receive the shared video stream sent by the first XR glasses via a wireless network, or the second XR glasses receive the shared video stream sent by the first XR glasses via a wired network.

[0114] In some embodiments, the shared video stream includes a plurality of video frames.

[0115] Step 430: Sending tag information to the first XR glasses;

[0116] The marking information is associated with the target element in the shared video stream. Alternatively, it can be understood that the marking information is marking information for the target element in the shared video stream.

[0117] The marking information is sent by the second XR glasses, or generated and sent by the second XR glasses.

[0118] Step 450: Displaying the tag information associated with the target element in the shared video stream.

[0119] Both the first XR glasses and the second XR glasses can display the marker information through the shared video stream.

[0120] It should be understood that the execution of the above steps can be adjusted according to actual conditions, for example, step 450 is performed before step 430, or step 450 and step 430 are performed simultaneously.

[0121] In summary, the method provided in the embodiment of the present application forms efficient interaction between the first XR glasses and the second XR glasses by transmitting a shared video stream and marking information between the first XR glasses and the second XR glasses, thereby realizing two-way field of view sharing. The design of associating the marking information with the target element is conducive to improving the accuracy of displaying the marking information and the precision of field of view sharing, further improving the user experience.

[0122] Figure 5 A flow chart of a method for sharing vision based on XR provided by an exemplary embodiment of the present application is shown. Taking the method performed by the second XR glasses as an example, the method includes at least some of the following steps:

[0123] Step 510: receiving a shared video stream sent by the first XR glasses;

[0124] For related content, please refer to step 410, which will not be repeated here.

[0125] Step 520: Sending the marking information and the frame number corresponding to the marking information to the first XR glasses;

[0126] The marking information is associated with the target element in the shared video stream. Alternatively, it can be understood that the marking information is marking information for the target element in the shared video stream.

[0127] The marking information is sent by the second XR glasses, or generated and sent by the second XR glasses.

[0128] The shared video stream includes multiple video frames. The frame number corresponding to the marking information is, that is, the frame number of the video frame associated with the marking moment of the marking information. The video frame associated with the marking moment of the marking information may be understood as the video frame corresponding to the marking information, or the video frame where the marking information is located, or the video frame associated with the marking information, or the video frame displayed by the second XR glasses when the marking information is generated.

[0129] The target element is an element marked by the marking information. The type of the target element includes at least one of the following: coordinates, moving targets, pixel blocks, display elements. The display elements include at least one of the following: text, pictures, symbols, controls, and color blocks.

[0130] Step 530: Displaying the tag information associated with the target element in the shared video stream.

[0131] The second XR glasses display the marking information associated with the target element in a third video frame in the shared video stream.

[0132] In some embodiments, the third video frame includes one video frame or multiple video frames.

[0133] In some embodiments, the third video frame includes a video frame associated with a marking moment of the marking information; or, the third video frame includes a video frame associated with a generation moment of the marking information.

[0134] It should be understood that the execution of the above steps can be adjusted according to actual conditions, for example, step 530 is performed before step 520, or step 530 and step 520 are performed simultaneously.

[0135] In summary, the method provided in the embodiment of the present application, by transmitting a shared video stream and marking information between the first XR glasses and the second XR glasses, enables efficient interaction between the first XR glasses and the second XR glasses, realizes two-way field of view sharing, and is beneficial to improving the efficiency and convenience of communication between users. The design of associating the marking information with the frame number and the target element is beneficial to improving the accuracy of the displayed marking information and the accuracy of the field of view sharing, further improving the user experience. Moreover, since the target element supports multiple types, the flexibility of displaying the marking information is effectively improved, making the method provided in the embodiment of the present application applicable to a variety of field of view sharing scenarios. In addition, compared with the interaction between traditional mobile phones and computers, the interaction between the two XR glasses can bring users a more immersive and rich interactive experience and communication experience.

[0136] Figure 6 A flow chart of an XR-based field of view sharing method provided by an exemplary embodiment of the present application is shown. Taking the method performed by the first XR glasses and the second XR glasses as an example, the method includes at least some of the following steps:

[0137] Step 601: The first XR glasses generate and send a shared video stream;

[0138] In some embodiments, the first XR glasses support Video See-Through (VST) capability.

[0139] The first XR glasses generate a shared video stream with a first frame rate. Alternatively, the first XR glasses generate a VST graph with a first frame rate.

[0140] In some embodiments, the value of the first frame rate is determined based on the performance of the first XR glasses. For example, when the performance of the first XR glasses is better, the first frame rate is 60 frames; when the performance of the first XR glasses is worse, the first frame rate is 30 frames.

[0141] The first XR glasses determine the frame number of each video frame in the shared video stream. Or it can be understood that each video frame in the shared video stream has a one-to-one corresponding frame number.

[0142] The first XR glasses send the shared video stream corresponding to the frame number to the second XR glasses, that is, the first XR glasses send the VST image of the first frame rate corresponding to the frame number to the second XR glasses.

[0143] The first XR glasses locally cache the most recent N video frames corresponding to the shared video stream. That is, the first XR glasses locally scroll and cache the most recently generated N frames of VST images.

[0144] Step 602: The second XR glasses display marking information associated with the target element in the shared video stream;

[0145] The second XR glasses display the marking information in the shared video stream, and the marking information is associated with the target element in the shared video stream.

[0146] Step 603: The second XR glasses send marking information;

[0147] In some embodiments, the marking information is generated by a control device associated with the second XR glasses, and the control device associated with the second XR glasses includes: at least one of a handle, a smart phone, a tablet computer, a smart watch, a smart bracelet, a desktop computer, a portable computer, etc. Exemplarily, the user marks the shared video stream by operating the handle, the handle obtains the user operation information, generates the marking information based on the user operation information, and sends the marking information to the second XR glasses. Exemplarily, the user marks the shared video stream on the smart phone, the smart phone generates the marking information based on the user operation information, and sends the marking information to the second XR glasses.

[0148] In some embodiments, the marking information is generated by the second XR glasses. Exemplarily, the user marks the shared video stream by hand movement, and the second XR glasses recognize the user's hand movement information and generate the marking information based on the hand movement information. Exemplarily, the second XR glasses receive user operation information sent by a control device associated with the second XR glasses, and the second XR glasses generate the marking information based on the user operation information.

[0149] The second XR glasses send the marking information and the frame number corresponding to the marking information to the first XR glasses. The frame number corresponding to the marking information is, that is, the frame number of the video frame associated with the marking moment of the marking information. The video frame associated with the marking moment of the marking information may be understood as the video frame corresponding to the marking information, or the video frame where the marking information is located, or the video frame associated with the marking information, or the video frame displayed by the second XR glasses when the marking information is generated.

[0150] In some embodiments, the second XR glasses package the marking information and the frame number corresponding to the marking information and send them to the first XR glasses.

[0151] In some embodiments, the second XR glasses send the marking information and the frame number corresponding to the marking information to the first XR glasses respectively. For example, the second XR glasses first send the marking information to the first XR glasses, and then send the frame number corresponding to the marking information to the first XR glasses. For another example, the second XR glasses first send the frame number corresponding to the marking information to the first XR glasses, and then send the marking information to the first XR glasses.

[0152] In some embodiments, the marking information includes at least one of operation information, auxiliary lines, colors, texts, pictures, symbols, emoticons, etc.

[0153] Step 604: The first XR glasses determine a target element associated with the marking information in the first video frame;

[0154] The first XR glasses determine a first video frame corresponding to a frame sequence number corresponding to the marking information in the shared video stream, where the first video frame is a video frame associated with a marking moment of the marking information.

[0155] For example, Figure 7 As shown, the most recent N video frames corresponding to the shared video stream cached locally by the first XR glasses include frames 1 to N, the frame number corresponding to the marking information is z, and the first XR glasses determine the first video frame based on the frame number z.

[0156] The first XR glasses determine a target element associated with the marker information in the first video frame.

[0157] The target element is an element marked by the marking information. The type of the target element includes at least one of the following: coordinates, moving targets, pixel blocks, display elements. The display elements include at least one of the following: text, pictures, symbols, controls, and color blocks.

[0158] Step 605: The first XR glasses track the target element in the second video frame;

[0159] The second video frame is a video frame located after the first video frame in the shared video stream. That is, the first XR glasses display the second video frame after displaying the first video frame.

[0160] In some embodiments, the target element is tracked in the second video frame in the shared video stream, or understood as, the target element is determined in the second video frame in the shared video stream.

[0161] Step 606: The first XR glasses display marking information associated with the target element in the second video frame.

[0162] The first XR glasses display the marking information in the second video frame, and the marking information in the second video frame is associated with the target element in the second video frame. That is, the marking information in the second video frame corresponds to the target element in the second video frame.

[0163] In some embodiments, the type of target element includes coordinates, such as Figure 8 As shown, step 604 can be implemented as step 6042 , step 605 can be implemented as step 6052 , and step 606 can be implemented as step 6062 .

[0164] Step 6042: The first XR glasses determine a first coordinate associated with the marker information in the first video frame;

[0165] Exemplarily, after receiving the marking information and the frame number, the first XR glasses determine the first video frame and restore the marking information in the first video frame. In addition, the first XR glasses determine the first coordinate in the first video frame as (20, 29), indicating that the associated coordinates of the marking information are (20, 29), or it is understood that the coordinates corresponding to the marking information are (20, 29), or it is understood that the coordinates of the marking information in the first video frame are (20, 29).

[0166] Step 6052: The first XR glasses determine a second coordinate in the second video frame;

[0167] In some embodiments, the first coordinate in the first video frame is the same as or different from the second coordinate in the second video frame.

[0168] In some embodiments, the coordinate information of the first video frame is the same as the coordinate information of the second video frame. Exemplarily, the second coordinate is also (20, 29). That is, the associated coordinates of the marking information in the first video frame are (20, 29), and the associated coordinates of the marking information in the second video frame are also (20, 29).

[0169] In some embodiments, the coordinate information of the first video frame is different from the coordinate information of the second video frame. For example, the second video frame is obtained after the first video frame is processed, and the processing method includes at least one of offset, enlargement, reduction, rotation, flipping, etc. Based on the coordinate information of the first video frame and the coordinate information of the second video frame, the second coordinate corresponding to the first coordinate is determined. Exemplarily, the second video frame is obtained by offsetting and enlarging the first video frame, and the first coordinate (20,29) is converted to obtain the second coordinate (30,29). Among them, the display element corresponding to the first coordinate in the first video frame is the same as the display element corresponding to the second coordinate in the second video frame. For example, the first coordinate corresponds to the UI control "Search" in the first video frame, and the second coordinate also corresponds to the UI control "Search" in the second video frame.

[0170] Step 6062: The first XR glasses display marking information associated with the second coordinates in the second video frame.

[0171] The marking information is associated with the second coordinate and displayed. Exemplarily, the marking information is displayed at the second coordinate in the second video frame.

[0172] Since fewer resources are required to determine the first coordinate and the second coordinate, and the determination method is simple and convenient, the method of displaying marking information through the first coordinate and the second coordinate has high simplicity, convenience, and resource utilization efficiency, and will not have a negative impact on network latency, thereby affecting the playback, transmission, and reception of shared video streams.

[0173] In some embodiments, the type of target element includes a moving target, such as Fig. 9 As shown, step 604 can be implemented as step 6044, step 605 can be implemented as step 6054, and step 606 can be implemented as step 6064.

[0174] Step 6044: The first XR glasses determine a moving target associated with the marker information in the first video frame;

[0175] Exemplarily, after receiving the marking information and the frame number, the first XR glasses determine the first video frame and restore the marking information in the first video frame. In addition, the first XR glasses determine the moving target associated with the marking information in the first video frame, that is, determine the moving target corresponding to the marking information, that is, determine the moving target marked by the marking information.

[0176] Step 6054: The first XR glasses determine a moving target in the second video frame;

[0177] The first XR glasses track the motion coordinates in the second video frame based on a motion tracking algorithm; and determine the target element in the second video frame based on the tracking result of the moving target.

[0178] The motion tracking algorithm includes at least one of an optical flow method, a Kalman filtering method, and an ego motion method.

[0179] Step 6064: The first XR glasses display marking information associated with the moving target in the second video frame.

[0180] The marking information is associated with the moving target in the second video frame and displayed. Exemplarily, the marking information is displayed on the moving target in the second video frame.

[0181] Since the motion tracking method has a high accuracy, the display of the marking information also has a high accuracy. In addition, if the motion tracking method uses motion prediction technology, the display of the marking information can have a higher flexibility and lower network latency. When the difference between the display element in the first video frame and the display element in the second video frame is large, the motion tracking method can be used to accurately and quickly display the marking information.

[0182] In some embodiments, the type of target element includes coordinates, such as Fig.10 As shown, step 604 can be implemented as step 6046, step 605 can be implemented as step 6056, and step 606 can be implemented as step 6066.

[0183] Step 6046: The first XR glasses determine, in the first video frame, a first pixel block associated with the marker information;

[0184] Exemplarily, after receiving the marking information and the frame number, the first XR glasses determine the first video frame. The first XR glasses divide the first video frame into M1 pixel blocks, where M1 is a positive integer. The first XR glasses restore the marking information in the first video frame. Furthermore, the first XR glasses determine the first pixel block associated with the marking information in the M1 pixel blocks, that is, determine the first pixel block corresponding to the marking information, that is, determine the first pixel block marked by the marking information.

[0185] Step 6056: The first XR glasses determine a second pixel block in the second video frame;

[0186] The first XR glasses divide the second video frame into M2 pixel blocks, where M2 is a positive integer; the first XR glasses determine a second pixel block matching the first pixel block in the second video frame based on a pixel block matching algorithm, that is, determine a second pixel block associated with the marker information in the second video frame, where M1 is the same as or different from M2.

[0187] Step 6066: The first XR glasses display marking information associated with the second pixel block in the second video frame.

[0188] The marking information is associated with the second pixel block and displayed. Exemplarily, the second pixel block in the second video frame displays the marking information.

[0189] Since the pixel matching method has a high accuracy, the display of the marking information also has a high accuracy. In addition, when the display elements in the first video frame and the display elements in the second video frame are blocked, deformed, or have grayscale changes, the pixel matching method can accurately display the marking information and avoid noise influence as much as possible.

[0190] It should be understood that Figure 8 , Fig. 9 , Fig.10 The illustrated embodiments can be used alone or in combination, so that the method provided by the embodiments of the present application has good robustness and flexibility. For example, the target element includes coordinates and a moving target, and the first XR glasses determine the first coordinates and the moving target in the first video frame, and determine the second coordinates and the moving target in the second video frame. For another example, the target element includes coordinates and a pixel block, and the first XR glasses determine the first coordinates and the first pixel block in the first video frame, and determine the second coordinates and the second pixel block in the second video frame. For another example, the target element includes a moving target and a pixel block, and the first XR glasses determine the moving target and the first pixel block in the first video frame, and determine the moving target and the second pixel block in the second video frame. For another example, the target element includes coordinates, a moving target, and a pixel block, and the first XR glasses determine the first coordinates, the moving target, and the first pixel block in the first video frame, and determine the second coordinates, the moving target, and the second pixel block in the second video frame. The design of displaying the marking information in the shared video stream based on multiple types of target elements at the same time can further improve the display accuracy of the marking information.

[0191] In summary, the method provided in the embodiment of the present application, by transmitting a shared video stream and marking information between the first XR glasses and the second XR glasses, enables efficient interaction between the first XR glasses and the second XR glasses, realizes two-way field of view sharing, and is beneficial to improving the efficiency and convenience of communication between users. The design of associating the marking information with the frame number and the target element is beneficial to improving the accuracy of the displayed marking information and the accuracy of the field of view sharing, further improving the user experience. Moreover, since the target element supports multiple types, the flexibility of displaying the marking information is effectively improved, making the method provided in the embodiment of the present application applicable to a variety of field of view sharing scenarios. In addition, compared with the interaction between traditional mobile phones and computers, the interaction between the two XR glasses can bring users a more immersive and rich interactive experience and communication experience.

[0192] Fig.11 A schematic diagram of an XR-based field of view sharing method provided by an exemplary embodiment of the present application is shown, taking the method being performed by a first XR pair of glasses and a second XR pair of glasses as an example.

[0193] The first user 1111 wears the first XR glasses 1112, and the second user 1121 wears the second XR glasses 1122. The first XR glasses 1112 and the second XR glasses 1122 interact with each other through the network device 113.

[0194] The first XR glasses 1112 send a shared video stream to the second XR glasses 1122. The first XR glasses 1112 display video frames 1141, and the second XR glasses 1122 display video frames 1142.

[0195] The second user 1121 marks the video frame 1142 and adds an auxiliary line 115 where the text "Classification" is displayed.

[0196] The second XR glasses 1122 send the marking information and the frame number corresponding to the marking information to the first XR glasses. For example, the marking information indicates that the coordinates (10, 25) on the video frame 1142 are marked with an auxiliary line 115, and the frame number is the frame number corresponding to the video frame 1142.

[0197] After receiving the marking information and the frame number corresponding to the marking information, the first XR glasses 1112 determine the video frame 1142 in the local cache, restore the marking information and display it in the video frame 1142.

[0198] The first XR glasses 1112 determine the coordinates (12, 26) in the video frame 1141 based on the coordinate information of the video frame 1142 and the coordinate information of the video frame 1141. The first XR glasses 1112 display the auxiliary line 115 at the coordinates (12, 26) on the video frame 1141.

[0199] To sum up, the method provided in the embodiment of the present application can accurately and timely display the marking information from the second XR glasses 1122 within the field of view of the first XR glasses 1112, that is, the marking information that the second user 1121 wants the first user to observe can be accurately and timely conveyed to the first user 1111, so that two-way field of view sharing is formed between the first XR glasses 1112 and the second XR glasses 1122, and the communication efficiency between the first user 1111 and the second user 1121 is effectively improved.

[0200] It should be understood that the method provided in the embodiment of the present application is applicable to any scenario that requires or supports vision sharing, such as general vision sharing scenarios, vision sharing scenarios for teaching purposes, business sharing scenarios for gaming purposes, and so on.

[0201] Fig.12 The structural block diagram of an XR-based field of view sharing device provided by an exemplary embodiment of the present application is shown. The device includes at least some of the following modules: a first sending module 1220, a first receiving module 1240, and a first processing module 1260:

[0202] A first sending module 1220, configured to send a shared video stream to a second pair of XR glasses, where the shared video stream is a video stream shared between the device and the second pair of XR glasses;

[0203] A first receiving module 1240 is configured to receive marking information sent by the second XR glasses, where the marking information is associated with a target element in the shared video stream;

[0204] The first processing module 1260 is configured to display the marking information associated with the target element in the shared video stream.

[0205] In some embodiments, the first processing module 1260 is further configured to determine the target element associated with the tag information in a first video frame in the shared video stream; track the target element in a second video frame in the shared video stream; and display the tag information associated with the target element in the second video frame;

[0206] The target element is a display element marked by the marking information, the first video frame is a video frame associated with the marking moment of the marking information, and the second video frame is a video frame in the shared video stream that is located after the first video frame.

[0207] In some embodiments, the target element includes coordinates; the first processing module 1260 is also used to determine, in the first video frame based on the coordinate information of the first video frame, a first coordinate associated with the marking information; and, based on the coordinate information of the second video frame, determine in the second video frame a second coordinate corresponding to the first coordinate.

[0208] In some embodiments, the target element includes a moving target; the first processing module 1260 is further used to track the moving target in the second video frame based on a motion tracking algorithm; and based on the tracking result of the moving target, determine the target element in the second video frame.

[0209] In some embodiments, the target element includes a pixel block; the first processing module 1260 is also used to divide the first video frame into M pixel blocks, where M is a positive integer; determine a first pixel block associated with the tag information among the M pixel blocks; and based on a pixel block matching algorithm, determine a second pixel block matching the first pixel block in the second video frame.

[0210] In some embodiments, the first receiving module 1240 is further configured to receive the marking information and the frame sequence number corresponding to the marking information sent by the second XR glasses;

[0211] The first processing module 1260 is further configured to determine the first video frame corresponding to the frame sequence number in the shared video stream; and determine the target element associated with the marking information in the first video frame.

[0212] In some embodiments, the first processing module 1260 is further configured to locally cache the most recently displayed N frames of video corresponding to the shared video stream, where N is a positive integer.

[0213] In some embodiments, the first processing module 1260 is further used to determine the number of frames N during local caching based on the network delay value, and the size of the number of frames N is positively correlated with the network delay value.

[0214] In some embodiments, the apparatus provided by the embodiments of the present application includes a first receiving module 1240, which supports the execution of all the receiving steps performed by the first XR glasses in the above-mentioned method embodiments.

[0215] In some embodiments, the apparatus provided by the embodiments of the present application includes multiple first receiving modules 1240, and the multiple first receiving modules 1240 respectively support the execution of some of the receiving steps performed by the first XR glasses in each of the above-mentioned method embodiments.

[0216] In some embodiments, the steps performed by different first receiving modules 1240 are completely the same, partially the same, or completely different.

[0217] In some embodiments, the apparatus provided by the embodiments of the present application includes a first sending module 1220, which supports the execution of all sending steps performed by the first XR glasses in the above-mentioned method embodiments.

[0218] In some embodiments, the apparatus provided by the embodiments of the present application includes multiple first sending modules 1220, and the multiple first sending modules 1220 respectively support the execution of part of the sending steps performed by the first XR glasses in each of the above-mentioned method embodiments.

[0219] In some embodiments, the steps performed by different first sending modules 1220 are completely the same, partially the same, or completely different.

[0220] In some embodiments, the device provided by the embodiments of the present application includes a first processing module 1260, which supports the execution of all determination-related steps, processing-related steps, and display-related steps performed by the first XR glasses in the above-mentioned method embodiments.

[0221] In some embodiments, the device provided by the embodiments of the present application includes multiple first processing modules 1260, which respectively support the execution of some of the determination-related steps and / or processing-related steps and / or display-related steps performed by the first XR glasses in the above-mentioned method embodiments.

[0222] In some embodiments, the steps performed by different first processing modules 1260 are completely the same, partially the same, or completely different.

[0223] In summary, the device provided in the embodiment of the present application forms efficient interaction between the device and the second XR glasses by transmitting shared video streams and marking information, thereby realizing two-way field of view sharing. The design of associating marking information with target elements is conducive to improving the accuracy of displaying marking information and the precision of field of view sharing, further improving the user experience.

[0224] Fig.13 The structural block diagram of an XR-based field of view sharing device provided by an exemplary embodiment of the present application is shown. The device includes at least some of the following modules: a second receiving module 1320, a second sending module 1340, and a second processing module 1360:

[0225] A second receiving module 1320 is configured to receive a shared video stream sent by the first XR glasses, where the shared video stream is a video stream shared between the first XR glasses and the device;

[0226] A second sending module 1340 is configured to send marking information to the first XR glasses, where the marking information is associated with a target element in the shared video stream;

[0227] The second processing module 1360 is configured to display the marking information associated with the target element in the shared video stream.

[0228] In some embodiments, the second sending module 1340 is further used to send marking information and a frame number corresponding to the marking information to the first XR glasses.

[0229] In some embodiments, the target element includes at least one of the following: coordinates; a moving target; a pixel block.

[0230] In some embodiments, the apparatus provided by the embodiments of the present application includes a second receiving module 1320, which supports the execution of all receiving steps performed by the second XR glasses in the above-mentioned method embodiments.

[0231] In some embodiments, the apparatus provided by the embodiments of the present application includes multiple second receiving modules 1320, and the multiple second receiving modules 1320 respectively support the execution of some of the receiving steps performed by the second XR glasses in the above-mentioned various method embodiments.

[0232] In some embodiments, the steps performed by different second receiving modules 1320 are completely the same, partially the same, or completely different.

[0233] In some embodiments, the apparatus provided by the embodiments of the present application includes a second sending module 1340, which supports the execution of all sending steps performed by the second XR glasses in the above-mentioned method embodiments.

[0234] In some embodiments, the apparatus provided by the embodiments of the present application includes multiple second sending modules 1340, and the multiple second sending modules 1340 respectively support the execution of part of the sending steps performed by the second XR glasses in each of the above-mentioned method embodiments.

[0235] In some embodiments, the steps performed by different second sending modules 1340 are completely the same, partially the same, or completely different.

[0236] In some embodiments, the device provided by the embodiments of the present application includes a second processing module 1360, which supports the execution of all determination-related steps, processing-related steps, and display-related steps performed by the second XR glasses in the above-mentioned method embodiments.

[0237] In some embodiments, the device provided by the embodiments of the present application includes multiple second processing modules 1360, which respectively support the execution of some of the determination-related steps and / or processing-related steps and / or display-related steps performed by the second XR glasses in the above-mentioned method embodiments.

[0238] In some embodiments, the steps performed by different second processing modules 1360 are completely the same, partially the same, or completely different.

[0239] In summary, the device provided in the embodiment of the present application forms efficient interaction between the device and the first XR glasses by transmitting a shared video stream and marking information, thereby realizing two-way vision sharing. The design of associating marking information with target elements is conducive to improving the accuracy of displaying marking information and the precision of vision sharing, further improving the user experience.

[0240] It should be noted that the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0241] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0242] Fig.14 A schematic diagram of the structure of an electronic device (or terminal device) provided by some exemplary embodiments of the present application is shown. The electronic device 1400 includes: a processor 1401, a receiver 1402, a transmitter 1403, a memory 1404 and a bus 1405.

[0243] The processor 1401 includes one or more processing cores, and the processor 1401 executes various functional applications and information processing by running software programs and modules. In some embodiments, the processor 1401 can be used to implement the functions and steps of the first processing module 1260 and / or the second processing module 1360 described above.

[0244] The receiver 1402 and the transmitter 1403 may be implemented as a communication component, which may be a communication chip. In some embodiments, the receiver 1402 may be used to implement the functions and steps of the first receiving module 1240 and / or the second receiving module 1320 described above. In some embodiments, the transmitter 1403 may be used to implement the functions and steps of sending the first sending module 1220 and / or the second sending module 1340 described above.

[0245] The memory 1404 is connected to the processor 1401 via a bus 1405. The memory 1404 may be used to store at least one instruction, and the processor 1401 may be used to execute the at least one instruction to implement each step in the above method embodiment.

[0246] In addition, the memory 1404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof. The volatile or non-volatile storage device includes but is not limited to: a magnetic disk or an optical disk, an Electrically Erasable Programmable Read Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Static Random-Access Memory (SRAM), a Read-Only Memory (ROM), a magnetic storage device, a flash memory, and a Programmable Read-Only Memory (PROM).

[0247] In some embodiments, the receiver 1402 receives signals / data independently, or the processor 1401 controls the receiver 1402 to receive signals / data, or the processor 1401 requests the receiver 1402 to receive signals / data, or the processor 1401 cooperates with the receiver 1402 to receive signals / data.

[0248] In some embodiments, the transmitter 1403 independently sends signals / data, or the processor 1401 controls the transmitter 1403 to send signals / data, or the processor 1401 requests the transmitter 1403 to send signals / data, or the processor 1401 cooperates with the transmitter 1403 to send signals / data.

[0249] In an exemplary embodiment of the present application, a computer-readable storage medium is further provided, wherein at least one program is stored in the computer-readable storage medium, and the at least one program is loaded and executed by a processor. The computer-readable storage medium is used to implement the XR-based field of view sharing method provided by the above-mentioned various method embodiments.

[0250] In an exemplary embodiment of the present application, a chip is also provided, which includes a programmable logic circuit and / or program instructions. When the chip runs on an electronic device, it is used to implement the XR-based field of view sharing method provided by the above-mentioned various method embodiments.

[0251] In an exemplary embodiment of the present application, a computer program product is further provided. When the computer program product is executed on a processor of a computer device, the computer device executes the above-mentioned XR-based field of view sharing method.

[0252] In an exemplary embodiment of the present application, a computer program is further provided. The computer program includes computer instructions. A processor of a computer device executes the computer instructions, so that the computer device executes the above-mentioned XR-based field of view sharing method.

[0253] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0254] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A field of view sharing method based on XR, characterized in that: The method is performed by a first pair of XR glasses, and the method includes: Sending a shared video stream to a second XR glasses, where the shared video stream is a video stream shared between the first XR glasses and the second XR glasses; receiving marking information sent by the second XR glasses, where the marking information is associated with a target element in the shared video stream; The marking information associated with the target element is displayed in the shared video stream.

2. The method according to claim 1, characterized in that The displaying the marking information associated with the target element in the shared video stream includes: In a first video frame in the shared video stream, determining the target element associated with the marking information; Tracking the target element in a second video frame in the shared video stream; Displaying the marking information associated with the target element in the second video frame; The target element is a display element marked by the marking information, the first video frame is a video frame associated with the marking moment of the marking information, and the second video frame is a video frame in the shared video stream that is located after the first video frame.

3. The method according to claim 2, characterized in that The target element includes coordinates; The step of determining, in the first video frame in the shared video stream, the target element associated with the tag information comprises: Based on the coordinate information of the first video frame, determining a first coordinate associated with the marking information in the first video frame; Tracking the target element in the second video frame includes: Based on the coordinate information of the second video frame, determining a second coordinate corresponding to the first coordinate in the second video frame; The display element corresponding to the first coordinate in the first video frame is the same as the display element corresponding to the second coordinate in the second video frame.

4. The method according to claim 2, characterized in that: The target element includes a moving target; Tracking the target element in the second video frame includes: Based on a motion tracking algorithm, tracking the moving target in the second video frame; Based on the tracking result of the moving target, the target element is determined in the second video frame.

5. The method according to claim 2, characterized in that: The target element includes a pixel block; The step of determining, in the first video frame in the shared video stream, the target element associated with the tag information comprises: Divide the first video frame into M pixel blocks, where M is a positive integer; Determining a first pixel block associated with the marking information among the M pixel blocks; Tracking the target element in the second video frame includes: Based on a pixel block matching algorithm, a second pixel block matching the first pixel block is determined in the second video frame.

6. The method according to any one of claims 2 to 5, characterized in that: The receiving the marking information sent by the second XR glasses includes: receiving the marking information and the frame sequence number corresponding to the marking information sent by the second XR glasses; The step of determining, in the first video frame in the shared video stream, the target element associated with the tag information comprises: Determine the first video frame corresponding to the frame sequence number in the shared video stream; The target element associated with the marking information is determined in the first video frame.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The most recently displayed N video frames corresponding to the shared video stream are cached locally, where N is a positive integer.

8. The method according to claim 7, characterized in that The method further comprises: The number of frames N during local caching is determined based on the network delay value, and the size of the number of frames N is positively correlated with the network delay value.

9. A field of view sharing method based on XR, characterized in that: The method is performed by a second pair of XR glasses, and the method includes: receiving a shared video stream sent by the first XR glasses, where the shared video stream is a video stream shared between the first XR glasses and the second XR glasses; Sending marking information to the first XR glasses, wherein the marking information is associated with a target element in the shared video stream; The marking information associated with the target element is displayed in the shared video stream.

10. The method according to claim 9, characterized in that The sending the marking information to the first XR glasses includes: Send the marking information and the frame number corresponding to the marking information to the first XR glasses.

11. The method according to claim 9 or 10, characterized in that: The target element includes at least one of the following: Coordinates; moving target; pixel block.

12. An XR-based field of view sharing device, characterized in that: The device comprises: A first sending module, configured to send a shared video stream to a second pair of XR glasses, where the shared video stream is a video stream shared between the device and the second pair of XR glasses; A first receiving module, configured to receive marking information sent by the second XR glasses, where the marking information is associated with a target element in the shared video stream; The first processing module is used to display the marking information associated with the target element in the shared video stream.

13. An XR-based field of view sharing device, characterized in that: The device comprises: A second receiving module, configured to receive a shared video stream sent by the first XR glasses, where the shared video stream is a video stream shared between the first XR glasses and the device; A second sending module, configured to send marking information to the first XR glasses, where the marking information is associated with a target element in the shared video stream; The second processing module is used to display the marking information associated with the target element in the shared video stream.

14. An electronic device, characterized in that: The electronic device comprises: processor; a transceiver connected to the processor; a memory for storing executable instructions for the processor; The processor is configured to load and execute the executable instructions to implement the XR-based field of view sharing method as described in any one of claims 1 to 8, or the XR-based field of view sharing method as described in any one of claims 9 to 11.

15. A computer-readable storage medium, characterized in that: The readable storage medium stores executable instructions, which are loaded and executed by a processor to implement the XR-based field of view sharing method as described in any one of claims 1 to 8, or the XR-based field of view sharing method as described in any one of claims 9 to 11.

16. A chip, characterized in that: The chip includes a programmable logic circuit or a program, and the chip is used to implement the XR-based field of view sharing method as described in any one of claims 1 to 8, or the XR-based field of view sharing method as described in any one of claims 9 to 11.

17. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the XR-based field of view sharing method as described in any one of claims 1 to 8, or the XR-based field of view sharing method as described in any one of claims 9 to 11.

18. A computer program, characterized in that The computer program includes computer instructions, and the processor of the computer device executes the computer instructions, so that the computer device performs the XR-based field of view sharing method as described in any one of claims 1 to 8, or the XR-based field of view sharing method as described in any one of claims 9 to 11.

Images