XR teaching interaction method and device, XR equipment and storage medium

Abstract

The embodiment of the invention provides an XR teaching interaction method and device, XR equipment and a storage medium, relates to the technical field of XR, solves the problem that users of the XR equipment are difficult to communicate effectively in an XR teaching scene, and can separately push data streams to other XR equipment by additionally generating a corresponding window in a virtual space corresponding to the XR equipment, so that the user experience is improved. Therefore, a window for communication is provided for the users, so that the users can better and effectively communicate with each other through the XR equipment.

Description

Technical Field

[0001] The present application relates to the field of XR (Extended Reality) technology, and in particular to an XR teaching interaction method, apparatus, XR device, and storage medium. Background Art

[0002] Interactive teaching methods can present teaching content more vividly, assist teachers and students in teaching activities, and thus improve teaching quality, and are therefore increasingly widely used in classrooms. XR technology is a general term for immersive technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR). Interactive teaching methods based on XR technology can create an immersive teaching environment that is virtual or a combination of virtual and real. Teachers and students wearing XR devices can immerse themselves in teaching activities in this created teaching environment.

[0003] However, the inventors discovered during practical application that in XR teaching scenarios, it's difficult for teachers and students to move around while wearing XR devices, making effective communication difficult when users need individual tutoring or group discussions. For example, when students encounter problems during teaching, teachers in the virtual environment can't provide timely guidance. This makes it difficult for teachers to take into account students' learning progress and fail to adjust teaching plans in a timely manner, thus affecting teaching quality. Summary of the Invention

[0004] The present application provides an XR teaching interaction method, apparatus, XR device, and storage medium, which solves the problem of difficulty in effective communication between users of XR devices in XR teaching scenarios. This solution can generate additional corresponding windows in the virtual space corresponding to the XR device to push data streams to other XR devices respectively, thereby providing users with a window for communication, so that users can better communicate effectively through XR devices.

[0005] In a first aspect, the present application provides an XR teaching interaction method, which is applied to a first XR device in an XR teaching system. The XR teaching system includes the first XR device and a second XR device. The first XR device and the second XR device are both used to display interactive data through virtual spaces provided by each device. The XR teaching interaction method includes:

[0006] Establish a connection with the second XR device based on the preset streaming method;

[0007] Generate a first window in the virtual space, and push a first window data stream corresponding to the first window to a second XR device based on a preset streaming method, so that the second XR device can reproduce the first window;

[0008] Upon receiving an independent streaming request initiated by the second XR device, generating a second window corresponding to the target second XR device associated with the independent streaming request in the virtual space;

[0009] Based on the preset streaming mode, the second window data stream corresponding to the second window is pushed to the target second XR device, so that the target second XR device can reproduce the second window.

[0010] In a second aspect, the present application further provides an XR teaching interaction device, which is applied to an XR device in an XR teaching system. The XR teaching system includes a first XR device and a second XR device. The first XR device and the second XR device are both used to display interactive data through virtual spaces provided by each device. The XR teaching interaction device includes:

[0011] a connection establishing module, configured to establish a connection with the second XR device based on a preset streaming mode;

[0012] A first window streaming module is configured to generate a first window in the virtual space and push a first window data stream corresponding to the first window to the second XR device based on a preset streaming method, so that the second XR device can reproduce the first window;

[0013] a request response module configured to, upon receiving an independent streaming request initiated by a second XR device, generate a second window corresponding to a target second XR device associated with the independent streaming request in the virtual space;

[0014] The second window streaming module is configured to push the second window data stream corresponding to the second window to the target second XR device based on a preset streaming method, so that the target second XR device can reproduce the second window.

[0015] In a third aspect, the present application further provides an XR device, comprising:

[0016] one or more processors;

[0017] A storage device is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the XR teaching interaction method as described above.

[0018] In a fourth aspect, the present application also provides a storage medium storing computer-executable instructions, which, when executed by a processor, are used to execute the XR teaching interaction method as described above.

[0019] This application generates additional windows for other XR devices that initiate independent streaming requests in the virtual space corresponding to the XR device, so as to push data streams to other XR devices respectively, thereby providing users with a window for communication. In the XR teaching scenario, teachers can provide guidance to students by operating the window, so that teachers and students can better communicate effectively through XR devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the steps of the XR teaching interaction method provided in one embodiment of the present application;

[0021] Figure 2 A schematic diagram of interaction between a teacher and a student in a teaching activity provided in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of the interaction after a student initiates an independent streaming request provided in one embodiment of the present application;

[0023] Figure 4 A schematic diagram of the steps for generating a second window provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the interaction before merging windows provided in one embodiment of the present application;

[0025] Figure 6 A schematic diagram of interaction after merging windows provided in one embodiment of the present application;

[0026] Figure 7 A schematic diagram of the steps of window merging processing provided in one embodiment of the present application;

[0027] Figure 8 A schematic diagram of the steps of pushing window data flow provided in one embodiment of the present application;

[0028] Figure 9 A schematic diagram of the structure of an XR teaching interaction device provided in one embodiment of the present application;

[0029] Figure 10 A schematic diagram of the structure of an XR device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and examples. It will be understood that the specific embodiments described herein are merely used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only portions related to the embodiments of the present application, rather than all structures, are shown in the accompanying drawings. After reading this specification, those skilled in the art should be able to understand that, as long as the technical features do not contradict each other, any combination of the technical features may constitute an optional embodiment.

[0031] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship. In the description of this application, "multiple" means two or more, and "several" means one or more.

[0032] XR devices are devices that apply XR technology, such as VR head-mounted displays, AR smart glasses, VR helmets and other wearable devices, and are equipped with operating handles to provide users with a virtual space in terms of vision and other senses. The operating handles can also be used to operate objects in the virtual space, giving users in the virtual space an immersive user experience.

[0033] XR devices can be used to cooperate with teachers and students in interactive teaching activities. In the XR teaching scenario, teachers and students wear XR devices to communicate, interact, lecture and other teaching activities in an immersive teaching environment. That is, XR devices can use XR technology to build a virtual space to display teaching content, and play related audio through the speakers on the device or through audio playback devices paired with wireless communication technologies such as Bluetooth and Star Flash.

[0034] However, in XR teaching scenarios, teachers and students wearing XR devices have limited visibility, making it difficult for them to move around. In scenarios where users need to communicate, such as individual tutoring between teachers and students or group discussions between students, effective communication through XR devices is difficult.

[0035] To this end, the present application provides an XR teaching interaction method, which is applied to a first XR device in an XR teaching system. The XR teaching system includes a first XR device and a second XR device, each of which is used to display interactive data in a virtual space provided by each device to facilitate communication between users.

[0036] It is worth noting that, for the sake of clarity, in this solution, the first XR device and the second XR device are used to distinguish the two ends of the interaction in the XR teaching system, and the XR teaching interaction method of this application is executed in the first XR device, and the first XR device pushes the stream to the second XR device. However, when the second XR device is used as the XR device for pushing the stream, the second XR device can also execute the XR teaching interaction method of this application. That is, in the specific description, the first XR device is used as the main body to describe this solution more clearly. If any one of the XR devices is used as the main body, it is the first XR device, and the other XR devices are the second XR devices.

[0037] Figure 1 This is a schematic diagram of the steps of an XR teaching interaction method provided in an embodiment of the present application. The method can be applied to a first XR device. The specific steps are as follows:

[0038] Step S110: Establish a connection with the second XR device based on the preset streaming method.

[0039] In the XR teaching system, there are multiple streaming methods to facilitate the transmission of interactive data between the first XR device and the second XR device. Therefore, after the first XR device determines the corresponding preset streaming method, it establishes a connection with the second XR device according to the communication protocol corresponding to the preset streaming method.

[0040] In some embodiments, the streaming mode includes a point-to-point streaming mode and a relay forwarding mode. The point-to-point streaming mode is used to push interactive data directly to the second XR device, and the relay forwarding mode is used to transmit interactive data, such as video data, audio data, etc., to the second XR device through the server in the XR teaching system, and the preset streaming mode adopted is any of the above streaming modes. For example, when the relay forwarding mode is used as the preset streaming mode, both the first XR device and the second XR device establish a communication connection with the server in the XR teaching system to facilitate data transmission.

[0041] Step S120: Generate a first window in the virtual space, and push a first window data stream corresponding to the first window to the second XR device based on a preset streaming method, so that the second XR device can reproduce the first window.

[0042] The first XR device generates a first window in its corresponding virtual space. The first XR device uses the display content displayed in the first window as video data and the audio collected by the device as audio data, thereby generating a first window data stream corresponding to the first window, and then pushes the first window data stream to the second XR device based on a preset streaming method.

[0043] After receiving the first window data stream, the second XR device will reproduce the first window in the corresponding virtual space, that is, generate a window in its corresponding virtual space to display the display content of the first window, and play audio through audio playback devices such as speakers on the device.

[0044] It can be understood that the first window serves as a public screen window between the first XR device and the second XR device, and the second XR device reproduces the display content of the first window through the same window. For example, in an XR teaching scenario, the teacher's XR device can serve as the first XR device and push the first window data stream corresponding to the first window to each student's XR device (which serves as the second XR device), so that each student can view the same display content on their corresponding XR device.

[0045] Step S130: When an independent streaming request initiated by the second XR device is received, a second window corresponding to the target second XR device associated with the independent streaming request is generated in the virtual space.

[0046] The second XR device can initiate an independent streaming request to the first XR device. For example, if the user of the second XR device needs to communicate, they can initiate an independent streaming request through the second XR device to inform the user of the first XR device. After receiving the independent streaming request, the first XR device generates a second window in its corresponding virtual space.

[0047] It is worth noting that the generated second window is associated with the target second XR device that initiates the independent streaming request. For example, when there are two target second XR devices that initiate the independent streaming request, the first XR device correspondingly generates two second windows and associates them with the two target second XR devices respectively.

[0048] Step S140: Push the second window data stream corresponding to the second window to the target second XR device based on the preset streaming mode, so that the target second XR device can reproduce the second window.

[0049] After generating the second window, the first XR device pushes the second window data stream to the target second XR device corresponding to the second window based on a preset streaming method, so that the target second XR device reproduces the second window according to the second window data stream after receiving the second window data stream.

[0050] For example, the present solution is described by taking the teacher's XR device as the first XR device and the student's XR device as the second XR device. It is conceivable that the teacher's XR device and the student's XR device can be determined by logging into the XR device account.

[0051] For ease of description, the teacher device is used as the XR device on the teacher side, and the student device is used as the XR device on the student side. Figure 2 This is a schematic diagram of the interaction between teachers and students in teaching activities provided in an embodiment of the present application, such as Figure 2 As shown in the figure, the virtual space corresponding to the XR device is represented by a box. In the XR teaching scenario, a first window W11 can be generated in the virtual space V1 corresponding to the teacher's device as a public screen window. For example, teaching content can be displayed through the first window W11. Then, the teacher's device pushes the first window data stream corresponding to the first window W11 to the student device, so that the student device generates a corresponding window in its virtual space to replicate the first window W11, such as window W12 displayed in the virtual space V2 corresponding to student A's XR device, window W13 displayed in the virtual space V3 corresponding to student B's XR device, and window W14 displayed in the virtual space V4 corresponding to student C's XR device.

[0052] Figure 3 The interactive diagram after the student initiates an independent push stream request provided for an embodiment of the present application is shown in the figure. As shown in the figure, when student A encounters a problem in class and needs to ask the teacher for advice, student A can initiate an independent push stream request to the teacher device through the student device A worn by the student (indicated by a dotted line with an arrow in the figure). The teacher device generates a second window W21 in the corresponding virtual space V1 in response to the independent push stream request and pushes the second window data stream corresponding to the second window W21 to the student device A. The student device A can then reproduce the second window W21 in the corresponding virtual space V2. As shown in the figure, the display content of the second window W21 is reproduced in the virtual space V2 as window W22, that is, the second window W21 serves as a shared window between the teacher device and the student device A. Both the teacher and the student A can edit the display content of the window in the space of their devices and transmit the relevant window data stream between the teacher device and the student device A to facilitate the teacher to tutor the student A.

[0053] From the above scheme, it can be seen that in the XR teaching system, the first XR device responds to the independent streaming request of the second XR device, generates a second window in the virtual space corresponding to the second XR device that initiates the independent streaming request, and pushes the window data stream corresponding to the second window to the second XR device, so that the second device can reproduce the second window, thereby providing users with a window for communication, realizing one-to-one or one-to-many communication, so that users can communicate through the XR device while wearing the XR device.

[0054] In one embodiment, the first XR device needs to generate a corresponding second window for the received independent streaming request to facilitate additional data exchange with the second XR device that initiated the independent streaming request. Figure 4 A schematic diagram of the steps for generating a second window provided in an embodiment of the present application, wherein the specific steps are as follows:

[0055] Step S210: Based on the reception time of the corresponding independent streaming request, copy the display content of the first window to determine the target display content.

[0056] Step S220: corresponding to an independent streaming request, a second window is generated in the virtual space, and the target display content is displayed in the second window.

[0057] It is understandable that the first XR device records the time of receipt of the independent push stream request, and thus copies the display content of the first window based on the reception time, that is, according to the reception time, the display content displayed by the first window at the reception time is selected and used as the target display content. It is conceivable that when the first XR device receives independent push stream requests sent by multiple different second XR devices, the first XR device generates a second window in the virtual space corresponding to each independent push stream request, that is, the first XR device generates a corresponding number of second windows based on the number of independent push stream requests.

[0058] The generated second window displays the target display content. Specifically, the target display content displayed in the generated second window is a copy of the display content in the first window at the time the independent push request was received. For different independent push requests, the first XR device generates second windows with different display content. Each second window is also associated with the corresponding target second XR device that initiated the independent push request, allowing the user to prioritize the push to the target second XR device through the first XR device.

[0059] For example, referring to the example above, in an XR teaching scenario, while presenting a knowledge point, the teacher device receives an independent streaming request from a student device. In response, the teacher device copies the display content of the first window, such as the presented knowledge point, as the target display content. Furthermore, the teacher device generates a second window in response to the independent streaming request. This second window is associated with the student device, meaning that the student device serves as the target second XR device. Furthermore, the target display content is displayed in the second window.

[0060] Therefore, the first XR device responds to independent streaming requests initiated by different second XR devices. The first XR device generates a second window, and then pushes the second window data stream corresponding to the second window to the corresponding second XR device, providing users with a window for communication, so that users can better communicate effectively through the first XR device and the second XR device.

[0061] In some embodiments, when there are multiple second windows, the user can click, drag, or perform other operations on the second window in the virtual space provided by the first XR device to select or merge the second windows. Therefore, in the first XR device, when the first XR device receives a window movement operation of any second window by the user, such as dragging the second window, the first XR device determines the second window to be merged and then performs a window merging process on the second window to be merged. The second windows to be merged are second windows whose overlapping areas meet a preset ratio requirement.

[0062] Moreover, the first XR device also re-associates the second XR device associated with the merged window. For example, after any two second windows are merged, the merged window is associated with the two second XR devices associated with the two merged second windows.

[0063] It is conceivable that the window merging process can be repeated, that is, when there are multiple second windows, the user can merge any number of second windows to synchronously push the stream to different second XR devices.

[0064] For example, Figure 5 and Figure 6 As shown, Figure 5 This is a schematic diagram of the interaction before merging windows provided in an embodiment of the present application. Figure 6 A schematic diagram of the interaction after merging windows provided in an embodiment of the present application. In the XR teaching scenario, corresponding to the independent streaming requests of student device A and student device B, two second windows are displayed in the virtual space V1 corresponding to the teacher's device, such as window W21 corresponding to student device A and window W31 corresponding to student device B. If the teacher determines that the issues that the two students need to communicate are the same, the teacher can correspondingly merge window W21 and window W31 through the teacher's device to obtain window Wm1, which is associated with student device A and student device B. The teacher's device pushes the corresponding window data stream to student device A and student device B, so that window Wm1 can be reproduced through window Wm2 in the virtual space V2 corresponding to student device A, and window Wm1 can be reproduced through window Wm3 in the virtual space V3 corresponding to student device B. Therefore, the teacher can tutor student A and student B at the same time through window Wm1.

[0065] In one embodiment, Figure 7 This is a schematic diagram of the steps for window merging processing provided in an embodiment of the present application. The specific steps are as follows:

[0066] Step S310: Detect the window position of the source window in real time to determine whether the source window has an overlapping area with another second window, where the source window is the second window selected for the window moving operation.

[0067] Step S320: If there is an overlapping area between the source window and the target window, determine whether the area ratio of the overlapping area in the source window is greater than or equal to a preset ratio, and the target window is the second window that has an overlapping area with the source window.

[0068] Step S330: If the area ratio of the overlapping region in the source window is greater than or equal to the preset ratio, determine that the source window and the target window are the second windows to be merged, merge the source window and the target window to generate a merged window, and associate the merged window with the second XR device corresponding to the source window and the second XR device corresponding to the target window.

[0069] It is understood that the source window serves as the second window selected for the window move operation. In response to the window move operation, the first XR device detects the window position of the source window in real time, such as the coordinates of each vertex of the window in the corresponding coordinate system in the virtual space, to determine whether the source window overlaps with the other second window. It is conceivable that in some embodiments, a window plane is set in the virtual space to display the first window and the second window. Accordingly, a coordinate system is constructed using the window plane to determine the vertex coordinates of each window and locate the window position of each window on the window plane.

[0070] The target window is a second window that overlaps with the source window. For example, when a vertex of the source window falls within the range defined by the vertices of another second window during the movement of the source window, the source window and the second window will overlap accordingly. The second window is the target window. The first XR device determines the area ratio of the overlapping area in the source window to determine whether the area ratio is greater than or equal to a preset ratio.

[0071] When the area ratio of the overlapping area in the source window is greater than or equal to the preset ratio, the first XR device determines that the source window and the target window are windows to be merged, and merges the source window and the target window to generate a merged window, and then associates the merged window with the second XR device corresponding to the source window and the second XR device corresponding to the target window, so that the window data stream can be pushed to the corresponding second XR device at the same time when pushing the stream.

[0072] Therefore, a window merging function is provided in the XR device to push the corresponding window data stream to multiple second XR devices, effectively synchronizing one-to-many data transmission, so that teachers can communicate with multiple students at the same time and provide better tutoring.

[0073] In one embodiment, in response to the push operation, the first XR device collects the second window data stream corresponding to the selected second window and pushes it to the target second XR device, so that the target second XR device can reproduce the second window in its corresponding virtual space. Figure 8 As shown, Figure 8 A schematic diagram of the steps of pushing window data flow provided in an embodiment of the present application, the specific steps are as follows:

[0074] Step S410: In response to a push start operation on the window to be pushed, a target second XR device corresponding to the window to be pushed is determined based on a correspondence between the second window and the second XR device, where the window to be pushed is the second window selected for the push start operation.

[0075] Step S420: collect the display content in the window to be pushed in real time and obtain the audio data after the response time of the push start operation to generate a second window data stream.

[0076] Step S430: Push the second window data stream to the target second XR device based on the preset streaming method.

[0077] It is understandable that the window to be pushed is the second window selected for the push stream initiation operation. For example, if there are multiple second windows on the first XR device, before pushing the stream, the user needs to select a second window as the window to be pushed to push data to the target second XR device corresponding to the window. It is conceivable that after the first XR device generates a second window, it will record the second XR device corresponding to the second window. Therefore, after determining the window to be pushed, the corresponding target second XR device can also be determined.

[0078] The first XR device collects the display content in the window to be pushed in real time, and the first XR device also obtains audio data collected by the audio collection device on the device to obtain audio data after the response time of the push start operation, thereby generating a second window data stream. When the preset push mode is determined, the first XR device pushes the second window data stream to the target second XR device based on the preset push mode, so that the target second XR device reproduces the second window in the corresponding virtual space.

[0079] For example, in an XR teaching scenario, in response to independent streaming requests from student devices A and B, two second windows appear in the virtual space corresponding to the teacher's device: Window I for student device A and Window II for student device B. The teacher selects Window I as the window to be pushed, and student device A as the target second XR device. The teacher's device captures the display content of Window I and the corresponding audio, which is then used as the second window data stream for push to the target second XR device.

[0080] In one embodiment, a server is further provided in the XR teaching system so that the first XR device and the second XR device can transmit interactive data through the server, that is, the streaming method adopted between the first XR device and the second XR device can be a point-to-point streaming method or a relay forwarding method. The first XR device can select one of the above methods as the preset streaming method based on the bandwidth information obtained from the server, thereby realizing the push of interactive data.

[0081] The corresponding streaming service is registered on the server to provide relay forwarding services for the first XR device and the second XR device, thereby receiving the window data stream sent by the first XR device and pushing the stream to the second XR device. The server detects the bandwidth occupancy of the streaming service and transmits the bandwidth occupancy as bandwidth information to the first XR device. The first XR device determines the preset push streaming method to be adopted based on the bandwidth occupancy of the streaming service and the preset threshold.

[0082] For example, the first XR device compares the bandwidth usage with a preset threshold and determines a preset streaming mode based on the comparison result. If the bandwidth usage is less than the preset threshold, the relay forwarding mode is used as the preset streaming mode; if the bandwidth usage is greater than or equal to the preset threshold, the point-to-point streaming mode is used as the preset streaming mode to avoid the high bandwidth usage causing significant delays in the transmission of interactive data.

[0083] It should be noted that, in some embodiments, the user may pre-set the streaming mode to be adopted, such as selecting the relay forwarding mode as the preset streaming mode. Correspondingly, the first XR device may also switch the streaming mode based on the above scheme, that is, by comparing the bandwidth occupancy rate and the preset threshold value, so that when the bandwidth occupancy rate is less than the preset threshold value, the relay forwarding mode is still maintained as the preset streaming mode, and when the bandwidth occupancy rate is greater than or equal to the preset threshold value, the preset streaming mode is switched to the point-to-point streaming mode.

[0084] Therefore, by comparing bandwidth occupancy rates, XR devices can select appropriate streaming methods, which helps maintain smooth data interaction between XR devices and enables timely delivery of interactive data to avoid large delays that affect the transmission of interactive data.

[0085] Figure 9 This is a schematic diagram of the structure of an XR teaching interaction device provided in one embodiment of the present application. The device is applied to the first XR device in the XR teaching system. The XR teaching system includes the first XR device and the second XR device. The first XR device and the second XR device are both used to display interactive data through the virtual space provided by each. The device is used to execute the XR teaching interaction method provided in the above embodiment and has functional modules and beneficial effects corresponding to the execution method. As shown in the figure, the device includes a connection establishment module 501, a first window streaming module 502, a request response module 503 and a second window streaming module 504.

[0086] Among them, the connection establishment module 501 is configured to establish a connection with the second XR device based on a preset streaming method. The first window streaming module 502 is configured to generate a first window in the virtual space, and push the first window data stream corresponding to the first window to the second XR device based on the preset streaming method, so that the second XR device can reproduce the first window. The request response module 503 is configured to generate a second window corresponding to the target second XR device associated with the independent streaming request in the virtual space when receiving an independent streaming request initiated by the second XR device. The second window streaming module 504 is configured to push the second window data stream corresponding to the second window to the target second XR device based on the preset streaming method, so that the target second XR device can reproduce the second window.

[0087] Based on the above embodiment, the request response module 502 is further configured to:

[0088] Based on the reception time of the corresponding independent streaming request, copy the display content of the first window to determine the target display content;

[0089] Corresponding to an independent streaming request, a second window is generated in the virtual space, and the target display content is displayed in the second window.

[0090] Based on the above embodiment, the device further includes a window merging module. When there are multiple second windows, the window merging module is configured as follows:

[0091] In response to a window move operation on any second window, a window merging process is performed on the plurality of second windows to be merged, and a second XR device corresponding to the merged window is determined.

[0092] Based on the above embodiment, the window merging module is further configured as follows:

[0093] detecting a window position of a source window in real time to determine whether the source window has an overlapping area with another second window, where the source window is the second window selected for the window move operation;

[0094] In the case where there is an overlapping area between the source window and the target window, determining whether the area ratio of the overlapping area in the source window is greater than or equal to a preset ratio, the target window is the second window that has an overlapping area with the source window;

[0095] If the area ratio of the overlapping region in the source window is greater than or equal to the preset ratio, the source window and the target window are determined to be the second windows to be merged, and the source window and the target window are merged to generate a merged window, and the merged window is associated with the second XR device corresponding to the source window and the second XR device corresponding to the target window.

[0096] Based on the above embodiment, the second window streaming module 504 is further configured as follows:

[0097] In response to a push-stream initiation operation on the window to be pushed, determining a target second XR device corresponding to the window to be pushed based on a correspondence between the second window and the second XR device, where the window to be pushed is the second window selected for the push-stream initiation operation;

[0098] Collect the display content in the window to be pushed in real time and obtain the audio data after the response time of the push start operation to generate a second window data stream;

[0099] Based on the preset streaming method, push the second window data stream to the target second XR device.

[0100] Based on the above embodiment, the preset push streaming mode is a point-to-point push streaming mode or a relay forwarding mode. The point-to-point push streaming mode is used to push the interactive data directly to the second XR device, and the relay forwarding mode is used to transmit the interactive data to the second XR device through the server in the XR teaching system. The device also includes a push streaming selection module, which is configured as follows:

[0101] Based on a preset threshold and the bandwidth occupancy rate of the server for the streaming media service, the preset streaming mode to be adopted is determined, and the streaming media service is used to provide relay forwarding services for the first XR device and the second XR device.

[0102] Based on the above embodiment, the streaming selection module is further configured as follows:

[0103] Compare the preset threshold and bandwidth usage;

[0104] When the bandwidth usage is less than the preset threshold, the preset streaming mode is switched to the relay forwarding mode;

[0105] When the bandwidth usage is greater than or equal to a preset threshold, the preset streaming mode is switched to a point-to-point streaming mode.

[0106] It is worth noting that in the embodiment of the above-mentioned XR teaching interaction device, the modules are divided only according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0107] Figure 10 This is a schematic diagram of the structure of an XR device provided in one embodiment of the present application. The device is used to execute the XR teaching interaction method provided in the above embodiment and has functional modules and beneficial effects corresponding to the execution method. As shown in the figure, the XR device includes a processor 601, a memory 602, an input device 603, and an output device 604. The number of processors 601 can be one or more, and the figure uses one processor 601 as an example; the processor 601, memory 602, input device 603, and output device 604 can be connected via a bus or other means, and the figure uses a bus connection as an example. The memory 602, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the XR teaching interaction method in the embodiment of the present application. The processor 601 executes the corresponding various functional applications and data processing by running the software programs, instructions, and modules stored in the memory 602, that is, implementing the above-mentioned XR teaching interaction method.

[0108] The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data recorded or created during use, etc. In addition, the memory 602 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 602 may further include a memory remotely located relative to the processor 601, and these remotely located memories may be connected to the terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0109] The input device 603 can be used to input corresponding digital or character information to the processor 601, and generate key signal input related to the user settings and function control of the device; the output device 604 can be used to send or display key signal output related to the user settings and function control of the device.

[0110] An embodiment of the present application also provides a storage medium storing computer-executable instructions, which, when executed by a processor, are used to perform relevant operations in the XR teaching interaction method provided in any embodiment of the present application.

[0111] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0112] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0113] Note that the above are only preferred embodiments of the present application and the technical principles employed. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present application. The scope of the present application is determined by the scope of the appended claims.

Claims

1. An XR teaching interaction method, characterized in that: A first XR device is used in an XR teaching system, the XR teaching system including the first XR device and the second XR device, the first XR device and the second XR device are both used to display interactive data through virtual spaces provided by each device, and the XR teaching interaction method includes: Establishing a connection with the second XR device based on the preset streaming method; Generate a first window in the virtual space, and push a first window data stream corresponding to the first window to the second XR device based on the preset streaming method, so that the second XR device reproduces the first window; Upon receiving an independent streaming request initiated by the second XR device, generating a second window in the virtual space corresponding to the target second XR device associated with the independent streaming request; Based on the preset streaming mode, the second window data stream corresponding to the second window is pushed to the target second XR device, so that the target second XR device reproduces the second window.

2. The XR teaching interaction method according to claim 1, characterized in that: Upon receiving the independent streaming request initiated by the second XR device, generating, in the virtual space, a second window corresponding to the target second XR device associated with the independent streaming request includes: Copying the display content of the first window based on the reception time of the independent streaming request to determine the target display content; Corresponding to one of the independent streaming requests, a second window is generated in the virtual space, and the target display content is displayed in the second window.

3. The XR teaching interaction method according to claim 1, characterized in that: In the case of receiving the independent streaming request initiated by the second XR device, after generating the second window corresponding to the second XR device in the virtual space, if there are multiple second windows, the method further includes: In response to a window move operation on any of the second windows, a window merging process is performed on a plurality of second windows to be merged, and a second XR device corresponding to the merged window is determined.

4. The XR teaching interaction method according to claim 3, characterized in that: The step of performing a window merging process on a plurality of second windows to be merged in response to a window move operation on any second window, and determining a second XR device corresponding to the merged window includes: detecting a window position of a source window in real time to determine whether the source window has an overlapping area with another second window, the source window being the second window selected by the window moving operation; In the case where there is an overlapping area between the source window and the target window, determining whether a proportion of an area of ​​the overlapping area in the source window is greater than or equal to a preset proportion, the target window being a second window having an overlapping area with the source window; If the area ratio of the overlapping region in the source window is greater than or equal to the preset ratio, the source window and the target window are determined to be second windows to be merged, and the source window and the target window are merged to generate a merged window, and the merged window is associated with the second XR device corresponding to the source window and the second XR device corresponding to the target window.

5. The XR teaching interaction method according to claim 1, characterized in that: The pushing, based on the preset streaming mode, the second window data stream corresponding to the second window to the target second XR device so that the target second XR device reproduces the second window includes: In response to a push-stream initiation operation on a window to be pushed, determining a target second XR device corresponding to the window to be pushed based on a correspondence between the second window and the second XR device, where the window to be pushed is the second window selected for the push-stream initiation operation; Collecting the display content in the window to be pushed in real time and obtaining the audio data after the response time of the push start operation to generate the second window data stream; Based on the preset streaming mode, push the second window data stream to the target second XR device.

6. The XR teaching interaction method according to any one of claims 1 to 5, characterized in that: The preset streaming mode is a point-to-point streaming mode or a relay forwarding mode. The point-to-point streaming mode is used to push the interactive data directly to the second XR device, and the relay forwarding mode is used to transmit the interactive data to the second XR device through the server in the XR teaching system. Before establishing a connection with the second XR device based on the preset streaming mode, the method further includes: Based on a preset threshold and the bandwidth occupancy of the server for the streaming service, a preset streaming mode is determined, and the streaming service is used to provide relay forwarding services for the first XR device and the second XR device.

7. The XR teaching interaction method according to claim 6, characterized in that: The determining of a preset streaming mode based on a preset threshold and a bandwidth occupancy rate of the server for the streaming service, wherein the streaming service is used to provide a relay forwarding service for the first XR device and the second XR device, includes: Comparing the preset threshold with the bandwidth occupancy rate; When the bandwidth occupancy rate is less than the preset threshold, switching the preset streaming mode to the relay forwarding mode; When the bandwidth occupancy rate is greater than or equal to the preset threshold, the preset streaming mode is switched to the point-to-point streaming mode.

8. An XR teaching interactive device, characterized in that: A first XR device used in an XR teaching system, the XR teaching system including the first XR device and the second XR device, both of which are used to display interactive data through virtual spaces provided by each device, the XR teaching interaction device including: a connection establishing module, configured to establish a connection with the second XR device based on a preset streaming mode; a first window streaming module, configured to generate a first window in the virtual space, and push a first window data stream corresponding to the first window to the second XR device based on the preset streaming method, so that the second XR device reproduces the first window; a request response module configured to, upon receiving an independent streaming request initiated by the second XR device, generate a second window in the virtual space corresponding to the target second XR device associated with the independent streaming request; The second window streaming module is configured to push the second window data stream corresponding to the second window to the target second XR device based on the preset streaming mode, so that the target second XR device can reproduce the second window.

9. An XR device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when one or more of the programs are executed by one or more of the processors, so that the one or more processors implement the XR teaching interaction method as claimed in any one of claims 1 to 7.

10. A storage medium storing computer-executable instructions, characterized in that: When executed by a processor, the computer-executable instructions are used to execute the XR teaching interaction method according to any one of claims 1 to 7.

Images