XR interactive teaching methods, devices, XR equipment and storage media

Abstract

This application provides an XR teaching interaction method, device, XR equipment, and storage medium, relating to the field of XR technology. It solves the problem of effective communication between users of XR devices in XR teaching scenarios. This solution can generate additional windows in the virtual space corresponding to the XR device to push data streams to other XR devices, thereby providing users with a window for communication and enabling users to communicate more effectively through XR devices.

Description

Technical Field

[0001] This application relates to the field of XR (Extended Reality) technology, and in particular to an XR teaching interaction method, device, XR equipment and storage medium. Background Technology

[0002] Interactive teaching methods can present teaching content more vividly and effectively, assisting teachers and students in their learning activities and improving teaching quality, thus leading to their increasingly widespread application in classrooms. XR technology is a general term for immersive technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). Interactive teaching methods based on XR technology can construct virtual or hybrid virtual-real immersive learning environments, allowing teachers and students wearing XR devices to engage in immersive learning activities within these environments.

[0003] However, the inventors discovered during practical application that in XR teaching scenarios, it is inconvenient for teachers and students to move around while wearing XR devices, and effective communication is difficult when users need one-on-one tutoring or group discussions. For example, when students encounter problems during the teaching process, teachers cannot provide timely guidance in the virtual environment, making it difficult for teachers to keep track of students' learning progress and adjust teaching plans in a timely manner, thus affecting the quality of teaching. Summary of the Invention

[0004] This application provides an XR teaching interaction method, device, XR equipment, and storage medium, which solves the problem of ineffective communication between users of XR devices in XR teaching scenarios. This solution can generate additional windows in the virtual space corresponding to the XR device to push data streams to other XR devices, thereby providing users with a window for communication, so that users can communicate more effectively through XR devices.

[0005] In a first aspect, this application provides an XR teaching interaction method, applied to a first XR device in an XR teaching system. The XR teaching system includes a first XR device and a second XR device, both of which are used to display interactive data through their respective provided virtual spaces. The XR teaching interaction method includes:

[0006] Based on the preset streaming method, a connection is established with the second XR device;

[0007] A first window is generated in the virtual space, and the first window data stream corresponding to the first window is pushed to the second XR device according to the preset push streaming method so that the second XR device can reproduce the first window;

[0008] Upon receiving an independent streaming request initiated by a second XR device, a second window corresponding to the target second XR device associated with the independent streaming request is generated in the virtual space;

[0009] Based on a preset streaming method, the second window data stream corresponding to the second window is pushed to the target second XR device so that the target second XR device can reproduce the second window.

[0010] Secondly, this application also provides an XR teaching interactive device, applied to an XR device in an XR teaching system. The XR teaching system includes a first XR device and a second XR device, both of which are used to display interactive data through their respective virtual spaces. The XR teaching interactive device includes:

[0011] The connection establishment module is configured to establish a connection with the second XR device based on a preset streaming method.

[0012] The first window push module is configured to generate a first window in a virtual space and push the first window data stream corresponding to the first window to the second XR device based on a preset push method, so that the second XR device can reproduce the first window;

[0013] The request response module is configured to generate a second window in the virtual space corresponding to the target second XR device associated with the independent push stream request when it receives an independent push stream request initiated by the second XR device.

[0014] The second window push module is configured to push the second window data stream corresponding to the second window to the target second XR device based on a preset push method, so that the target second XR device can reproduce the second window.

[0015] Thirdly, this application also provides an XR device, which includes:

[0016] One or more processors;

[0017] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the XR teaching interaction method described above.

[0018] Fourthly, this application also provides a storage medium storing computer-executable instructions, which, when executed by a processor, are used to perform the XR teaching interaction method described above.

[0019] This application generates additional windows for other XR devices that initiate independent streaming requests within the virtual space corresponding to the XR device, thereby pushing data streams to other XR devices and providing users with a window for communication. In XR teaching scenarios, teachers can guide students by operating the window, enabling better and more effective communication between teachers and students through XR devices. Attached Figure Description

[0020] Figure 1 A schematic diagram illustrating the steps of an XR teaching interaction method provided in an embodiment of this application;

[0021] Figure 2 An interactive diagram of a teacher and student conducting teaching activities, provided as an embodiment of this application;

[0022] Figure 3 This is an interactive diagram illustrating the process after a student initiates an independent push stream request, as provided in one embodiment of this application.

[0023] Figure 4 This is a schematic diagram illustrating the steps for generating a second window according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the interaction before the merged window provided in one embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the interaction after merging windows according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram illustrating the steps of window merging processing provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram illustrating the steps of a push window data stream provided in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the structure of an XR teaching interactive device provided in an embodiment of this application;

[0029] Figure 10 This is a schematic diagram of the structure of an XR device provided in an embodiment of this application. Detailed Implementation

[0030] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of this application. Furthermore, it should be noted that, for ease of description, the accompanying drawings only show the parts related to the embodiments of this application, not all structures. Those skilled in the art, after reading this specification, should be able to conceive that any combination of technical features can constitute an optional implementation method, provided that the technical features do not contradict each other.

[0031] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In the description of this application, "multiple" means two or more, and "several" means one or more.

[0032] XR devices are wearable devices that utilize XR technology, such as VR headsets, AR smart glasses, and VR helmets, and are equipped with controllers to provide users with a virtual space in terms of visual and other sensory experiences. Users can also use the controllers to operate objects in the virtual space, giving them an immersive user experience.

[0033] XR devices can be used to facilitate interactive teaching activities between teachers and students. In XR teaching scenarios, teachers and students can communicate, interact, and lecture in an immersive teaching environment by wearing XR devices. In other words, XR devices can use XR technology to build virtual spaces to display teaching content and play relevant audio through speakers on the device or through audio playback devices paired with wireless communication technologies such as Bluetooth and StarFlash.

[0034] However, in XR teaching scenarios, both teachers and students experience limited movement due to obstructed vision while wearing XR devices. In situations requiring communication, such as one-on-one tutoring or group discussions, effective communication between users is difficult using XR devices.

[0035] In response, this application provides an XR teaching interaction method, which is applied to a first XR device in an XR teaching system. The XR teaching system includes a first XR device and a second XR device, both of which are used to display interactive data through their respective virtual spaces to facilitate communication between users.

[0036] It is worth noting that, for clarity, this solution distinguishes the two ends of the XR teaching system through a first XR device and a second XR device. The XR teaching interaction method of this application is executed in the first XR device, pushing the stream from the first XR device to the second XR device. However, when the second XR device acts as the pushing XR device, it can also execute the XR teaching interaction method of this application. That is, the description focuses on the first XR device for clarity, and any one of the XR devices is considered the first XR device, while the others are considered the second XR devices.

[0037] Figure 1 This is a schematic diagram illustrating the steps of an XR teaching interaction method provided in an embodiment of this application. The method can be applied to a first XR device, and the specific steps are as follows:

[0038] Step S110: Establish a connection with the second XR device based on the preset streaming method.

[0039] In an XR teaching system, there are multiple streaming modes to facilitate the transmission of interactive data between the first XR device and the second XR device. Therefore, after the first XR device determines the corresponding preset streaming mode, it establishes a connection with the second XR device according to the communication protocol corresponding to the preset streaming mode.

[0040] In some embodiments, the streaming method includes a point-to-point streaming method and a relay forwarding method. The point-to-point streaming method is used to directly push interactive data to the second XR device, while the relay forwarding method is used to transmit interactive data, such as video data and audio data, to the second XR device through a server in the XR teaching system. The preset streaming method used is any one of the aforementioned streaming methods. For example, when the relay forwarding method is the preset streaming method, both the first XR device and the second XR device establish a communication connection with the server in the XR teaching system to facilitate data transmission.

[0041] Step S120: Generate a first window in the virtual space, and push the first window data stream corresponding to the first window to the second XR device based on the preset push streaming method, so that the second XR device can reproduce the first window.

[0042] The first XR device generates a first window in its corresponding virtual space. The first XR device uses the content displayed in the first window as video data and the audio collected by the device as audio data, thereby generating a first window data stream corresponding to the first window. Then, it pushes the first window data stream to the second XR device based on a preset streaming method.

[0043] After receiving the data stream from the first window, the second XR device will reproduce the first window in the corresponding virtual space. That is, it will generate a window in its corresponding virtual space to display the content of the first window and play audio through audio playback devices such as speakers on the device.

[0044] It is understandable that the first window serves as a shared screen window between the first and second XR devices, with the second XR device replicating the content displayed in the first window through the same window. For example, in an XR teaching scenario, the teacher's XR device can act as the first XR device, pushing the data stream of the first window corresponding to the first window to each student's XR device (which acts as the second XR device), so that each student can view the same display content on their corresponding XR device.

[0045] Step S130: Upon receiving an independent streaming request initiated by the second XR device, generate a second window in the virtual space corresponding to the target second XR device associated with the independent streaming request.

[0046] The second XR device can initiate an independent streaming request to the first XR device. For example, if the user of the second XR device needs to communicate, they can initiate an independent streaming request through the second XR device to inform the user of the first XR device. Upon receiving the independent streaming request, the first XR device generates a second window in its corresponding virtual space.

[0047] It is worth noting that the generated second window is associated with the target second XR device that initiated the independent streaming request. For example, when there are two target second XR devices that initiated the independent streaming request, the first XR device generates two second windows accordingly and associates them with the two target second XR devices mentioned above.

[0048] Step S140: Based on the preset streaming method, push the second window data stream corresponding to the second window to the target second XR device so that the target second XR device can reproduce the second window.

[0049] After the second window is generated, the first XR device pushes the second window data stream to the target second XR device corresponding to the second window based on a preset push streaming method, so that the target second XR device can reproduce the second window according to the second window data stream after receiving it.

[0050] For example, this solution is illustrated by using the teacher's XR device as the first XR device and the student's XR device as the second XR device. It is conceivable that the teacher's XR device and the student's XR device can be identified by logging into the XR device's account.

[0051] For ease of description, the teacher's device is used as the teacher's XR device, and the student's device is used as the student's XR device. Figure 2 This application provides an example of an interactive diagram illustrating a teaching activity between a teacher and students, such as... Figure 2 As shown in the diagram, the virtual space corresponding to the XR device is represented by a box. In an XR teaching scenario, a first window W11, which serves as a public screen window, can be generated in the virtual space V1 corresponding to the teacher's device. For example, teaching content can be displayed through the first window W11. Then, the teacher's device pushes the data stream of the first window corresponding to the first window W11 to the student's device, so that the student's device can generate corresponding windows in its virtual space to reproduce the first window W11. For example, window W12 is displayed in the virtual space V2 corresponding to student A's XR device, window W13 is displayed in the virtual space V3 corresponding to student B's XR device, and window W14 is displayed in the virtual space V4 corresponding to student C's XR device.

[0052] Figure 3 The figure illustrates the interaction after a student initiates an independent streaming request, as provided in one embodiment of this application. When student A encounters a problem in class and needs to consult the teacher, student A can initiate an independent streaming request to the teacher's device via their student device A (represented by a dashed line with an arrow in the figure). The teacher's device, in response to this independent streaming request, generates a second window W21 in the corresponding virtual space V1 and pushes the second window data stream corresponding to W21 to student device A. Student device A can then reproduce the second window W21 in the corresponding virtual space V2. As shown in the figure, the display content of the second window W21 is reproduced in virtual space V2 as window W22. That is, the second window W21 serves as a shared window between the teacher's device and student device A. Both the teacher and student A can edit the display content of the window in their respective device's space and transmit related window data streams between the teacher's device and student device A, facilitating the teacher's guidance of student A.

[0053] As can be seen from the above scheme, in the XR teaching system, the first XR device responds to the independent streaming request of the second XR device. The second XR device that initiated the independent streaming request generates a second window in the virtual space and pushes the window data stream corresponding to the second window to the second XR device so that the second device can reproduce the second window, thereby providing a window for users to communicate and realize one-to-one or one-to-many communication, so that users can communicate through the XR device while wearing it.

[0054] In one embodiment, for each independent streaming request received, the first XR device needs to generate a corresponding second window to facilitate additional data interaction with the second XR device that initiated the independent streaming request. Figure 4 This is a schematic diagram illustrating the steps for generating a second window according to an embodiment of this application. The specific steps are as follows:

[0055] Step S210: Based on the receiving time of the corresponding independent push stream request, copy the display content of the first window to determine the target display content.

[0056] Step S220: For each independent streaming request, a second window is generated in the virtual space, and the target content is displayed in the second window.

[0057] Understandably, the first XR device records the reception time of each independent streaming request, and then copies the display content of the first window based on that reception time. That is, it selects the display content shown in the first window at the reception time and uses it as the target display content. It is conceivable that when the first XR device receives multiple independent streaming requests from different second XR devices, it generates a second window in virtual space for each independent streaming request; that is, the first XR device generates a corresponding number of second windows based on the number of independent streaming requests.

[0058] The generated second window displays the target content, which is copied from the first window according to the reception time corresponding to the independent streaming request. For different independent streaming requests, the first XR device generates a second window with different display content, and each second window is also associated with the corresponding target second XR device that initiated the independent streaming request, so that the user can determine which target second XR device to prioritize streaming to through the first XR device.

[0059] For example, referring to the above example, in an XR teaching scenario, when displaying a knowledge point, the teacher's device receives an independent streaming request from a student's device. Correspondingly, the teacher's device copies the content displayed in the first window, such as the knowledge point content shown above, as the target display content. Furthermore, the teacher's device generates a second window for this independent streaming request, which is associated with the student's device; that is, the student's device acts as the target second XR device. In addition, the target display content is displayed in the second window.

[0060] Therefore, in response to independent streaming requests initiated by different second XR devices, the first XR device generates a second window and then pushes the corresponding second window data stream to the corresponding second XR device, providing users with a window for communication, so that users can communicate more effectively through the first XR device and the second XR device.

[0061] In some embodiments, when multiple second windows exist, the user can click, drag, or perform other operations on the second windows within the virtual space provided by the first XR device to select or merge them. Therefore, when the first XR device receives a user's window movement operation on any second window, such as dragging it, the first XR device, after identifying the second window to be merged, performs window merging processing on it. The second window to be merged consists of several overlapping areas that meet a preset percentage requirement.

[0062] Furthermore, the first XR device also re-associates the second XR device associated with the merged window. For example, after any two second windows are merged, the merged window is associated with the two second XR devices associated with the two merged second windows.

[0063] It is conceivable that this window merging process can be repeated, meaning that in the presence of multiple second windows, users can merge any number of second windows to simultaneously push streams to different second XR devices.

[0064] For example, such as Figure 5 and Figure 6 As shown, Figure 5 This is a schematic diagram of the interaction before the merge window is provided in one embodiment of this application. Figure 6 This is a schematic diagram of the interaction after merging windows according to an embodiment of this application. In an XR teaching scenario, corresponding to the independent push requests of student device A and student device B, two second windows are displayed in the virtual space V1 corresponding to the teacher device, such as window W21 corresponding to student device A and window W31 corresponding to student device B. If the teacher determines that the problem that the two students need to communicate about is the same, the teacher can merge windows W21 and W31 through the teacher device to obtain window Wm1. Window Wm1 is associated with student device A and student device B. The teacher device pushes the corresponding window data stream to student device A and student device B so that window Wm1 can be reproduced in the virtual space V2 corresponding to student device A through window Wm2, and in the virtual space V3 corresponding to student device B through window Wm3. Therefore, the teacher can tutor student A and student B simultaneously through window Wm1.

[0065] In one embodiment, Figure 7 This is a schematic diagram illustrating the steps of window merging processing provided in an embodiment of this application. The specific steps are as follows:

[0066] Step S310: Real-time detection of the source window's position to determine whether the source window overlaps with other second windows. The source window is the second window selected by the window movement operation.

[0067] Step S320: If there is an overlapping area between the source window and the target window, determine whether the area ratio of the overlapping area in the source window is greater than or equal to the preset ratio, and the target window is the second window that has an overlapping area with the source window.

[0068] Step S330: If the area ratio of the overlapping region in the source window is greater than or equal to the preset ratio, then the source window and the target window are determined to be the second window to be merged, and the source window and the target window are merged to generate a merged window, and the merged window is associated with the second XR device corresponding to the source window and the second XR device corresponding to the target window.

[0069] Understandably, the source window serves as the second window selected by the window movement operation. During the response to the window movement operation, the first XR device detects the source window's position in real time, such as the coordinates of each vertex in the virtual space's corresponding coordinate system, to determine whether the source window overlaps with other second windows. In some embodiments, a window plane is set in the virtual space to display the first and second windows. Correspondingly, a coordinate system is constructed using the window plane to determine the vertex coordinates of each window, thereby locating the window position of each window on the window plane.

[0070] The target window is a second window that overlaps with the source window. For example, during the movement of the source window, if a vertex of the source window falls within the area defined by the vertices of another second window, the source window will overlap with the second window, which is the target window. The first XR device determines the area ratio of the overlapping area within the source window to determine whether the area ratio is greater than or equal to a preset ratio.

[0071] When the area ratio of the overlapping region in the source window is greater than or equal to the preset ratio, the first XR device determines the source window and the target window as windows to be merged, and then merges the source window and the target window to generate a merged window. The merged window is then associated with the second XR device corresponding to the source window and the second XR device corresponding to the target window, so that the window data stream can be pushed to the corresponding second XR device at the same time during the streaming.

[0072] Therefore, XR devices provide a window merging function to push the corresponding window data stream to multiple second XR devices, effectively synchronizing one-to-many data transmission, so that teachers can communicate with multiple students at the same time and provide better guidance.

[0073] In one embodiment, in response to a streaming operation, the first XR device acquires the second window data stream corresponding to the selected second window and pushes it to a target second XR device, so that the target second XR device can reproduce the second window in its corresponding virtual space. Figure 8 As shown, Figure 8 This is a schematic diagram illustrating the steps of a push window data stream provided in an embodiment of this application. The specific steps are as follows:

[0074] Step S410: In response to the push stream start operation of the window to be pushed, based on the correspondence between the second window and the second XR device, determine the target second XR device corresponding to the window to be pushed, and the window to be pushed is the second window selected by the push stream start operation.

[0075] Step S420: Real-time acquisition of the displayed content in the window to be pushed and acquisition of audio data after the response time of the push stream start operation to generate a second window data stream.

[0076] Step S430: Based on the preset streaming method, push the second window data stream to the target second XR device.

[0077] Understandably, the window to be pushed serves as the second window selected for the push streaming initiation operation. For example, if multiple second windows exist on the first XR device, the user needs to select one as the window to be pushed before pushing data to the target second XR device corresponding to that window. It's conceivable that after generating a second window, the first XR device will record the second XR device corresponding to that window. Therefore, once the window to be pushed is determined, the corresponding target second XR device can also be determined.

[0078] The first XR device captures the content displayed in the push window in real time, and also acquires audio data collected by its on-device audio acquisition device to obtain audio data after the response time of the push operation, thereby generating a second window data stream. With a preset push method determined, the first XR device pushes this second window data stream to the target second XR device based on the preset push method, so that the target second XR device can reproduce the second window in the corresponding virtual space.

[0079] For example, in an XR teaching scenario, for independent streaming requests from student device A and student device B, the teacher's device displays two second windows in its virtual space, such as window I for student device A and window II for student device B. The teacher selects window I as the window to be pushed, correspondingly designating student device A as the target second XR device. The teacher's device captures the content displayed on window I and obtains the corresponding audio, which is then used as the second window data stream to push to the target second XR device.

[0080] In one embodiment, the XR teaching system is further provided with a server so that the first XR device and the second XR device can transmit interactive data through the server. That is, the streaming method used between the first XR device and the second XR device can be a point-to-point streaming method or a relay forwarding method. The first XR device can select one of the above methods as the preset streaming method according to the bandwidth information obtained from the server, thereby realizing the push of interactive data.

[0081] The server registers a corresponding streaming media service to provide relay forwarding services for the first and second XR devices, enabling the reception of window data streams sent by the first XR device and the pushing of streams to the second XR device. The server detects the bandwidth utilization of the streaming media service and transmits this bandwidth utilization as bandwidth information to the first XR device. The first XR device determines the preset pushing method to be used based on the bandwidth utilization of the streaming media service and a preset threshold.

[0082] For example, the first XR device compares the bandwidth utilization rate with a preset threshold to determine the preset streaming method based on the comparison result. If the bandwidth utilization rate is less than the preset threshold, the relay forwarding method is used as the preset streaming method; if the bandwidth utilization rate is greater than or equal to the preset threshold, the point-to-point streaming method is used as the preset streaming method to avoid the high bandwidth utilization rate causing a large delay in the transmission of interactive data.

[0083] It should be noted that in some embodiments, the user may pre-set the streaming method, such as selecting relay forwarding as the preset streaming method. Correspondingly, the first XR device may also switch the streaming method based on the above scheme. That is, by comparing the bandwidth utilization rate and the preset threshold, if the bandwidth utilization rate is less than the preset threshold, the relay forwarding method will still be maintained as the preset streaming method, while if the bandwidth utilization rate is greater than or equal to the preset threshold, the preset streaming method will be switched to the point-to-point streaming method.

[0084] Therefore, by comparing bandwidth utilization, XR devices can select appropriate streaming methods, which helps XR devices maintain smooth data interaction and can transmit interactive data in a timely manner, so as to avoid large delays that affect the transmission of interactive data.

[0085] Figure 9 This is a schematic diagram of the structure of an XR teaching interaction device provided in an embodiment of this application. The device is applied to a first XR device in an XR teaching system. The XR teaching system includes a first XR device and a second XR device, both of which are used to display interactive data through their respective virtual spaces. The device is used to execute the XR teaching interaction method provided in the above embodiment and has the corresponding functional modules and beneficial effects. As shown in the figure, the device includes a connection establishment module 501, a first window streaming module 502, a request response module 503, and a second window streaming module 504.

[0086] The connection establishment module 501 is configured to establish a connection with the second XR device based on a preset streaming method. The first window streaming module 502 is configured to generate a first window in a virtual space and push the first window data stream corresponding to the first window to the second XR device based on a preset streaming method, so that the second XR device can reproduce the first window. The request response module 503 is configured to generate a second window corresponding to the target second XR device associated with the independent streaming request in the virtual space when it receives an independent streaming request initiated by the second XR device. The second window streaming module 504 is configured to push the second window data stream corresponding to the second window to the target second XR device based on a preset streaming method, so that the target second XR device can reproduce the second window.

[0087] Based on the above embodiments, the request response module 502 is further configured as follows:

[0088] Based on the receiving time of the corresponding independent push stream request, copy the display content of the first window to determine the target display content;

[0089] For each independent streaming request, a second window is generated in the virtual space, and the target content is displayed in the second window.

[0090] Based on the above embodiments, the device further includes a window merging module. For cases where multiple second windows exist, the window merging module is configured as follows:

[0091] In response to a window movement operation on any second window, a window merging process is performed on several second windows to be merged, and the second XR device corresponding to the merged window is determined.

[0092] Based on the above embodiments, the window merging module is further configured as follows:

[0093] The window position of the source window is detected in real time to determine whether the source window overlaps with other second windows. The source window is the second window selected by the window movement operation.

[0094] If there is an overlapping area between the source window and the target window, determine whether the area ratio of the overlapping area in the source window is greater than or equal to the preset ratio, and the target window is the second window that has an overlapping area with the source window.

[0095] If the area ratio of the overlapping region in the source window is greater than or equal to the preset ratio, the source window and the target window are determined to be the second window to be merged, and the source window and the target window are merged to generate a merged window. The merged window is then associated with the second XR device corresponding to the source window and the second XR device corresponding to the target window.

[0096] Based on the above embodiments, the second window streaming module 504 is further configured as follows:

[0097] In response to the push stream start operation of the window to be pushed, the target second XR device corresponding to the window to be pushed is determined based on the correspondence between the second window and the second XR device. The window to be pushed is the second window selected by the push stream start operation.

[0098] The system collects the content displayed in the window to be pushed in real time and obtains the audio data after the response time of the push operation to generate a second window data stream.

[0099] Based on the preset streaming method, a second window data stream is pushed to the target second XR device.

[0100] Based on the above embodiments, the preset streaming method is either point-to-point streaming or relay forwarding. Point-to-point streaming directly pushes interactive data to the second XR device, while relay forwarding transmits interactive data to the second XR device via a server in the XR teaching system. The device also includes a streaming selection module, configured as follows:

[0101] Based on preset thresholds and the server's bandwidth utilization rate for streaming media services, the preset streaming method is determined, and the streaming media service is used to provide relay forwarding services for the first XR device and the second XR device.

[0102] Based on the above embodiments, the streaming selection module is further configured as follows:

[0103] Compare preset thresholds with bandwidth utilization;

[0104] When the bandwidth utilization rate is less than the preset threshold, the preset push streaming mode will be switched to the relay forwarding mode;

[0105] When the bandwidth utilization rate is greater than or equal to the preset threshold, the preset push streaming mode will be switched to the point-to-point push streaming mode.

[0106] It is worth noting that in the above-mentioned embodiments of the XR teaching interactive device, the modules are divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each module are only for easy differentiation and are not used to limit the scope of protection of this application.

[0107] Figure 10 This is a schematic diagram of the structure of an XR device provided in an embodiment of this application. The device is used to execute the XR teaching interaction method provided in the above embodiment and has corresponding functional modules and beneficial effects for executing the method. As shown in the figure, the XR device includes a processor 601, a memory 602, an input device 603, and an output device 604. The number of processors 601 can be one or more; one processor 601 is shown as an example in the figure. The processor 601, memory 602, input device 603, and output device 604 can be connected via a bus or other means; a bus connection is shown as an example in the figure. The memory 602, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the XR teaching interaction method in the embodiments of this application. The processor 601 executes various corresponding functional applications and data processing by running the software programs, instructions, and modules stored in the memory 602, thereby realizing the above-mentioned XR teaching interaction method.

[0108] The memory 602 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data recorded or created during use. Furthermore, the memory 602 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 602 may further include memory remotely located relative to the processor 601, which can be connected to a terminal device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] The input device 603 can be used to input corresponding digital or character information to the processor 601, and to generate key signal inputs related to the user settings and function control of the device; the output device 604 can be used to send or display key signal outputs related to the user settings and function control of the device.

[0110] This application also provides a storage medium storing computer-executable instructions, which, when executed by a processor, are used to perform relevant operations in the XR teaching interaction method provided in any embodiment of this application.

[0111] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0112] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0113] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. An XR pedagogical interaction method, characterized in that, A first XR device is used in an XR teaching system, the XR teaching system including a first XR device and a second XR device, both the first XR device and the second XR device being used to display interactive data through their respective provided virtual spaces, the XR teaching interaction method including: Based on the preset streaming method, a connection is established with the second XR device; A first window is generated in the virtual space, and a first window data stream corresponding to the first window is pushed to the second XR device based on the preset streaming method, so that the second XR device can reproduce the first window; Upon receiving an independent streaming request initiated by the second XR device, a second window corresponding to the target second XR device associated with the independent streaming request is generated in the virtual space; Based on the preset streaming method, the second window data stream corresponding to the second window is pushed to the target second XR device so that the target second XR device can reproduce the second window; Wherein, upon receiving an independent streaming request initiated by the second XR device, generating a second window in the virtual space corresponding to the target second XR device associated with the independent streaming request includes: Based on the receiving time of the corresponding independent push request, the display content of the first window is copied to determine the target display content; For each independent streaming request, a second window is generated in the virtual space, and the target display content is displayed in the second window. In the presence of multiple second windows, in response to a window movement operation on any of the second windows, a window merging process is performed on several second windows to be merged, and the second XR device corresponding to the merged window is determined. The step of responding to a window movement operation on any of the second windows, performing window merging processing on a plurality of second windows to be merged, and determining the second XR device corresponding to the merged window includes: The window position of the source window is detected in real time to determine whether the source window overlaps with other second windows, wherein the source window is the second window selected by the window movement operation; If the source window and the target window have an overlapping area, it is determined whether the area ratio of the overlapping area in the source window is greater than or equal to a preset ratio, and the target window is a second window that has an overlapping area with the source window; If the area ratio of the overlapping region in the source window is greater than or equal to the preset ratio, then the source window and the target window are determined to be the second window to be merged, and the source window and the target window are merged to generate a merged window, and the merged window is associated with the second XR device corresponding to the source window and the second XR device corresponding to the target window.

2. The XR pedagogical interaction method of claim 1, wherein, The step of pushing the second window data stream corresponding to the second window to the target second XR device based on the preset streaming method, so that the target second XR device can reproduce the second window, includes: In response to the push-to-the-window operation, based on the correspondence between the second window and the second XR device, the target second XR device corresponding to the push-to-the-window is determined, and the push-to-the-window is the second window selected by the push-to-the-window operation. The display content within the window to be pushed is collected in real time, and audio data after the response time of the push operation is obtained, so as to generate the second window data stream; Based on the preset streaming method, the second window data stream is pushed to the target second XR device.

3. The XR pedagogical interaction method of any of claims 1-2, wherein, The preset streaming method is either point-to-point streaming or relay forwarding. The point-to-point streaming method is used to directly push interactive data to the second XR device, and the relay forwarding method is used to transmit interactive data to the second XR device through the server in the XR teaching system. Before establishing a connection with the second XR device based on a preset streaming method, the method further includes: Based on a preset threshold and the server's bandwidth utilization rate for the streaming media service, a preset streaming method is determined, and the streaming media service is used to provide relay forwarding services for the first XR device and the second XR device.

4. The XR pedagogical interaction method of claim 3, wherein, The step of determining the preset streaming method based on a preset threshold and the server's bandwidth utilization rate for the streaming media service, wherein the streaming media service is used to provide relay forwarding services for the first XR device and the second XR device includes: Compare the preset threshold with the bandwidth utilization rate; When the bandwidth utilization rate is less than the preset threshold, the preset push streaming mode will be switched to the relay forwarding mode; When the bandwidth utilization rate is greater than or equal to the preset threshold, the preset streaming mode is switched to the point-to-point streaming mode.

5. An XR teaching interaction device, characterized in that, A first XR device is applied in an XR teaching system, the XR teaching system including a first XR device and a second XR device, both the first XR device and the second XR device being used to display interactive data through their respective provided virtual spaces, the XR teaching interactive device including: The connection establishment module is configured to establish a connection with the second XR device based on a preset streaming method; The first window streaming module is configured to generate a first window in the virtual space and push the first window data stream corresponding to the first window to the second XR device based on the preset streaming method, so that the second XR device can reproduce the first window; The request response module is configured to generate a second window in the virtual space corresponding to the target second XR device associated with the independent streaming request when receiving an independent streaming request initiated by the second XR device. The second window push module is configured to push the second window data stream corresponding to the second window to the target second XR device based on the preset push method, so that the target second XR device can reproduce the second window; The request-response module is specifically configured as follows: Based on the receiving time of the corresponding independent push request, the display content of the first window is copied to determine the target display content; For each independent streaming request, a second window is generated in the virtual space, and the target display content is displayed in the second window. The window merging module is configured as follows: In the presence of multiple second windows, in response to a window movement operation on any of the second windows, a window merging process is performed on several second windows to be merged, and the second XR device corresponding to the merged window is determined. The window merging module is specifically configured as follows: The window position of the source window is detected in real time to determine whether the source window overlaps with other second windows, wherein the source window is the second window selected by the window movement operation; If the source window and the target window have an overlapping area, it is determined whether the area ratio of the overlapping area in the source window is greater than or equal to a preset ratio, and the target window is a second window that has an overlapping area with the source window; If the area ratio of the overlapping region in the source window is greater than or equal to the preset ratio, then the source window and the target window are determined to be the second window to be merged, and the source window and the target window are merged to generate a merged window, and the merged window is associated with the second XR device corresponding to the source window and the second XR device corresponding to the target window.

6. An XR device, comprising: include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more of the processors, cause the one or more of the processors to implement the XR teaching interaction method as claimed in any one of claims 1-4.

7. A storage medium having stored computer-executable instructions, wherein: The computer-executable instructions, when executed by a processor, are used to perform the XR teaching interaction method as described in any one of claims 1-4.

Images