Equipment communication method and device, equipment, storage medium and program product

Abstract

The invention discloses an equipment communication method and device, equipment, a storage medium and a program product, and relates to the technical field of computers. The method comprises the following steps: establishing a first long connection with a first control device through the Internet, and establishing a second long connection with an augmented reality device through the Internet; establishing a pairing relationship between the first control device and the augmented reality device; under the condition that the augmented reality equipment executes a state change process related to the first application program, receiving state real-time information sent by the augmented reality equipment through the second long connection; and sending state real-time information to the first control equipment through the first long connection based on the pairing relationship. According to the method, equipment interaction can be more stable by means of a long connection communication mode, and it can be guaranteed that the augmented reality equipment sends the state real-time information to the first control equipment more timely and accurately. The method can be applied to various scenes such as smart home scenes and smart game scenes.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a device communication method, apparatus, device, storage medium, and program product. Background Art

[0002] With the development of computer technology, smart head-mounted displays have gradually become a new type of interactive device that participates in the daily lives of users. Smart head-mounted displays usually need to be managed and controlled by external terminal devices (such as smart phones, tablets, computers, etc.) to achieve corresponding functions and operations. Smart head-mounted displays are widely used in augmented reality, virtual reality and other fields, and can provide users with richer and more convenient interactive experience.

[0003] In related technologies, smart head displays are usually connected to external terminal devices through Bluetooth near-field communication technology. The external terminal device can control the smart head display based on the Bluetooth connection, and the smart head display can also feedback information such as the device operating status to the external terminal device based on the Bluetooth connection.

[0004] However, due to the limited connection range of Bluetooth technology, external terminal devices can only achieve device control, device communication and other processes when they are within a small distance from the smart headset. In addition, the device connection status may also be affected by physical obstacles, interference and other factors, resulting in poor connection stability. Summary of the invention

[0005] The embodiments of the present application provide a device communication method, apparatus, device, storage medium and program product, which can make the device interaction more stable by means of a long connection communication method, and help ensure that the extended reality device sends the real-time status information to the first control device more timely and accurately. The technical solution is as follows.

[0006] In one aspect, a device communication method is provided, the method being executed by a server, the method comprising:

[0007] Establishing a first long connection with a first control device through an Internet network, and establishing a second long connection with an extended reality device through the Internet network, wherein a pairing relationship exists between the first control device and the extended reality device;

[0008] When the extended reality device executes a state change process related to the first application, receiving real-time state information sent by the extended reality device through the second persistent connection;

[0009] The real-time status information is sent to the first control device through the first long connection based on the pairing relationship, where the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0010] On the other hand, a device communication method is provided, the method being executed by a first control device, the method comprising:

[0011] A first long connection is established with a server via an Internet network, wherein the first control device is paired with an extended reality device, and a second long connection is established between the extended reality device and the server via the Internet network;

[0012] In response to receiving a device control operation, sending a first control instruction to the server through the first persistent connection, wherein the device control operation is used to control the operation of the extended reality device, and the first control instruction is used to control the extended reality device to execute a state change process related to the first application;

[0013] Receive real-time status information corresponding to the extended reality device, where the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0014] In another aspect, a device communication method is provided, the method being performed by an extended reality device, the method comprising:

[0015] A second long connection is established with the server through the Internet, the extended reality device is paired with the first control device, and the first control device has a first long connection established with the server through the Internet;

[0016] generating status real-time information when executing a status change process associated with the first application;

[0017] The real-time status information is sent to the server through the second long connection, and the server is used to send the real-time status information to the first control device through the first long connection based on the pairing relationship, and the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0018] In another aspect, a device communication apparatus is provided, the apparatus comprising:

[0019] a connection establishment module, configured to establish a first long connection with a first control device through an Internet network, and to establish a second long connection with an extended reality device through the Internet network, wherein a pairing relationship exists between the first control device and the extended reality device;

[0020] an information receiving module, configured to receive real-time status information sent by the extended reality device through the second persistent connection when the extended reality device executes a status change process related to the first application;

[0021] An information sending module is used to send the real-time status information to the first control device through the first long connection based on the pairing relationship, and the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0022] In another aspect, a device communication apparatus is provided, the apparatus comprising:

[0023] A connection establishment module, configured to establish a first long connection with a server via an Internet network, wherein a first control device and an extended reality device are paired, and a second long connection is established between the extended reality device and the server via the Internet network;

[0024] an instruction sending module, configured to send a first control instruction to the server through the first persistent connection in response to receiving a device control operation, wherein the device control operation is used to control the operation of the extended reality device, and the first control instruction is used to control the extended reality device to execute a state change process related to the first application;

[0025] An information receiving module is used to receive real-time status information corresponding to the extended reality device, and the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0026] In another aspect, a device communication apparatus is provided, the apparatus comprising:

[0027] a connection establishing module, configured to establish a second long connection with the server via the Internet, wherein the extended reality device is paired with the first control device, and the first control device has established a first long connection with the server via the Internet;

[0028] An information generation module, for generating real-time status information when executing a status change process related to the first application;

[0029] An information sending module is used to send the real-time status information to the server through the second long connection, and the server is used to send the real-time status information to the first control device through the first long connection based on the pairing relationship, and the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0030] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a device communication method as described in any of the above-mentioned embodiments of the present application.

[0031] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a device communication method as described in any of the above-mentioned embodiments of the present application.

[0032] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the device communication method described in any of the above embodiments.

[0033] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0034] By establishing a first long connection between the Internet and the first control device and a second long connection between the Internet and the extended reality device, the extended reality device can receive real-time status information more stably when executing the state change process related to the first application, and then can send real-time status information to the first control device more stably, so that the first control device can synchronize the display and state change process. This process makes the device interaction more stable with the help of the long connection communication method, which helps to ensure that the extended reality device sends the real-time status information to the first control device more timely and accurately, reduces the network burden, saves resource consumption, and makes the device interaction suitable for application scenarios with strong real-time and frequent interactions. It can also be applied to remote operation scenarios where the distance between the first control device and the extended reality device is far, improving the application of device interaction under the device pairing relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1is a structural block diagram of a communication system provided by an exemplary embodiment of the present application;

[0037] Figure 2 is a flow chart of a method for executing device communication by a server provided by an exemplary embodiment of the present application;

[0038] Figure 3 is a flow chart of a method for executing device communication by a server provided by another exemplary embodiment of the present application;

[0039] Figure 4 is a flow chart of a method for executing device communication by a server provided by another exemplary embodiment of the present application;

[0040] Figure 5 is a schematic diagram of display status real-time information provided by an exemplary embodiment of the present application;

[0041] Figure 6 is a flowchart of a method for executing device communication by a first control device provided by an exemplary embodiment of the present application;

[0042] Figure 7 is a flowchart of a method for performing device communication on an extended reality device provided by an exemplary embodiment of the present application;

[0043] Figure 8 It is a schematic diagram of the overall architecture of device interaction provided by an exemplary embodiment of the present application;

[0044] Fig. 9 is a structural block diagram of a device communication apparatus provided by an exemplary embodiment of the present application;

[0045] Fig.10 is a structural block diagram of a device communication apparatus provided by another exemplary embodiment of the present application;

[0046] Fig.11 is a structural block diagram of a device communication apparatus provided by another exemplary embodiment of the present application;

[0047] Fig.12 It is a structural block diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0049] In related technologies, smart head-mounted displays are usually connected to external terminal devices through Bluetooth near-field communication technology. External terminal devices can control smart head-mounted displays based on Bluetooth connections, and smart head-mounted displays can also feedback information such as device operating status to external terminal devices based on Bluetooth connections. However, due to the limited connection range of Bluetooth technology, external terminal devices can only achieve device control and device communication processes within a short distance from the smart head-mounted display, and the device connection status may also be affected by physical obstacles, interference and other factors, resulting in poor connection stability.

[0050] In an embodiment of the present application, a device communication method is introduced, which can make the device interaction more stable by means of a long connection communication method, and help ensure that the extended reality device sends the real-time status information to the first control device more timely and accurately. The device communication method proposed in the embodiment of the present application can be applied to smart home scenarios, smart game scenarios, industrial Internet of Things scenarios, smart medical scenarios, remote health detection scenarios, smart transportation scenarios, smart agriculture scenarios, smart public service scenarios, smart retail scenarios, smart logistics scenarios and other communication scenarios in which the control device and the extended reality device interact, which are not limited here.

[0051] In an optional embodiment, the device communication method is applied to a smart home scenario as an example.

[0052] Optionally, taking the execution subject as a server as an example, a first long connection is established with a first control device through the Internet, and a second long connection is established with an extended reality device through the Internet, wherein a pairing relationship exists between the first control device and the extended reality device.

[0053] Schematically, the first control device and the extended reality device are each logged in with the same device account, such as the device account is an account logged in to the first control device and the extended reality device remotely, wherein the first control device is a mobile phone terminal logged in to account a at location W, and the user is the daughter, and the extended reality device is an XR headset logged in to account a at location M, and the user is the mother; a first long connection is established between the first control device and the server through the Internet to achieve stable communication between the first control device and the server; in addition, a second long connection is established between the extended reality device and the server through the Internet to achieve stable communication between the extended reality device and the server; the pairing relationship between the first control device and the extended reality device can be achieved by logging in to the same device account.

[0054] Optionally, when the extended reality device executes a state change process related to the first application, real-time status information sent by the extended reality device through the second long connection is received; and the real-time status information is sent to the first control device through the first long connection.

[0055] Illustratively, a second long connection is established between the server and the extended reality device, so that when the extended reality device executes a state change process related to the first application, the real-time status information generated based on the state change process can be sent to the server, so that the server sends the real-time status information to the first control device through the first long connection based on the pairing relationship between the first control device and the extended reality device. The real-time status information is used to instruct the first control device to synchronously display the state change process related to the first application, so that the first control device with the pairing relationship can render and display the real-time status information. For example, a daughter in place W can promptly understand the operation status of the XR headset used by her mother, so as to prompt the mother in place M how to operate the XR headset, etc.

[0056] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of the relevant regions.

[0057] The communication system involved in the embodiment of the present application is described. The device communication method provided in the embodiment of the present application is implemented by the terminal and the server through data interaction; for illustration, please refer to Figure 1 The communication system involves a first control device 110, a server 120 and an extended reality device 130.

[0058] The first control device 110 can be implemented as a terminal, including but not limited to mobile terminals such as mobile phones, tablet computers, portable laptop computers, intelligent voice interaction devices, smart home appliances, and vehicle-mounted terminals. It can also be implemented as a desktop computer, etc., and can also be implemented as another extended reality device different from the extended reality device 130.

[0059] The extended reality device 130 (Extended Reality, XR) is a comprehensive device that integrates at least one of the technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), and can provide an immersive and interactive experience for the user. The extended reality device 130 includes, but is not limited to, virtual reality devices such as head-mounted displays, motion controllers, force feedback devices (such as tactile gloves, force feedback external devices), augmented reality devices such as smart glasses and AR helmets, mixed reality devices, holographic projection devices that project stereoscopic three-dimensional images through laser, light field and other technologies, etc., which are not limited here.

[0060] In some embodiments, the first control device 110 and the extended reality device 130 both have device communication functions, the server 120 establishes a first long connection with the first control device 110 through the Internet, and the server 120 establishes a second long connection with the extended reality device 130 through the Internet.

[0061] The first long connection and the second long connection refer to different long connection communications connecting interactive devices. The long connection communication allows the device to remain open for a certain period of time after being connected to the server 120, thereby enabling continuous two-way data transmission, that is, the first control device 110 can communicate with the server 120 continuously for a certain period of time based on the first long connection. Similarly, the extended reality device 130 can also communicate with the server 120 continuously for a certain period of time based on the second long connection, which is not limited here.

[0062] In some embodiments, when the extended reality device 120 executes a state change process related to the first application, the server 120 receives real-time status information sent by the extended reality device through the second long connection.

[0063] Illustratively, when the extended reality device 120 executes a state change process related to the first application, the extended reality device 120 generates real-time status information based on the state change process, and the real-time status information can show the real-time changes in the state of the first application; the extended reality device 120 can then send the real-time status information to the server 120 based on the second long connection.

[0064] In some embodiments, the server 120 sends real-time status information to the first control device 110 through the first long connection based on the pairing relationship between the first control device and the extended reality device.

[0065] The real-time status information is used to instruct the first control device 110 to synchronously display the state change process related to the first application. Schematically, the server 120 acts as a transfer device when there is a pairing relationship between the first control device 110 and the extended reality device 120. After receiving the real-time status information, the first control device 110 can render and display it to determine in real time the result of the state change process related to the first application executed by the extended reality device 130.

[0066] It is worth noting that the above-mentioned server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server.

[0067] In combination with the above-mentioned noun introduction and application scenarios, the device communication method provided by the present application is described, and the method is applied to a server as an example. Figure 2 As shown, the method includes the following steps 210 to 230.

[0068] Step 210: Establish a first long connection with the first control device through the Internet, and establish a second long connection with the extended reality device through the Internet.

[0069] Illustratively, the first control device can be implemented as a mobile terminal including but not limited to a mobile phone, a tablet computer, a portable laptop computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc., and can also be implemented as a desktop computer, etc., and can also be implemented as another extended reality device different from the extended reality device.

[0070] Extended reality devices are a type of comprehensive device that integrates at least one of virtual reality, augmented reality, mixed reality and other technologies to provide an immersive and interactive experience for users. Extended reality devices include but are not limited to head-mounted XR headsets, motion controllers, force feedback devices, etc.

[0071] Taking the execution subject as a server as an example, the server can establish a communication connection with the first control device, and can also establish a communication connection with the extended reality device. Optionally, the server is used to specifically support and manage the extended reality device, and the server can be implemented as a cloud server or a local server of the platform to which the extended reality device is connected; and / or, the extended reality device can be implemented as a background server of an application installed and running in the device, such as: the extended reality device is installed with application store A, and when the extended reality device runs application store A, the background server of application store A is connected as the server for executing the device communication method.

[0072] In some embodiments, when the server can establish a communication connection with the first control device and can also establish a communication connection with the extended reality device, the first control device and the extended reality device can perform a short-term communication process with the server. For example, taking the communication process between the first control device and the server as an example, the first control device sends a communication request to the server. If the first control device receives a request confirmation feedback from the server, the first control device can send a communication message to the server, so that the server receives the communication message to realize the communication process between the first control device and the server; similarly, the extended reality device can also communicate and interact with the server in this way.

[0073] In some embodiments, considering that the amount of interactive processing data between the device and the server in the above short-term communication process is large, a long connection can be established between the device and the server through a long connection technology.

[0074] In principle, long connection technology refers to a technical method of establishing and maintaining a persistent network connection between a device and a server, rather than establishing a new connection every time a communication occurs. The main feature of a long connection is that once the connection is established, it remains open, allowing multiple data exchanges without the need to frequently establish and close connections, thereby reducing communication overhead and improving efficiency; long connections are suitable for applications that require continuous data exchange, such as real-time communications, instant messaging, online games, video streaming, etc.

[0075] Persistent connection technology can be implemented through a variety of protocols, including the Transmission Control Protocol (TCP), the TCP-based Internet Communication Protocol (WebSocketProtocol, WebSocket), the Quick User Datagram Protocol Connection (Quick UDP Internet Connection, QUIC) or at least one of other communication protocols that support persistent connections. Persistent connections can be established through the above protocols to provide some form of persistent connection.

[0076] Illustratively, a first long connection is established between the first control device and the server via the Internet using a long connection technology, and a second long connection is established between the extended reality device and the server via the Internet using a long connection technology.

[0077] In schematic form, the Internet is an infrastructure for information flow that can support both long-distance and short-distance communication services, with more diverse communication methods. Long connections are established through the Internet using long connection technology, that is, the connection between the device and the server in network communication remains valid for a certain period of time to reduce the frequent connection establishment and disconnection process.

[0078] The first long connection enables the first control device and the server to maintain a relatively stable communication process for a certain period of time, and the second long connection enables the extended reality device and the server to maintain a relatively stable Internet network for a certain period of time.

[0079] Optionally, the first control device and / or the extended reality device automatically establishes a long connection with the server when the device is turned on; or, the first control device and / or the extended reality device automatically establishes a long connection with the server when the device runs a specified application; or, the first control device and / or the extended reality device establishes a long connection with the server when the device receives a long connection operation, etc. The embodiments of the present application are not limited to this.

[0080] There is a pairing relationship between the first control device and the extended reality device.

[0081] Illustratively, a pairing relationship is a connection of mutual trust and secure communication established between devices in some way, through which the first control device and the extended reality device can recognize and authenticate each other during communication and implement encryption protection.

[0082] In some embodiments, a pairing relationship is established between devices logged in with the same account, such as: a first control device is logged in with a first account, and an extended reality device is logged in with a second account. If the first account and the second account are the same account logged in to different devices (such as the first account and the second account are A platform accounts logged in through the same mobile phone number, etc.), then the first control device can search for other devices logged in with the same account based on the logged in first account, such as other devices including extended reality devices logged in with the same account. Similarly, the extended reality device can also find other devices (such as the first control device) based on the way of logging in with the same account; the server can automatically establish a pairing relationship between different devices logged in with the same account based on this method, and can also establish a pairing relationship between devices based on the selection of device users (such as users of the first control device) on the premise of logging in with the same account, which is not limited here.

[0083] In some embodiments, a pairing relationship is established between the first control device and the extended reality device based on a historical connection relationship. Schematically, the historical connection relationship is a relationship in which a communication connection exists between the first control device and the extended reality device in a historical time period, such as: the first control device and the extended reality device have been in the same wireless fidelity (WiFi) environment in the past month, or the first control device and the extended reality device have been connected to Bluetooth in the past week, etc., then it is deemed that there is a historical connection relationship between the first control device and the extended reality device; if under the historical connection relationship, the first control device and the extended reality device both allow a pairing relationship for rapid communication based on the historical connection relationship in a future time period (such as under the historical connection relationship, the first control device allows the extended reality device to quickly communicate with it in the next week, and the extended reality device also allows the extended reality device to quickly communicate with it in the next week), then it is deemed that there is a pairing relationship between the first control device and the extended reality device under the historical connection relationship, etc.

[0084] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0085] Step 220: When the extended reality device executes a state change process related to the first application, receive real-time status information sent by the extended reality device through the second long connection.

[0086] Illustratively, the first application is any application related to the extended reality device. Optionally, the first application is an application that has been installed on the extended reality device; or, the first application is an application to be installed on the extended reality device, such as: the first application is any application that can be downloaded and listed in an application store installed on the extended reality device; or, the first application is an application whose installation package is stored in an external device connected to the extended reality device, such as: the external device is connected to a universal serial bus (USB) of the extended reality device by wire, in which files such as installation packages and images are pre-stored, and any installation package can be used as the first application that can be downloaded to the extended reality device, etc., which is not limited here.

[0087] Illustratively, the state change process is a change process of a state change condition related to the first application, and is used to characterize a change condition of the first application on the extended reality device.

[0088] Optionally, the state change process includes a program download process of the first application. Schematically, the first application is an application that has not been downloaded to the extended reality device. If the extended reality device is currently downloading the first application, the program download process is regarded as a state change process related to the first application.

[0089] Optionally, the state change process includes a program update process of the first application. Schematically, the first application is an application that has been installed on the extended reality device, and if the extended reality device is currently updating the first application (such as updating to the latest version, etc.), the program update process is regarded as a state change process related to the first application.

[0090] Optionally, the state change process includes a program running process of the first application. Schematically, the first application is an application that has been installed on the extended reality device. If the extended reality device is currently running the first application (e.g., the first application is a game application, and a virtual game is currently being performed through the game application), the program running process is regarded as a state change process related to the first application, etc. The above state change process is only an illustrative example and is not limited here.

[0091] In some embodiments, when the extended reality device executes a state change process related to the first application, the extended reality device generates state real-time information based on the state change process.

[0092] Schematically, the real-time status information is used to characterize the real-time changes in the status of the extended reality device when executing a state change process related to the first application; for example: for multiple timestamps of the state change process, state change results corresponding to the multiple timestamps are generated, and the real-time status information is obtained by combining the multiple state change results, wherein the multiple timestamps are usually selected based on preset rules, such as the timestamp interval is one microsecond, etc., which is not limited here.

[0093] Optionally, when the state change process is a state download process, the extended reality device can generate a program download progress as real-time status information based on the state download process, and the program download progress is used to reflect the download status of the extended reality device at multiple timestamps when downloading the first application; and / or, the extended reality device can generate a program download result as real-time status information based on the state update process, and the program download result is used to reflect the phased situation when the extended reality device downloads the first application, such as the program download result is "downloading", "downloading completed", etc.; and / or, the extended reality device can generate a download completion countdown as real-time status information based on the state update process, and the download completion countdown is used to reflect the predicted countdown of the extended reality device completing the download process of the first application; the extended reality device can generate a program download rate as real-time status information based on the state update process, and the program download rate is used to reflect the network rate when the extended reality device downloads the first application, etc., which is not limited here.

[0094] Optionally, when the state change process is a state update process, the extended reality device can generate a program update progress as real-time status information based on the state update process, and the program update progress is used to reflect the download status at multiple timestamps when the extended reality device updates the first application; and / or, the extended reality device can generate a program update result as real-time status information based on the state update process, and the program update result is used to reflect the phased situation when the extended reality device updates the first application, such as the program update result is "updating", "update completed", etc.; and / or, the extended reality device can generate an update completion countdown as real-time status information based on the state update process, and the update completion countdown is used to reflect the predicted countdown of the extended reality device completing the update process of the first application; the extended reality device can generate a program update rate as real-time status information based on the state update process, and the program update rate is used to reflect the network rate when the extended reality device updates the first application, etc., which is not limited here.

[0095] Optionally, when the state change process is a program running process, the extended reality device can generate running screen information as real-time status information based on the program running process, and the running screen information is used to reflect the simulated display screen when the extended reality device runs the first application; and / or, the extended reality device can generate running audio information as real-time status information based on the program running process, and the running audio information is used to reflect the audio played when the extended reality device runs the first application; and / or, the extended reality device can generate running status information as real-time status information based on the program running process, and the running status information is used to reflect the state of the extended reality device running the first application, such as the running status information is "normal operation", "operation failure", etc., which is not limited here.

[0096] In some embodiments, the extended reality device sends the real-time status information generated in real time to the server through the second long connection established with the server.

[0097] Illustratively, the extended reality device generates real-time status information in real time and sends it to the server through the second long connection in real time; or, the extended reality device generates real-time status information in real time and periodically sends it to the server through the second long connection, such as the extended reality device sends the real-time status information to the server through the second long connection every preset time period, and the preset time period is a pre-set time period, such as 1 second, 0.1 second, etc., which is not limited here.

[0098] Step 230: Send real-time status information to the first control device through the first persistent connection based on the pairing relationship.

[0099] Illustratively, when a first long connection is established between the first control device and the server, if the server receives real-time status information sent by the extended reality device, the server can send the real-time status information to the first control device through the first long connection, so that the first control device with a paired relationship can be aware of the situation related to the extended reality device executing the first application.

[0100] In some embodiments, the real-time status information generated by the extended reality device includes an information receiving entity. Therefore, the server can determine the first control device as the information receiving entity by identifying the real-time status information, and then send the real-time status information to the first control device when a first long connection is established between the server and the first control device.

[0101] In some embodiments, the real-time status information includes instruction reception parameters (indicating an information receiving subject) and program execution information, and the program execution information is the situation in the real-time status information that characterizes the situation when the extended reality device executes the first application program; when the server receives the real-time status information and parses to determine the information receiving subject and the program execution information, the server can determine the first control device based on the information receiving subject, and thus, when a first long connection is established between the server and the first control device, the program execution information in the real-time status information is sent to the first control device. At this time, there is no need to send the instruction reception parameters characterizing the information receiving subject to the first control device, thereby reducing the amount of information transmission and improving transmission efficiency.

[0102] The real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0103] Schematically, the real-time status information reflects the real-time situation when the extended reality device executes the first application. If the server feeds back the real-time status information to the first control device, the first control device can understand the state change process of the extended reality device with which it is paired to execute the first application based on the real-time status information; the first control device can render and display the received real-time status information on the device screen, thereby timely representing the execution status of the extended reality device with which it is paired for the first application.

[0104] In some embodiments, when the status real-time information does not include an information receiving subject, the server can receive the status real-time information sent by the extended reality device through the second long connection, but does not send the status real-time information to the first control device.

[0105] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0106] To sum up, the long connection communication method makes device interaction more stable, which helps to ensure that the extended reality device sends real-time status information to the first control device more promptly and accurately, reduces network burden, saves resource consumption, and makes device interaction suitable for application scenarios with strong real-time and frequent interactions. It can also be applied to remote operation scenarios where the distance between the first control device and the extended reality device is far, thereby improving the application range of device interaction under the device pairing relationship.

[0107] In an optional embodiment, the extended reality device may execute a state change process related to the first application and generate real-time state information based on the state adjustment operation received by itself, and may also execute a state change process related to the first application and generate real-time state information based on the first control instruction targeted by the first control operation. Figure 3 As shown above Figure 2 The illustrated step 220 may also be implemented as the following step 310 or steps 320 to 340 .

[0108] Step 310: When the extended reality device performs a state change process related to the first application based on the state adjustment operation, receive real-time state information sent by the extended reality device through the second long connection.

[0109] The state adjustment operation is an operation received by the extended reality device and executed on the first application.

[0110] Illustratively, the state adjustment operation is an operation related to the first application program received by the extended reality device, and the state adjustment operation is used to trigger the extended reality device to execute a state change process related to the first application program. Optionally, the state adjustment operation includes at least one of an eye movement operation (determining the operation position and executing the state adjustment operation as the user's eyes move), a gesture trigger operation, a voice trigger operation, and the like, which are not limited here.

[0111] Optionally, when the first application is an application not installed on the extended reality device, the state adjustment operation can be implemented as a program download operation for installing the first application on the extended reality device. For example, the extended reality device is installed with an application store, and a trigger operation for a program download control corresponding to the first application in the application store can be received as a program download operation to trigger the extended reality device to execute a program download process of downloading the first application as a state change process.

[0112] Optionally, when the first application is an application installed on the extended reality device, the state adjustment operation can be implemented as a program update operation for updating the first application on the extended reality device. For example, a program update control corresponding to the first application is used as a program update operation based on a trigger operation on the program update control to trigger the extended reality device to execute a program update process of updating the first application as a state change process.

[0113] Optionally, when the first application is an application installed on the extended reality device, the state adjustment operation can also be implemented as a program running operation for running the first application on the extended reality device. For example, a program start operation for the first application is used as a program running operation, or a function trigger operation for a program function in the first application is used as a program running operation, etc., to trigger the extended reality device to execute the program running process of running the first application as a state change process, etc. The program running process includes at least one stage process of multiple running stages in the program running, which is not limited here.

[0114] In some embodiments, when the extended reality device performs a state change process based on the state adjustment operation, the extended reality device generates state real-time information based on the state change process and sends the state real-time information to the server through the second long connection.

[0115] Optionally, the extended reality device generates program execution information based on the executed first application, and uses the program execution information as the real-time status information; or, the extended reality device generates program execution information based on the executed first application, and also generates instruction sending parameters indicating the information sending subject based on the pairing relationship, and uses the program execution information and the instruction sending parameters together as the real-time status information, etc., which are not limited here.

[0116] Step 320: Receive a first control instruction sent by a first control device.

[0117] In some embodiments, in addition to executing a state change process related to the first application based on the state adjustment operation received by the extended reality device, the extended reality device can also execute a controlled state change process based on a first control instruction of a first control device with which it has a paired relationship.

[0118] Optionally, if it is desired to control the extended reality device through the first control device, the first control device may send the first control instruction to the server through the first long connection.

[0119] The first control instruction includes control execution information corresponding to the first application and a second device identifier corresponding to the extended reality device.

[0120] Illustratively, the control execution information is the content of the state change process related to the execution of the first application by the first control device, and the second device identifier is the unique indication corresponding to the extended reality device. Therefore, the first control instruction that combines the control execution information and the second device identifier can control the execution of the state change process related to the first application by the extended reality device.

[0121] Step 330: Send a first control instruction to the extended reality device based on the second device identifier.

[0122] Illustratively, after receiving the first control instruction, the server can obtain the second device identifier and control execution information therein by parsing the first control instruction; the server determines the extended reality device indicated to be controlled by the first control device based on the parsed second device identifier, and thereby sends the first control instruction to the extended reality device in a targeted manner through the second long connection.

[0123] Optionally, the server can also determine the extended reality device indicated to be controlled by the first control device based on the resolved second device identifier, and then send the control execution information in the first control instruction to the extended reality device through the second long connection to reduce the amount of data transmitted through the second long connection, which is not limited here.

[0124] Among them, the first control instruction is used to control the extended reality device to execute a state change process related to the first application based on the program execution information.

[0125] Illustratively, a first control instruction is sent to the extended reality device to control it to execute a state change process related to the first application based on the program execution information.

[0126] In an optional embodiment, the extended reality device controlled by the first control device is determined based on the second device identifier.

[0127] Illustratively, the server parses the first control instruction to determine the second device identifier, thereby specifically determining the extended reality device based on the second device identifier, so as to execute the information sending process to the extended reality device.

[0128] In an optional embodiment, the instruction sending level in the first control instruction is obtained.

[0129] The instruction sending level is used to represent the priority requirement for sending the first control instruction.

[0130] Schematically, the instruction sending level is a sending level pre-set for the first control instruction. The sending level is the degree of measures and guarantees taken during the information transmission process to ensure that information can correctly and safely reach the device. It is usually used to describe whether information can be ensured to arrive in order, whether it can be correctly transmitted, and whether lost data can be recovered during network transmission; therefore, the instruction sending level of the first control instruction is used to describe the degree of guarantee taken to ensure that the first control instruction can be correctly sent to the extended reality device.

[0131] Optionally, the sending level can also be called the message reliability level, and the sending level of the first control instruction is called the instruction sending level. The higher the instruction sending level, the higher the priority requirement of the first control instruction, and therefore the more accurately the first control instruction needs to be sent to the extended reality device.

[0132] In some embodiments, the sending level of the control instruction transmitted in the communication environment is determined according to the communication environment between the devices.

[0133] Illustratively, if the first control device and the extended reality device are in a communication environment Q with higher transmission accuracy, when transmitting the first control instruction in the communication environment Q, a pre-set higher sending level can be used as the instruction sending level of the first control instruction.

[0134] In some embodiments, the sending level corresponding to the control instruction is usually determined according to the importance of the transmitted control instruction.

[0135] Schematically, the importance is used to characterize the timeliness condition of sending the control instruction to the device. For example, the higher the timeliness of the control instruction, the higher the importance, and therefore the higher the sending level corresponding to the control instruction. For example, if the control instruction sent by the first control device to the extended reality device needs to be sent to the extended reality device in a timely manner (such as the download progress needs to be displayed in time), then the control instruction is more important, and a higher sending level is set in advance for the first control instruction as the instruction sending level, etc.

[0136] In an optional embodiment, a first control instruction is sent to the extended reality device based on an instruction sending level.

[0137] Illustratively, after the server determines the instruction sending level based on the first control instruction analysis, it executes the sending process of the first control instruction according to the instruction sending level, so as to realize the instruction sending process in a manner that strikes a balance between performance, delay and reliability.

[0138] Optionally, common sending levels include at least one of multiple forms such as at most once, as far as possible once, at least once and exactly once.

[0139] Among them, at most once can be expressed as "AT_MOST_ONCE", which can be called the first sending level; in this mode, information (such as the first control instruction) will be sent at most once, and if the information sending fails, it will not be retransmitted; this mode is suitable for applications that are not very sensitive to information loss but do not want to receive repeated information, such as some low-latency status update information, etc.; its characteristic is that the information sender does not make any subsequent confirmation after sending the information, and there is no retransmission mechanism. If the information is lost, it will never reach the receiving entity, ensuring that the receiving entity will not receive duplicate information.

[0140] Among them, "TRY_BEST_ONCE" can be expressed as "TRY_BEST_ONCE", which can be called the second sending level; in this mode, information (such as the first control instruction) is sent only once, but if the information fails to be sent, it can also be retransmitted.

[0141] Among them, at least once can be expressed as "AT_LEAST_ONCE", which can be called the third sending level; in this mode, the information (such as the first control instruction) will be sent at least once, and it will be ensured that the information is successfully received at least once. If the information fails to be sent, the sending entity will retry until the confirmation information is received; this mode is suitable for scenarios where it is necessary to ensure that the information is successfully delivered, and the information may be received repeatedly, such as: some data synchronization, database update, etc.; its characteristic is that the information will be retried until the receiving entity confirms that the reception is successful. Since this mode may cause message duplication, there is usually an additional deduplication mechanism to handle duplicate information.

[0142] Among them, exactly once can be expressed as "EXACTLY_ONCE", which can be called the fourth sending level; in this mode, information (such as the first control instruction) is guaranteed to be sent only once and will not be lost, and a confirmation mechanism and a deduplication mechanism are usually required to avoid sending duplicate information; this mode is suitable for applications that require high reliability and information cannot be repeated, such as: payment scenarios, order processing scenarios, etc.; its characteristic is that it ensures that information is only sent once and successfully reaches the receiving entity through confirmation, serial number and other mechanisms; this mode usually belongs to the most stringent sending level (message reliability level), so it needs to be implemented through complex protocol design.

[0143] It is worth noting that the above sending levels are only illustrative examples. There may also be other sending levels such as "No Reliability Level" (which means that information transmission does not require any reliability expression. Once the information is sent, the sending entity does not care whether the receiving entity successfully receives the information, and there is no need for the receiving entity to send any confirmation information, such as random advertising recommendation scenarios, etc.), which are not limited here.

[0144] Different sending levels have different meanings. Based on the first control instruction, a unique sending level can be determined as the instruction sending level. For example, if the instruction sending level of the first control instruction is "at least once", it means that the first control instruction is sent at least once. When one sending fails, the first control instruction can be sent repeatedly.

[0145] In some embodiments, when the instruction sending level is at a first preset level, a first control instruction is sent to the extended reality device once.

[0146] Illustratively, the first preset level is a pre-set sending level. When the server parses that the instruction sending level is the first preset level, the server can send a first control instruction to the extended reality device once based on the first preset level, that is, the first preset level is a sending level that limits the sending of a single message (first control instruction), that is, the first preset level is the above-mentioned first sending level.

[0147] In some embodiments, when the instruction sending level is at a second preset level, the first control instruction is cached in a database; the first control instruction is called from the database, and the first control instruction is sent to the extended reality device at least once.

[0148] Illustratively, the second preset level is a pre-set sending level that is different from the first preset level. When the server parses that the instruction sending level is the second preset level, the server can send the first control instruction to the extended reality device at least once based on the second preset level, that is, the server can send a single first control instruction to the extended reality device based on the second preset level, and can also send multiple first control instructions to the extended reality device based on the second preset level.

[0149] Illustratively, the second preset level includes at least one of the second sending level, the third sending level, and the fourth sending level; for example, when the instruction sending level is at the second sending level, the server may cache the first control instruction to a database, etc.

[0150] Optionally, when the server parses that the instruction sending level is at the second preset level, the server can first cache the first control instruction to be forwarded to the extended reality device into a database (DateBase, DB); wherein the database is a component set at the server for storing data to be forwarded to other devices.

[0151] Optionally, the server calls the first control instruction from the database and sends the first control instruction to the extended reality device at least once.

[0152] Schematically, multiple data are cached in the database, including a first control instruction; the multiple data correspond to cache timestamps respectively, and the server calls the multiple data in the order of the cache timestamps to forward them to the corresponding receiving entity (receiving device); when the server calls the first control instruction in sequence based on the cache timestamp, the server sends the first control instruction to the extended reality device at least once.

[0153] Schematically, multiple data are cached in the database, including a first control instruction; the multiple data correspond to sending priorities respectively, and the server calls the multiple data based on the sending priorities to forward them to the corresponding receiving entity (receiving device); when the server calls the first control instruction based on the sending priority, the server sends the first control instruction to the extended reality device at least once.

[0154] In some embodiments, when the server parses that the instruction sending level is at a second preset level, the server may first send a first control instruction; if the server fails to successfully send the first control instruction to the extended reality device (such as not receiving confirmation information for the first control instruction fed back by the extended reality device), the server will cache the first control instruction to be repeatedly forwarded to the extended reality device to the database, so as to repeatedly send the first control instruction through the database calling process.

[0155] Optionally, when the first control instruction is repeatedly sent through the calling process of the database, the first control instruction may also be sent by calling in the order of the cache timestamps and / or the sending priority, which is not limited here.

[0156] In some embodiments, instruction retransmission information in the first control instruction is obtained.

[0157] Optionally, when the server parses and finds that the instruction sending level of the first control instruction is at a second preset level, the server may send the first control instruction of the second preset level at least once until the first control instruction is sent to the extended reality device.

[0158] Illustratively, if the server successfully receives confirmation information fed back by the extended reality device after sending the first control instruction for the first time, the first control instruction may not be sent repeatedly; if the server fails to successfully receive confirmation information fed back by the extended reality device after sending the first control instruction for the first time, the first control instruction needs to be sent a second time. If confirmation information is received after the second sending, the first control instruction may not be sent repeatedly. Alternatively, if confirmation information is still not received after the second sending, the server continues to execute the process of sending the first control instruction for the third time, etc., which is not limited here.

[0159] Optionally, when the server parses that the instruction sending level of the first control instruction is at a second preset level, the server can also parse and determine instruction retransmission information from the first control instruction, where the instruction retransmission information is used to represent the number of repeated transmissions when the first control instruction is not successfully sent.

[0160] In an illustrative manner, the number of times the first control instruction is repeatedly sent to the extended reality device is constrained by the instruction retransmission information. Optionally, when the server parses and finds that the instruction sending level of the first control instruction is at the second preset level, the server can directly determine the instruction retransmission information from the first control instruction, and can also parse and determine the instruction retransmission information from the first control instruction after the first sending of the first control instruction fails (such as not receiving the confirmation information fed back by the extended reality device), etc., which is not limited here.

[0161] For example: the second preset level is the above-mentioned third sending level (at least once), and the instruction retransmission information determined by parsing the first control instruction is 5 times. Then, after the server fails to send the first control instruction, the server can resend the first control instruction to the extended reality device up to 5 times. Even if it still fails after 5 times, the server does not need to repeatedly send the first control instruction to the extended reality device.

[0162] Optionally, when repeatedly sending the first control instruction to the extended reality device based on the instruction retransmission information, the first control instruction of the instruction retransmission information can be sent to the extended reality device based on a preset interval time period. If the confirmation information fed back by the extended reality device for the first control instruction is not successfully received, the first control instruction is sent to the extended reality device once every 10 seconds, and is repeated 5 times in total; the first control instruction can also be called to repeatedly send the first control instruction based on the cache timestamp and / or sending priority when the first control instruction is cached in the database; the first control instruction can also be called to repeatedly send based on the repeated sending priority of the first control instruction (such as the priority of repeated sending predetermined by the first control device when the first control instruction is sent; or, the priority of repeated sending data automatically determined by the server based on the status of multiple data in the database, etc.), which is not limited here.

[0163] In some embodiments, the first control instruction is called from the database, and the first control instruction is sent once to the extended reality device.

[0164] Illustratively, the server calls the first control instruction from the database, and the server sends the first control instruction to the extended reality device.

[0165] Optionally, a single first control instruction is first sent to the extended reality device; if the first control instruction is not successfully sent to the extended reality device, the first control instruction is repeatedly sent to the extended reality device based on the number of repeated transmissions indicated by the instruction retransmission information.

[0166] Illustratively, when the server does not receive confirmation information from the extended reality device regarding the first control instruction, it is deemed that the first control instruction has not been successfully sent to the extended reality device. Therefore, the server can send at least one first control instruction to the extended reality device subject to a constraint on the number of repeated transmissions based on the instruction retransmission information determined by the analysis.

[0167] Since the number of repeated transmissions indicated by the instruction retransmission information is a preset constraint number for repeated transmissions when the extended reality device is not successfully sent, within the constraint of the number of repeated transmissions, when the first control instruction is successfully sent, there is no need to repeatedly send the first control instruction again; if the first control instruction is not successfully sent within the constraint of the number of repeated transmissions, the first control instruction continues to be retransmitted, which is not limited here.

[0168] In an optional embodiment, when multiple control instructions are received, the instruction sending order is determined based on the sending priorities respectively corresponding to the multiple control instructions.

[0169] Illustratively, if the server receives multiple control instructions, including a first control instruction, when the server needs to send multiple control instructions to corresponding devices (the first control instruction needs to be sent to an extended reality device, the second control instruction needs to be sent to a first terminal, etc.), the server can determine the sending priorities corresponding to the multiple control instructions.

[0170] The sending priority focuses on measuring the relationship between different control instructions. Therefore, the sending priority is used to characterize the importance of data (such as control instructions) relative to other data during transmission, processing or scheduling; the sending priority determines the data that should be transmitted or processed first when the network is congested, resources are limited or the system load is high. The sending priority can ensure that critical, urgent or important data (data with higher priority) can be processed or transmitted faster, while unimportant or non-urgent data (data with lower priority) can be delayed in processing or transmission.

[0171] Schematically, different from the above instruction sending level which focuses on expressing the control instruction itself, the sending priority focuses on comparing multiple control instructions, and the two are different concepts. Optionally, the sending priority is also pre-set information, and the sending priority is used to flexibly deal with the process of how to reasonably send control instructions and process control instructions (such as parsing control instructions) when there are multiple control instructions.

[0172] In an optional embodiment, multiple control instructions are sent to the extended reality device in a command sending order.

[0173] Optionally, the instruction sending order is determined based on the sending priorities respectively corresponding to the multiple control instructions, and the instruction sending order is the order of the multiple control instructions.

[0174] Schematically, the sending priority is preset to two levels, high priority and low priority, and at least one control instruction with a high priority is sent first, followed by at least one control instruction with a low priority, etc., which is not limited here. The sending order of at least one control instruction with a high priority can be determined randomly, or based on the receiving timestamp of the control instruction received by the server, or based on the cache timestamp of the control instruction cached in the database by the server, etc. The sending order of at least one control instruction with a low priority can also be determined based on this method, which is not limited here.

[0175] Illustratively, the sending priority can also be preset to multiple more detailed levels, such as priority 1 is higher than priority 2 which is higher than priority 3, etc., so as to determine a more accurate instruction sending order based on the more detailed priorities and execute the sending process based on this, which is not limited here.

[0176] In some embodiments, the first control instruction includes at least one of the above-mentioned control execution information, instruction sending parameters, instruction receiving parameters, instruction sending level and instruction retransmission information.

[0177] The instruction sending parameters include a first device identifier and a first account identifier corresponding to the first control device, and the first account identifier is the identifier of the first account logged in to the first control device; the instruction receiving parameters include a second device identifier and a second account identifier corresponding to the extended reality device, and the second account identifier is the identifier of the second account logged in to the extended reality device; the instruction sending level is used to characterize the priority requirement for sending the first control instruction; the instruction retransmission information is used to characterize the number of repeated transmissions when the first control instruction is not successfully sent; it is not limited here.

[0178] Step 340: When the extended reality device executes a state change process related to the first application based on the first control instruction, receive real-time status information sent by the extended reality device through the second long connection.

[0179] Illustratively, the extended reality device receives the first control instruction sent by the first control device and forwarded by the server through the second long connection, so that the extended reality device executes a state change process related to the first application based on the first control instruction.

[0180] Optionally, the extended reality device parses the first control instruction to determine control execution information, where the control execution information is content used to control the execution of a state change process related to the first application, so that the extended reality device executes the state change process related to the first application based on the control execution information.

[0181] Optionally, the server parses the first control instruction to determine control execution information, and then sends the control execution information to the extended reality device. The extended reality device can more efficiently execute the state change process related to the first application based on the control execution information.

[0182] In some embodiments, when the extended reality device executes a state change process related to the first application, it generates real-time status information representing the execution status and sends the real-time status information to the server through the second long connection, so that the server receives the real-time status information sent by the extended reality device through the second long connection.

[0183] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0184] To sum up, the long connection communication method makes device interaction more stable, which helps to ensure that the extended reality device sends real-time status information to the first control device more promptly and accurately, reduces network burden, saves resource consumption, and makes device interaction suitable for application scenarios with strong real-time and frequent interactions. It can also be applied to remote operation scenarios where the distance between the first control device and the extended reality device is far, thereby improving the application range of device interaction under the device pairing relationship.

[0185] In an embodiment of the present application, the content of executing a state change process related to a first application and generating real-time status information is introduced. The extended reality device can execute the process based on the state adjustment operation received by itself, or based on the control of the first control device over it, thereby enriching the way in which the extended reality device executes the state change process related to the first application; based on the long distance between the first control device and the extended reality device, the first control device can still control the extended reality device in a targeted manner through pairing relationships such as logging in to the same account, thereby improving the flexibility of device interaction and expanding the application scenarios of device interaction. The first control device can also receive real-time status information of the extended reality device to understand changes in the interactive device more promptly, thereby improving the accuracy of device interaction.

[0186] In an optional embodiment, a first long connection is established between the server and the first control device through a connection establishment request, and a second long connection is established between the server and the extended reality device through a second connection establishment request. Figure 4 As shown above Figure 2 The illustrated embodiment can also be implemented as the following steps 410 to 450; wherein Figure 2 Step 210 in the method can also be implemented as steps 410 to 430 as follows.

[0187] Step 410: Receive a first connection establishment request sent by a first control device through the Internet.

[0188] Illustratively, before the first control device establishes the first persistent connection with the server, the first control device may send a first connection establishment request to the server, where the first connection establishment request is request information for the first control device to establish the first persistent connection with the server.

[0189] The first connection establishment request includes a first device identifier corresponding to the first control device and a first account identifier of a first account logged in to the first control device.

[0190] Illustratively, in order for the server to determine the request sender after receiving the first connection request, the first connection request includes a first device identifier corresponding to the first control device, and the first device identifier is an identifier that uniquely represents the first control device; in addition, the first connection request also includes a first account identifier of a first account logged in to the first control device, and the first account identifier is used to uniquely indicate the first account.

[0191] Optionally, the first account is a device account logged in to the first control device, such as the first control device is a device with an I operating system, and the device account is an account logged in to the I operating system; or, the first account is a program account logged in to a preset application of the first control device, such as an account logged in to an application store, etc., which is not limited here.

[0192] In an optional embodiment, when the first connection request meets the preset connection conditions, based on the first program identifier in the first connection request, indicating the second application, a first long connection with a first connection duration is established between the first control device, and the second application is an application installed on the first control device with a connection requirement; or, when the first connection request meets the preset connection conditions, based on the first program identifier in the first connection request, indicating the third application, a first long connection with a second connection duration is established between the first control device and the first control device, and the third application is an application installed on the first control device with a connection requirement.

[0193] Among them, the third application is different from the second application, and the first connection duration is different from the second connection duration. For example: if the application (second application) with a long connection requirement is an application store, a first long connection of 10 minutes (first connection duration) is established; if the application (third application) with a long connection requirement is a reading software, a first long connection of 2 minutes (second connection duration) is established, etc.; different connection durations are used to achieve flexible application of long connection technology, avoid the problem of excessive data transmission due to long long connection durations, and ensure data transmission security and rationality.

[0194] Step 420: Receive a second connection establishment request sent by the extended reality device through the Internet.

[0195] Illustratively, before the extended reality device has not established a second long connection with the server, the extended reality device may send a second connection establishment request to the server, where the second connection establishment request is request information for the extended reality device to establish a second long connection with the server.

[0196] The second connection establishment request includes a second device identifier corresponding to the extended reality device and a second account identifier of a second account logged in to the extended reality device.

[0197] Illustratively, in order to enable the server to determine the request sender after receiving the second connection request, the second connection request includes a second device identifier corresponding to the extended reality device, and the second device identifier is an identifier that uniquely represents the extended reality device; in addition, the second connection request also includes a second account identifier of a second account logged in to the extended reality device, and the second account identifier is used to uniquely indicate the second account.

[0198] Optionally, the second account is a device account logged in to the extended reality device, such as the extended reality device is an XR headset running an I operating system, and the second account is an account logged in to the I operating system; or, the second account is a program account logged in to a preset application on the first control device, such as an account logged in to an app store, etc., which is not limited here.

[0199] Step 430: When the first connection request meets the preset connection condition, a first long connection is established with the first control device; when the second connection request meets the preset connection condition, a second long connection is established with the extended reality device.

[0200] Schematically, the preset connection condition is a pre-set condition that needs to be met for the server and the device to establish a long connection. Optionally, the preset connection condition includes at least one of multiple conditions such as identity authentication conditions, protocol type conditions, and device compatibility conditions. The identity authentication condition indicates that the request subject (device) sending the connection request needs to provide valid verification information; the protocol type condition indicates that the request subject (device) sending the connection request needs to comply with certain preset version protocols, etc.; the device compatibility condition includes at least one of the hardware compatibility, bandwidth compatibility, and performance calculation conditions between the request subject (device) sending the connection request and the server. The preset connection condition is not limited here.

[0201] In an optional embodiment, the first connection establishment request is parsed to obtain the first device identifier and the first account identifier.

[0202] Illustratively, the server obtains the first device identifier and the first account identifier by parsing the first connection establishment request.

[0203] Optionally, when the second application installed on the first control device requests long connection communication, the first connection request may also include a corresponding first program identifier. In this case, the first account logged in on the first control device may be a program account logged in on the second application. In addition, the first connection request may also include a device type of the first control device, such as a device type indicating that the first control device is a mobile terminal, a computer terminal, a head-mounted display device, etc., which is not limited here. In addition, the first connection request may also include a user ticket, which is used to implement identity authentication and authorization to ensure that only devices that meet the requirements can establish a long connection.

[0204] In an optional embodiment, when the first device identifier and the first account identifier meet the preset connection establishment condition, the first connection establishment information in the first connection establishment request is updated to the database, and the first long connection with the first control device is established.

[0205] Optionally, the server parses the first connection establishment request to obtain the first device identifier and the first account identifier therein, and may also obtain at least one of the above-mentioned first program identifier, device type and user ticket; at least one of the above-mentioned first device identifier, first account identifier, first program identifier, device type and user ticket may be collectively referred to as the first connection establishment information.

[0206] Optionally, the first connection information is encoded through serialization processing to obtain a first character string. After receiving the first connection request, the server parses the first connection request and decodes to determine the first connection information, and then verifies whether the first device identifier, the first account identifier, the first program identifier and the device type meet the requirements, and then verifies whether the user ticket is valid, that is, determines whether the preset connection conditions are met.

[0207] Illustratively, if the preset connection establishment condition is met, the server may update the first connection establishment information into a database for caching data, thereby completing the process of establishing the first long connection with the first control device.

[0208] In an optional embodiment, when the second device identifier and the second account identifier meet the preset connection conditions, the second connection information in the second connection request is updated to the database to establish a second long connection with the extended reality device.

[0209] Optionally, the server parses the second connection request to obtain the second device identifier and the second account identifier therein, and may also obtain at least one of the second program identifier (the program identifier of the application on the extended reality device requesting long connection communication), the device type and the user ticket sent by the extended reality device; at least one of the above-mentioned second device identifier, second account identifier, second program identifier, device type and user ticket may be collectively referred to as the second connection information.

[0210] Optionally, the second connection information is encoded through serialization processing to obtain a second character string. After receiving the second connection request, the server parses the second connection request and decodes to determine the second connection information, and then verifies whether the second device identifier, second account identifier, second program identifier and device type meet the requirements, and then verifies whether the user ticket is valid, that is, determines whether it meets the preset connection conditions.

[0211] Illustratively, if the preset connection establishment conditions are met, the server may update the second connection establishment information to a database for caching data, thereby completing the process of establishing a second long connection with the extended reality device.

[0212] Step 440: When the extended reality device executes a state change process related to the first application, receive real-time status information sent by the extended reality device through the second long connection.

[0213] Illustratively, the first application and the second application requesting long connection communication are usually different, such as: the second application is an application store installed on the first control device, and the first application is an application requested to be downloaded in the application store, etc., which is not limited here.

[0214] Illustratively, when the extended reality device executes a state change process related to the first application, the server receives real-time status information sent by the extended reality device through the second long connection, so that the server determines the program execution status of the extended reality device.

[0215] In an optional embodiment, when the extended reality device executes a program download process for the first application, the program download progress sent by the extended reality device through the second long connection is received as the real-time status information, and the status change process includes the program download process.

[0216] Illustratively, when the state change process is implemented as a program download process of the first application, the extended reality device can generate a program download progress in real time based on the program download process, and send the program download progress as real-time status information to the server; the extended reality device can also display the program download progress on the local end in real time.

[0217] In an optional embodiment, when the extended reality device executes a program update process for the first application, the program update progress sent by the extended reality device through the second long connection is received as the real-time status information, and the state change process includes the program update process.

[0218] Illustratively, when the state change process is implemented as a program update process of the first application, the extended reality device can generate a program update progress in real time based on the program update process, and send the program update progress as real-time status information to the server; the extended reality device can also display the program update progress on the local end in real time.

[0219] In an optional embodiment, when the extended reality device executes a program running adjustment process for the first application, the running screen data sent by the extended device through the second long connection is received as the real-time status information, and the state change process includes the program running adjustment process.

[0220] Illustratively, when the state change process is implemented as the program running process of the first application, the extended reality device can generate running screen data in real time based on the program running process, and send the running screen data to the server as real-time status information; the extended reality device can also render and display the running screen data on the local end in real time.

[0221] Step 450: Send real-time status information to the first control device through the first long connection.

[0222] The real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0223] In an optional embodiment, a first device identifier in the status real-time information is acquired, where the first device identifier is a device identifier of a first control device; and the status real-time information is sent to the first control device based on the first device identifier.

[0224] In an exemplary embodiment, the server parses the real-time status information, which includes the first device identifier; the server determines the first control device based on the first device identifier, and sends the real-time status information to the first control device. The first device identifier is used to establish a pairing relationship between the first control device and the extended reality device.

[0225] Optionally, the extended reality device generates real-time status information including the first device identifier when executing a state change process related to the first application based on a paired device selection operation, and the paired device selection operation is used to select the first control device from at least one device logged in with the same account. For example: the extended reality device receives a paired device selection operation showing at least one device logged in with the same account, and if a selection operation for the first control device is received, when executing a state change process related to the first application, the extended reality device generates real-time status information including the first device identifier corresponding to the first control device, so that when the real-time status information is sent to the server, the server establishes a pairing relationship between the first control device and the extended reality device based on the first device identifier; so that the real-time status information is sent to the first control device in a targeted manner through the pairing relationship.

[0226] In an optional embodiment, based on a pre-established pairing relationship between the first control device and the extended reality device, real-time status information is sent to the first control device via the first persistent connection.

[0227] Illustratively, the pairing relationship is achieved by logging in the same account to the first control device and the extended reality device. The extended reality device can then search for the first control device logged in with the same account, that is, the first account and the second account are the same, such as the first account is an account logged in to an application store installed on the first control device, and the second account is the same account logged in to the application store installed on the extended reality device, etc., so that the extended reality device can send pairing information to the server through the second long connection to pre-establish a pairing relationship between the first control device and the extended reality device; if the extended reality device then sends real-time status information to the server, the server sends the real-time status information to the first control device through the first long connection based on the pre-established pairing relationship, thereby achieving the purpose of information sending.

[0228] Optionally, when the extended reality device executes a state change process related to the first application based on the first control instruction sent by the first control device, the server establishes a pairing relationship between the first control device and the extended reality device; if the extended reality device subsequently sends real-time status information to the server, the server sends the real-time status information to the first control device based on the pre-established pairing relationship, etc., which is not limited here.

[0229] like Figure 5 As shown, taking the extended reality device as an XR head display, the first control device as a mobile terminal, and the real-time status information as the program download progress as an example, the display is as follows Figure 5 Schematic diagram shown. When the XR head display is downloading the first application 510 (such as a game application), the XR head display projects and displays an interface 520, which includes a program download progress 521. The program download progress 521 is used to represent the real-time progress of the program download. For example, if the download progress is 24% and the network speed is 11MB / s, at least one of the download progress and the network speed is used as the program download progress 521; when the XR head display and the mobile terminal are logged in with the same account (such as the first account logged in to the application store installed in the mobile terminal is the same as the second account logged in to the application store installed in the XR head display), the XR head display can send the program download progress 521 to the server through the second long connection, so that the server sends the program download progress 521 to the mobile terminal through the first long connection, thereby displaying an interface 530 on the mobile terminal, in which there are the introduction content of the first application 510 and the corresponding program download progress 521, such as only showing 24%, or only showing 11MB / s, or 24% and 11MB / s, which are not limited here.

[0230] In an optional embodiment, in response to the number of real-time status information sent by the first control device reaching a preset number threshold, the second long connection with the extended reality device is disconnected.

[0231] Illustratively, the preset number threshold is a pre-set threshold, such as the preset number threshold is 10; if the server receives 10 status real-time information sent by the first control device, the second long connection with the extended reality device is automatically disconnected.

[0232] In an optional embodiment, in response to receiving a first number of status real-time information among a plurality of status real-time information sent by the first control device that does not meet the information sending requirements, the second long connection with the extended reality device is disconnected.

[0233] Illustratively, the information sending requirement is a pre-set condition that the status real-time information cannot be accurately sent to other devices, or the status real-time information is not received completely. For example, the first number is preset to 5. If the server receives 5 status real-time messages that do not meet the information sending requirement (such as receiving 5 status real-time messages whose information content cannot be parsed), the second long connection with the extended reality device is automatically disconnected.

[0234] In an optional embodiment, in response to the connection duration of the second long connection reaching a preset duration threshold, the second long connection with the extended reality device is disconnected.

[0235] Illustratively, the preset duration threshold is a pre-set duration threshold, such as 10 seconds. When the connection duration between the server and the extended reality device accumulates to 10 seconds, the server automatically disconnects the second long connection, etc., which is not limited here.

[0236] The method of disconnecting the first long connection may also refer to the method of disconnecting the second long connection, which is not limited here. For example, in response to the connection duration of the first long connection reaching a preset duration threshold, disconnecting the first long connection with the first control device.

[0237] In an optional embodiment, first display information sent by a first control device through a first long connection is received at a first moment, and the first display information is information rendered and displayed on a first interface of the first control device based on status real-time information; second display information sent by an extended reality device through a second long connection is received at a first moment, and the second display information is information rendered and displayed on a second interface of the extended reality device based on status real-time information.

[0238] Illustratively, the first moment is an arbitrarily selected moment; or, the first moment is any periodic moment when periodic statistical information differences are performed through a preset period, etc.; at the first moment, the server counts the first display information currently displayed by the first control device through the first long connection, and also counts the second display information currently displayed by the extended reality device through the second long connection, and compares the difference between the two.

[0239] In an optional embodiment, in response to the information difference between the first display information and the second display information reaching a preset difference condition, at least one of the first long connection and the second long connection is disconnected.

[0240] Illustratively, the preset difference condition is a pre-set condition, such as the first display information is a 25% program download progress displayed on the first interface of the first control device, and the second display information is a 36% program download progress displayed on the second interface of the extended reality device. If the preset difference condition is a threshold of 5%, the information difference between the first display information and the second display information reaches the preset difference condition; therefore, the server can automatically disconnect the first long connection and retain the second long connection, or automatically disconnect the second long connection and retain the first long connection, or disconnect the first long connection and the second long connection at the same time, which is not limited here.

[0241] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0242] To sum up, the long connection communication method makes device interaction more stable, which helps to ensure that the extended reality device sends real-time status information to the first control device more promptly and accurately, reduces network burden, saves resource consumption, and makes device interaction suitable for application scenarios with strong real-time and frequent interactions. It can also be applied to remote operation scenarios where the distance between the first control device and the extended reality device is far, thereby improving the application range of device interaction under the device pairing relationship.

[0243] In the embodiment of the present application, the content of establishing a long connection based on a connection request is introduced. The long connection can reduce the overhead of establishing the connection, so that the connection can be used for a long time after connecting once, reducing excessive requests and other additional operations, and can also greatly improve the utilization efficiency of network resources. It is suitable for real-time applications or long-term communication scenarios, improves the first control device's understanding of the device execution of the extended reality device, and ensures the reliability of the connection.

[0244] In an optional embodiment, if Figure 6 As shown, it is a flowchart of a first control device executing a device communication method, which includes the following steps 610 to 630.

[0245] Step 610: Establish a first long connection with the server through the Internet.

[0246] Among them, there is a pairing relationship between the first control device and the extended reality device, and a second long connection is established between the extended reality device and the server.

[0247] In some embodiments, when the first control device is not logged into an account, in response to receiving a first account association operation for a second account logged into the extended reality device, the second account is automatically logged into the first control device, and the first account association operation is used to establish a pairing relationship between the first control device and the extended reality device.

[0248] Illustratively, the first account association operation is an operation received by the first control device and used to trigger association with the second account in the extended reality device.

[0249] Optionally, the first account association operation is implemented as an interface collection operation for the account interface corresponding to the second account. For example, the first control device uses a camera to capture the device interface of the extended reality device when the second account is logged in (such as the account homepage, the device main screen logged in to the second account, etc.), then the first control device is automatically logged in to the same account as the second account logged in by the extended reality device.

[0250] Optionally, the first account association operation is implemented as a device contact operation at a preset distance from the extended reality device with the second account logged in. For example, when the first control device is close to the extended reality device with the second account logged in, the same account as the second account logged in by the extended reality device is automatically logged in on the first control device, etc., which is not limited here.

[0251] In some embodiments, when neither the first control device nor the extended reality device has logged into an account, in response to receiving a device binding operation for the extended reality device, when the first control device logs into the first account, the first account is automatically logged into the extended reality device; or, when the extended reality device logs into the second account, the second account is automatically logged into the first control device.

[0252] Illustratively, the device binding operation is an operation for establishing a device binding relationship between the first control device and the extended reality device.

[0253] Optionally, the first control device establishes a device binding relationship with the extended reality device by collecting relevant data of the extended reality device (such as interface data, a graphic identification code on the main body of the extended reality device, etc.), such as the first control device shoots a preset interface (such as a main interface, etc.) of the extended reality device through a camera to establish a device binding relationship; or the first control device contacts the extended reality device for a preset time (such as continuous contact for 3 seconds) to establish a device binding relationship, etc. Based on the device binding operation, when the first control device or the extended reality device logs in to an account, the same account is automatically logged in on the device with the device binding relationship.

[0254] Optionally, the first control device collects relevant data of the extended reality device and biometric data corresponding to the first subject (such as palm print data, fingerprint data, facial data, etc.), and the first subject can be implemented as a subject that operates the extended reality device. For example, the first control device simultaneously captures the preset interface of the extended reality device and the palm print of the first subject through a camera to obtain interface data and palm print data. The first control device establishes a device binding relationship with the extended reality device based on the collected interface data, and automatically logs in to the first account corresponding to the first subject based on the biometric data, wherein the extended reality device synchronously logs in to the first account based on the binding association relationship, etc., which are not limited here.

[0255] In some embodiments, a first connection establishment request is sent to a server via the Internet, the first connection establishment request including a first device identifier corresponding to a first control device and a first account identifier of a first account logged in to the first control device; in response to receiving first request feedback information fed back by the server, a first long connection with the server is established.

[0256] Step 620: In response to receiving the device control operation, a first control instruction is sent to the server through the first persistent connection.

[0257] The device control operation is used to control the operation of the extended reality device, and the first control instruction is used to control the extended reality device to execute a state change process related to the first application based on the control execution information.

[0258] In some embodiments, the first control instruction further includes at least one of an instruction sending parameter, an instruction receiving parameter, an instruction sending level, and instruction retransmission information.

[0259] The instruction sending parameters include a first device identifier and a first account identifier corresponding to the first control device, and the first account identifier is the identifier of the first account logged in to the first control device; the instruction receiving parameters include a second device identifier and a second account identifier corresponding to the extended reality device, and the second account identifier is the identifier of the second account logged in to the extended reality device; the instruction sending level is used to characterize the priority requirement for sending the first control instruction; the instruction retransmission information is used to characterize the number of repeated transmissions when the first control instruction is not successfully sent.

[0260] Step 630: Receive real-time status information corresponding to the extended reality device.

[0261] The real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0262] In an optional embodiment, when the extended reality device is within the range of the first device corresponding to the first control device, real-time status information sent by the extended reality device through the short connection technology is received.

[0263] Illustratively, the first device range is usually a preset area range centered on the first control device, such as a 10-meter radius around the first device range; if the extended reality device is within the first device range of the first control device, the extended reality device can efficiently send real-time status information to the first control device through short connection technology. Short connection technology is a technology for performing data transmission within the first device range, and short connection technology includes at least one of a variety of technologies such as Bluetooth transmission technology, near field communication technology (NFC), and Zigbee technology. This method can flexibly switch between long connection technology and short connection technology based on the distance between devices, thereby improving the flexibility of technology use and data transmission accuracy.

[0264] With the help of long connection communication mode, device interaction is more stable, which helps to ensure that the extended reality device can send real-time status information to the first control device more promptly and accurately, reduce network burden and save resource consumption; with the switching of short connection technology and long connection technology, device interaction can also be made more flexible, which is not only suitable for application scenarios with strong real-time and frequent interactions, but can also be cleverly used in various operation scenarios such as long-range and short-range, so that the first control device can flexibly display relevant content based on real-time status information, and can also assist in controlling the extended reality device to a certain extent, thereby improving the application breadth and flexibility of device interaction under the device pairing relationship.

[0265] In an optional embodiment, if Figure 7 As shown, it is a flowchart of an extended reality device executing a device communication method, which includes the following steps 710 to 730.

[0266] Step 710: Establish a second long connection with the server through the Internet.

[0267] Among them, there is a pairing relationship between the extended reality device and the first control device, and a first long connection is established between the first control device and the server.

[0268] In some embodiments, when no account is logged in to the extended reality device, in response to receiving a second account association operation for a first account logged in to the first control device, the first account is automatically logged in to the extended reality device, and the second account association operation is used to establish a pairing relationship between the first control device and the extended reality device.

[0269] Illustratively, the second account association operation is an operation received by the extended reality device and used to trigger association with the first account in the first control device.

[0270] Optionally, the second account association operation is implemented as an interface collection operation for the account interface corresponding to the first account. For example, the extended reality device uses a camera to capture the device interface of the first control device when it logs in to the first account (such as the account homepage, the device main screen logged in to the first account, etc.), and then automatically logs in to the same account as the first account logged in by the first control device on the extended reality device.

[0271] Optionally, the second account association operation is implemented as a device contact operation at a preset distance from the first control device with the first account logged in. For example, when the extended reality device is close to the first control device with the first account logged in, the same account as the first account logged in by the first control device is automatically logged in on the extended reality device, etc., which is not limited here.

[0272] In some embodiments, when neither the first control device nor the extended reality device has logged into an account, in response to receiving a device binding operation for the extended reality device, when the extended reality device logs into the second account, the first account is automatically logged into the extended reality device; or, when the extended reality device logs into the second account, the second account is automatically logged into the first control device.

[0273] Illustratively, the device binding operation is used to establish a device binding relationship between the first control device and the extended reality device.

[0274] Optionally, the extended control device establishes a device binding relationship with the extended reality device by collecting relevant data of the first control device (such as interface data, a graphic identification code on the main body of the first control device, etc.), such as the first control device photographs a preset interface of the first control device through a camera to establish a device binding relationship; or, the first control device contacts the extended reality device for a preset time to establish a device binding relationship, etc.

[0275] Optionally, the extended control device collects relevant data of the first control device and biometric data corresponding to the second subject (such as palm print data, fingerprint data, facial data, etc.), and the second subject can be implemented as the subject operating the first control device. For example, the extended reality device continuously photographs the graphic identification code on the first control device body and the face of the second subject through a camera within a preset time (such as within 5 seconds) to obtain the graphic identification code data and facial data; the extended control device establishes a device binding relationship with the first control device based on the collected graphic identification code data, and automatically logs in to the second account corresponding to the second subject based on the biometric data, wherein the first control device synchronously logs in to the second account based on the binding association relationship, etc., which are not limited here.

[0276] Step 720: Generate real-time status information when executing a status change process related to the first application.

[0277] Optionally, the extended reality device performs a state change process related to the first application based on the state adjustment operation, thereby generating real-time state information.

[0278] Optionally, the extended reality device receives a first control instruction corresponding to the first control device, and executes a state change process related to the first application based on control execution information in the first control instruction, thereby generating real-time status information.

[0279] Step 730: Send the real-time status information to the server through the second long connection, and the server is used to send the real-time status information to the first control device through the first long connection.

[0280] The real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0281] In an optional embodiment, if the extended reality device receives a state adjustment operation of a first operation style to execute a state change process related to a first application, first information is generated as real-time status information; when the extended reality device receives a state adjustment operation of a second operation style to execute a state change process related to the first application, second information is generated as real-time status information.

[0282] The first operation style is different from the second operation style; the first information is different from the second information.

[0283] Illustratively, taking the status adjustment operation as a program download operation as an example, if the first operation style is a status adjustment operation implemented based on eye movement operation, the first information of the program download progress is generated as the real-time status information; if the second operation style is a status adjustment operation implemented based on gesture trigger operation, the second information of the program download network speed is generated as the real-time status information, etc., which is not limited here.

[0284] With the help of long connection communication mode, device interaction is more stable, which helps to ensure that the extended reality device sends real-time status information to the first control device more promptly and accurately, reduces network burden, saves resource consumption, and makes device interaction suitable for application scenarios with strong real-time and frequent interactions; the extended reality device can send real-time status information to the first control device by itself to prompt the first control device to provide assistance for it, and can also timely feedback real-time status information under the control of the first control device to improve the interaction effect between devices.

[0285] In an optional embodiment, the first control device is implemented as a mobile phone terminal, and the extended reality device is implemented as an XR headset. The above device communication method is a solution for realizing remote control of the XR headset by the mobile phone terminal based on the long connection technology in the interconnection and mutual control scenario of the mobile phone terminal and the XR headset, and synchronizing data to the mobile phone terminal in real time. Schematically, the above device communication method can also be implemented as follows.

[0286] In some embodiments, the mobile phone terminal establishes a communication connection with the server through a long connection technology (i.e., the first long connection mentioned above), and the XR headset establishes a communication connection with the server through a long connection technology (i.e., the second long connection mentioned above).

[0287] In principle, long connection technology refers to a technical method of establishing and maintaining a persistent network connection between a device and a server, rather than establishing a new connection every time a communication occurs. The main feature of a long connection is that once the connection is established, it remains open, allowing multiple data exchanges without the need to frequently establish and close connections, thereby reducing communication overhead and improving efficiency; long connections are suitable for applications that require continuous data exchange, such as real-time communications, instant messaging, online games, video streaming, etc.

[0288] Persistent connection technology can be implemented through a variety of protocols, including the Transmission Control Protocol (TCP), the TCP-based Internet Communication Protocol (WebSocketProtocol, WebSocket), the Quick User Datagram Protocol Connection (Quick UDP Internet Connection, QUIC) or at least one of other communication protocols that support persistent connections. Persistent connections can be established through the above protocols to provide some form of persistent connection.

[0289] A server is a device specifically responsible for supporting and managing XR devices (such as VR headsets, AR glasses, etc.) and user interactions. In this scenario, it can also be called an XR server. XR is a general term covering VR, AR, and MR. Therefore, the XR server can be implemented as a cloud server or a local server of the platform to which the XR device is connected (such as: the XR device is produced by the T platform, and the XR device is connected to the cloud server of the T platform); and / or, the XR server can be implemented as a background server for an application installed and running in the XR device (such as: the XR device is installed with application store A, and when the XR device runs application store A, it is connected to the background server of application store A); and / or, the XR server is an edge computing server (such as: a data center near the XR device), etc., which is not limited here.

[0290] In some embodiments, the XR headset and the mobile terminal respectively establish a long connection channel with the server when the device is started.

[0291] Optionally, taking the implementation of long connection communication in accordance with the WebSocket protocol as an example, for a mobile phone terminal, the mobile phone terminal may send a first connection establishment request to the server through the Websocket protocol.

[0292] Schematically, when a mobile terminal uses the HyperText Transfer Protocol (HTTP) to communicate, the Http header is used to carry the first authentication information required for the connection establishment process. The first authentication information is the authentication information of the mobile terminal. The authentication information (AuthHeader) includes: at least one of application identification (Identification, ID), device identification (ID), platform type, user identification (ID) and user ticket information.

[0293] Illustratively, the application identifier is the identifier of an application program that has a connection requirement when participating in the process of establishing a long connection (such as the first program identifier corresponding to the application program that has a connection requirement on the first control device). If the application identifier is the identifier corresponding to the application store, the application identifier is used to perform permission control analysis.

[0294] Illustratively, the device identifier is the identifier of the device used to establish a long connection with the server. For example, the device identifier of the authentication information sent by the mobile terminal is the device identifier of the mobile terminal, such as the model of the mobile terminal.

[0295] Illustratively, the platform type is the type of platform used to run the application, such as Android Operating System (Android), iPhone Operating System (iOS), website (web), etc.; or, the platform type is used to characterize different devices, such as: pre-setting "XR_PLATFORM" to represent an XR headset, "MOBILE_PLATFORM" to represent a mobile phone terminal, "UNKNOWN_PLATFORM" to represent an unknown device, etc., which is not limited here.

[0296] Illustratively, the user identifier is the account identifier of the account logged in on the device used to initiate the connection request, such as the identifier of the account logged in on the application store installed on the mobile terminal (such as the second application, third application, etc. mentioned above that have the connection requirement in the mobile terminal).

[0297] Illustratively, the user ticket is used to verify the identity of the requester who initiates the connection request. The user ticket may be, for example, a session identifier (Session ID), an open authorization token (OAuth token), etc., which are not limited here.

[0298] Optionally, the first connection establishment request sent by the mobile terminal is composed of a key-value format, where the key is fixed to be authentication (Authorization), and the value is authentication information expressed in a string format obtained by serializing the authentication information and encoding it.

[0299] Schematically, the authentication information is serialized using the Protocol Buffers (Protobuf) serialization method to convert the authentication information in the form of a data structure into a byte stream for easy network transmission; then the byte stream can be converted into a printable string through Standard Base64 Encoding; Base64 encoding is a method of encoding binary data into printable characters, which converts every three bytes (24 bits) of data into four characters (each character represents 6 bits), so that binary data that cannot be directly displayed can be safely transmitted in text form.

[0300] Schematically, the following is a schematic content of using the HTTP header (Header) to carry the authentication information (AuthHeader) required for the connection process. Optionally, the application identifier in the authentication information is represented by "string appid", the device identifier in the authentication information is represented by "string device_id", the platform type in the authentication information is represented by "Platform platform", the user identifier in the authentication information is represented by "string open_id", and the user ticket in the authentication information is represented by "string access_token", etc., which are not limited here.

[0301] Similarly, for XR headsets, the XR headsets can send a second connection request to the server through the Websocket protocol. Schematically, when the XR headset uses the Hypertext Transfer Protocol for communication, the HTTP header is used to carry the second authentication information (second connection information) required for the connection process; referring to the content of the above authentication information, the second authentication information includes the application ID on the XR headset that needs to establish a long connection to achieve data transmission, the device ID corresponding to the XR headset, the platform type of the XR headset (such as an XR headset), the user ID logged in on the XR headset, and the user ticket corresponding to the XR headset.

[0302] In some embodiments, a connection request is automatically sent to a server when the mobile terminal and / or the XR headset is turned on; or, a connection request is automatically sent to a server when the mobile terminal and / or the XR headset is turned on and runs an application that needs to transmit data through a long connection (such as an app store, etc.); or, after the mobile terminal and / or the XR headset is turned on, a connection request is manually triggered to generate a connection request and send it to the server, etc., which is not limited here.

[0303] In an optional embodiment, after receiving the connection request, the server uses standard Base64 decoding key as the value data of Authorization, and obtains the authentication information through protobuf deserialization; then, first verify the legitimacy of the application ID, device ID, platform type, and user ID, and then verify whether the user ticket is valid.

[0304] Illustratively, the server verifies the first authentication information (the above-mentioned first connection information) based on the first connection request. If the application ID, device ID, platform type, and user ID in the first authentication information are legal and the user ticket is valid, the server establishes a first long connection with the mobile terminal. Similarly, the server verifies the second authentication information (the above-mentioned second connection information) based on the second connection request. If the application ID, device ID, platform type, and user ID in the second authentication information are legal and the user ticket is valid, the server establishes a second long connection with the XR headset.

[0305] Optionally, after the server verifies the connection request (the first connection request and the second connection request), the server obtains the long connection session information (Session) based on the authentication information, and updates the long connection session information to the Redis storage (such as the first session message corresponding to the first authentication information and / or the second session information corresponding to the second authentication information) to complete the long connection process.

[0306] Schematically, the persistent connection session information includes the device identifier, platform type, and user identifier in the authentication information, as well as the connection identifier (such as represented by string conn_id) and gateway access layer information (GatewayInfo gateway_info); the connection identifier is an identifier used to characterize the connection status when establishing a connection based on a connection request, and the gateway access layer information refers to various information involved when the gateway processes device requests in network communications. In the persistent connection scenario, the gateway needs to manage sessions, maintain connections, perform routing, load balancing, security management and other processes. The gateway access layer information enables the gateway to ensure the stability, effectiveness and security of persistent connections.

[0307] In an optional embodiment, after the mobile phone terminal and the XR headset each establish a long connection with the server according to the above process, the mobile phone terminal can receive control instructions from the XR headset and perform corresponding operations, and can also receive operation data from the XR headset and render and display it; similarly, the XR headset can receive control instructions from the mobile phone terminal and perform corresponding operations, and can also receive operation data from the mobile phone terminal and render and display it, etc., which is not limited here.

[0308] In some embodiments, the following description is given by taking a mobile terminal controlling an XR headset to download an application as an example.

[0309] Optionally, the mobile terminal is logged in with a first account, and the XR headset is logged in with a second account, such as the first account is an account logged in on the mobile terminal, and the second account is an account logged in on the XR headset; or, the first account is an account logged in on an application (such as an app store) installed on the mobile terminal, and the second account is an account logged in on an application (such as an app store) installed on the XR headset.

[0310] Indicatively, the mobile terminal provides an account query function for querying other devices logged in with the same account; based on triggering the account query function (such as performing a trigger operation on the account query control), query other devices logged in with the same account, if the other devices include an XR headset, then the second account logged in with the XR headset is the same account as the first account of the mobile terminal. For example, based on the account query function, the mobile terminal can query the device ID of the XR headset logged in with the user ID (first account), that is, it can query and determine the user ID and device ID of the XR headset.

[0311] Illustratively, taking the example of a mobile terminal obtaining the user ID and device ID of an XR headset based on an account query function, the mobile terminal generates a query request based on the account query function, and the query request includes the query platform type of the platform indicated by the account query function (such as indicating to query the XR headset) and the user ID (first account) currently logged in by the mobile terminal; based on the query request issued by the mobile terminal, the mobile terminal may receive a query response, and the query response includes at least two situations, one is the error code situation, such as expressed as "Ret ret", the data type is Ret, and the ret field is 1, indicating the returned error code. Ret is usually a custom data type. If it is returned, it indicates that the query request is unsuccessful; the second is that the target device identifier (such as the model of the XR device) and the target platform user identifier (such as the account logged in on the XR device) are successfully returned and are not empty, which means that the request is processed normally.

[0312] In some embodiments, a pairing relationship is established between devices logged in to the same account, such as: the mobile phone terminal finds the XR headset logged in to the same account, and establishes a pairing relationship between the mobile phone terminal and the XR headset; or, after the mobile phone terminal finds the XR headset logged in to the same account, it sends a pairing request to the XR headset, and when the XR headset responds to the pairing request, a pairing relationship is established between the mobile phone terminal and the XR headset, etc., which is not limited here.

[0313] In some embodiments, taking the example of a mobile phone terminal controlling an XR headset to download or update an application (such as a first application), the mobile phone terminal constructs a download instruction as a control instruction to control the XR headset through the download instruction.

[0314] Optionally, the download instruction includes program execution information, and the program execution information is used to represent the operation information of controlling the process of changing the execution state of the XR headset. Schematically, if it is necessary to control the XR headset to download an application, the program execution information is the operation information used to control the XR headset to execute the application download process. For example: the program execution information includes at least an operation (action) and an operation object (such as app_id). The action of the download instruction is set to "ACTION_DOWNLOAD", which represents the download operation, and app_id is set to the program representation (or application ID) of the application to be downloaded.

[0315] Optionally, in addition to the download operation represented by "ACTION_DOWNLOAD", the update operation may also be represented by "ACTION_UPDATE", and the progress feedback operation (such as feedback on program download progress, program update progress, etc.) may also be represented by "ACTION_PROGRESS", which is not limited here.

[0316] In some embodiments, the control instruction includes, in addition to the above-mentioned program execution information, at least one of an instruction sending parameter, an instruction receiving parameter, an instruction sending level, and an instruction retransmission information.

[0317] Illustratively, the instruction sending parameters are used to characterize information related to the sending entity of the control instruction, such as the instruction sending parameters include the terminal identifier corresponding to the mobile phone terminal and the first account identifier of the first account; the instruction receiving parameters are used to characterize information related to the receiving entity indicating the reception of the control instruction, such as the instruction receiving parameters include the device identifier corresponding to the XR device and the second account identifier of the second account (if the first account and the second account are the same, the second account identifier is the first account identifier); the instruction sending level is used to characterize the priority requirement for sending the control instruction; the instruction retransmission information is used to characterize the number of times the control instruction is repeatedly sent when the control instruction is not successfully sent.

[0318] Optionally, the instruction sending level is a predetermined message reliability level during the instruction sending process, which determines the reliability guarantee level required for the control instruction during the transmission process, affecting the transmission strategy of the control instruction, error handling method, and response to loss delays and other issues.

[0319] Schematically, as shown above, a schematic division method of message reliability levels is introduced, such as: at most once, as much as possible once, at least once and exactly once. In addition, it can also include unknown levels, such as "UNKNOWN_LEVEL", which means that the message reliability level is not set or the message reliability level cannot be identified and determined, etc., which is not limited here.

[0320] Among them, at most once means that the control instruction is transmitted at most once, and it will not be transmitted multiple times even if the transmission fails; Try to once means that the control instruction is transmitted once under the best case, but it can be transmitted multiple times; At least once means that the control instruction is transmitted at least once, so it can be transmitted multiple times; Exactly once means that the control instruction is strictly guaranteed to be sent only once, without duplication or loss, and usually a complex mechanism is required to ensure this situation.

[0321] Optionally, the instruction retransmission information is the information that constrains the control instruction to retransmit at most when the instruction transmission level is implemented as "as much as possible once", "at least once" or "exactly once". For example, if the instruction retransmission information is 5, then when the control instruction fails to be sent (such as the XR device does not receive the control instruction, or receives an invalid control instruction, etc.), the control instruction can be sent repeatedly, and the control instruction can be sent repeatedly at most 5 times, which is not limited here.

[0322] For example: the mobile terminal constructs a long connection message body as a control instruction, wherein the instruction sending parameter is represented by "LcsMessage.sender", which is set to the first account identifier (user ID) and terminal identifier (terminal ID) of the first account logged in by the mobile terminal; the instruction sending parameter is represented by "LcsMessage.receiver", which is set to the second account identifier of the second account logged in by the XR headset and the device identifier of the XR headset; the program execution information is represented by "LcsMessage.message.message", which is set to the string serialized by the protobuf method for AppDownloadAndUpdate; the instruction sending level is represented by "LcsMessage.message.reliability", such as setting it to TRY_BEST_ONCE; the instruction retransmission information is represented by "LcsMessage.message.max_retry_count", such as setting it to 5, to try to ensure that the control instruction can reach the XR headset.

[0323] Illustratively, the mobile phone terminal constructs a first control instruction and sends it to the XR device. The first control instruction includes an instruction sending parameter (indicating user information of the mobile phone terminal as the sending subject), an instruction receiving parameter (indicating user information of the XR device as the receiving subject) and long connection transmission layer information; the user information indicates the user ID and the device ID, such as the instruction sending parameter includes a first device ID indicating the mobile phone terminal and a first account ID logged in to the mobile phone terminal, and the instruction receiving parameter includes a second device ID indicating the XR headset and a second account ID logged in to the XR headset; the long connection transmission layer information includes control execution information, instruction sending level (message reliability level), instruction retransmission information (maximum number of message retransmissions), instruction expiration time and instruction ID.

[0324] The expiration time is time constraint information for the first control instruction, ensuring that the control instruction is no longer processed or consumed after the specified time; the instruction identifier is a unique instruction identifier corresponding to the first control instruction, etc., which is not limited here.

[0325] In an optional embodiment, the mobile terminal sends the control instruction (such as LcsMessage) to the server (such as the XR background server) through the long connection channel (the first long connection); the server determines the receiving subject according to the instruction receiving parameters in the control instruction (i.e., the second account identifier and device identifier of the receiving subject), such as the receiving subject is an XR device; the server also confirms the corresponding user long connection session information based on the control instruction, and according to the message reliability level and the maximum number of retransmissions of the message, sends the application download instruction (such as the program execution information in the control instruction) to the XR headset through the long connection channel (the second long connection) between the XR headset. For example: If the message reliability level is set to "TRY_BEST_ONCE", it needs to be persisted to the database (DateBase, DB) first, and then try to forward it to the XR headset. If the forwarding fails, it is transferred to the priority queue and then retried.

[0326] Optionally, for the above message reliability levels, the priority levels increase from top to bottom; that is, "exactly once" has the highest priority, "at least once" has the second highest priority, and "as much as possible once" has the lowest priority. Since "at most once" cannot be transmitted repeatedly, there is no need to judge the priority situation.

[0327] In some embodiments, after the XR headset receives the program execution information pushed by the server, it parses the business custom message body and obtains the AppDownloadAndUpdate structure through protobuf deserialization operation, and then executes the corresponding application download action according to the action and appid therein, such as executing a download operation for the first application.

[0328] In some embodiments, the XR headset synchronizes the download progress to the mobile terminal in real time.

[0329] Schematically, the XR headset queries the account ID and terminal ID of the mobile terminal logged in with the same account based on the method of logging in to the same account; then, the XR headset constructs an application download progress instruction (real-time status information).

[0330] Optionally, refer to the content of the control instruction generated by the above-mentioned mobile terminal to implement the content of the XR headset to construct the application download progress instruction; schematically, the application download progress instruction includes download progress information, such as the action in the download progress information is set to ACTION_PROGRESS, and app_id is set to the application ID of the downloaded application.

[0331] Optionally, the XR device can also construct a long connection message body in the application download progress instruction, where LcsMessage.sender is set to the account ID and device ID of the XR headset; LcsMessage.receiver is set to the account ID and terminal ID of the mobile terminal; LcsMessage.message.message is set to the string serialized by AppDownloadAndUpdate through proto, etc. Optionally, since the application download progress instruction is not particularly important, LcsMessage.message.reliability can be set to AT_MOST_ONCE, and LcsMessage.message.max_retry_count can be set to 3.

[0332] In some embodiments, the XR headset sends the LcsMessage to the server through a long connection channel (second long connection); the server finds the corresponding user long connection session information according to the receiving subject in the download progress information, and sends the application download progress instruction (such as the download progress information) to the mobile terminal through the long connection channel according to the message reliability level and the maximum number of retransmissions of the message.

[0333] Since the application download progress instruction is set to AT_MOST_ONCE, it is directly forwarded to the mobile terminal. If the forwarding fails, a retry is attempted. The application can be discarded after the maximum number of retries.

[0334] In some embodiments, after receiving the long connection message pushed by the server, the mobile terminal parses out the business custom message body, obtains the AppDownloadAndUpdate structure through protobuf deserialization operation, and renders and displays the download progress of the corresponding application according to action and appid.

[0335] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0336] In some embodiments, the above-mentioned efficient long connection communication protocols are adopted, such as: WebSocket protocol, QUIC protocol, etc. These protocols have the characteristics of low latency and high throughput, and can maintain a stable connection under poor network conditions. The WebSocket protocol also supports full-duplex communication and can achieve real-time data transmission.

[0337] In some embodiments, an automatic reconnection mechanism can also be designed to cope with network interruptions or instability. When a device (mobile terminal / XR device / server, etc.) detects a connection interruption, it will automatically try to reestablish the connection to ensure the continuity and stability of data transmission. The reconnection mechanism includes an exponential backoff algorithm that can gradually increase the reconnection interval after multiple reconnection failures to avoid frequent reconnection and consumption of system resources.

[0338] In schematic form, the long connection between the mobile terminal / XR headset and the server can automatically reconnect after a network interruption; the exponential backoff algorithm means that when an operation fails or a conflict occurs, the system will delay for a certain period of time before retrying, and after each failure, the delay time (i.e., the retry interval) will increase exponentially. This growth method can effectively reduce the occurrence of conflicts and make the system more stable under high concurrency conditions.

[0339] In some embodiments, by introducing a heartbeat detection mechanism, the instructing device (mobile terminal / XR device, etc.) to periodically send a heartbeat packet to detect the connection status. If no heartbeat response is received within a preset time, the connection is considered disconnected and the reconnection mechanism is triggered. The heartbeat detection mechanism can detect and handle connection anomalies in a timely manner, improving connection reliability.

[0340] In some embodiments, the reliability level of the persistent connection message is defined as the above four levels, namely, unreliable (at most once), best effort delivery (best effort once), basically reliable (at least once), and highly reliable (guaranteed once). In the event of network interruption, abnormal system restart, system busyness, etc., the persistent connection system executes corresponding processing logic according to different message levels, including but not limited to retry, resend, etc., to improve the message reach rate.

[0341] In some embodiments, the priority level of long connection messages is defined as two levels, namely high priority (the system guarantees timely processing at any time, applicable scenarios are timely reach, delay-sensitive messages) and low priority (processed only when there are no high priority messages, such as normal status reports, push messages, etc.). When a single server node cannot handle it, high priority messages are processed first, and low priority messages can be delayed or not processed; when system resources are idle, both high priority and low priority messages are processed at the same time.

[0342] Schematically, the priority of the long connection message is pre-set. For example, if the priority of the download progress is set to high priority and the priority of the program running information is set to low priority, then when there is resource competition, the sending process of the download progress information is given priority to optimize the allocation and processing method of computing resources.

[0343] Optionally, there is a management system responsible for coordinating the resource processing process of multiple servers. When the resource load of multiple servers is large, the servers can be instructed to give priority to high-priority messages (control instructions) by reasonably allocating computing resources, and then process low-priority messages, etc. This is not limited here.

[0344] In some embodiments, the system supports load balancing and fault-tolerant design, which can distribute communication load among multiple servers to avoid single point failure. The load balancing mechanism can dynamically adjust the load distribution of the server to improve the processing capacity and response speed of the system.

[0345] In some embodiments, the mobile terminal includes:

[0346] The user interface component is used to receive control instructions input by the user; the communication component is used to transmit data with the XR headset through long connection technology; the data processing component is used to process the data received from the XR headset device and display it in the user interface.

[0347] In some embodiments, the XR head-mounted display device includes:

[0348] The communication component is used to transmit data with the mobile terminal through long connection technology; the control component is used to receive and execute control instructions from the mobile terminal; the data acquisition component is used to collect the status and data of the XR headset.

[0349] In some embodiments, the mobile terminal can remotely configure and manage the XR headset, including but not limited to firmware updates, parameter adjustments, and fault diagnosis.

[0350] In some embodiments, data synchronization of the mobile terminal includes synchronization of multiple data types, including but not limited to at least one of video data, audio data, sensor data, and user interaction data.

[0351] In some embodiments, the mobile terminal can analyze and process the data received from the XR headset and generate corresponding reports or prompt information.

[0352] In some embodiments, the XR headset can adjust the viewing angle, switch content, and start and stop functions according to the instructions of the mobile terminal.

[0353] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0354] In an optional embodiment, if Figure 8 As shown, the interactive architecture diagram for performing device interaction between a mobile terminal and an XR headset is described as follows.

[0355] It includes clients, including mobile terminals 810, such as terminals running Android systems and terminals running Apple systems, etc.; it also includes XR headsets 820, such as headsets running Android systems, which can also be implemented as headsets running Apple systems, etc.

[0356] Global Server Load Balancing (GSLB) is a distributed load balancing technology that realizes the transfer of traffic distribution between multiple servers or data centers as a client and backend. Through GSLB, the data (such as control instructions, etc.) sent by the client can be routed to the most appropriate server based on multiple factors (such as server health, geographical location, latency, load, etc.), thereby improving the availability, reliability and performance of the website or application.

[0357] There is usually a request gateway between the client and GSLB, which is responsible for processing requests from the client to GSLB and performing certain management and routing functions in the entire load balancing architecture; the network address and port of the request gateway (Request Gateway's IP+Port) is the entry point for communication between the client and GSLB. When the client sends a request, it will first reach GSLB through the request gateway. GSLB distributes traffic (such as the first control instruction, etc.) to the most suitable server or data center according to the load balancing strategy; GSLB can also feedback the distribution status to the client.

[0358] The background includes a data connection layer, a network working layer and a transport layer; the gateway of the data connection layer can send / delete user conversations (such as control instructions and other data) to adjust the session content that needs to be stored; in addition, the gateway can also send events generated during the interaction process to the event sequence, such as: user online events, message sending success events (such as successful sending of control instructions), message sending failure events (failure to send control instructions), etc.; in addition, the upload stream can be transmitted through the gateway, and the sending status of the long connection message can be determined by combining the upload stream. During the analysis of the sending status, the message reliability service of the long connection message will be analyzed. The long connection message is a message transmitted through the long connection technology, such as the first control instruction mentioned above, and the message reliability service is to determine the instruction sending level corresponding to the first control instruction.

[0359] In addition, the instruction sending level determined by the message reliability service can update the user message status in the storage so that the message can be sent in a manner that follows the instruction sending level; on the basis of sending the message (such as the first control instruction) in a manner that follows the instruction sending level, the message sending priority will be additionally determined; then, high-priority messages will be sent first based on the sending priority, and low-priority messages will be cached in the priority sequence so that low-priority messages can be sent after the high-priority messages are sent.

[0360] In some embodiments, in a weak network environment, you can consider using a long connection protocol with better performance, such as the QUIC protocol. If the near-field communication conditions are met, you can also switch to the BLE Bluetooth near-field communication mode to maximize the user experience.

[0361] It is worth noting that the above are merely illustrative examples and are not limited to the embodiments of the present application.

[0362] To sum up, the long connection communication method makes device interaction more stable, which helps to ensure that the extended reality device sends real-time status information to the first control device more promptly and accurately, reduces network burden, saves resource consumption, and makes device interaction suitable for application scenarios with strong real-time and frequent interactions. It can also be applied to remote operation scenarios where the distance between the first control device and the extended reality device is far, thereby improving the application range of device interaction under the device pairing relationship.

[0363] In the embodiment of the present application, a multi-terminal data communication scheme based on a long connection is introduced. It is highly dependent on the network quality of the device. In a weak network environment, long connections may be frequently disconnected and reconnected, and network delays may increase, affecting the user's real-time experience. The above process makes multi-terminal communication more instant and real-time, provides a persistent long-connection communication channel to avoid frequent connection and disconnection operations, can effectively reduce device power consumption and server resource consumption, is no longer limited by physical distance, is conducive to family members sharing device interaction experience, optimizes the smoothness of XR headset operation experience, and avoids the problem of low data transmission efficiency caused by discontinuous behaviors such as frequent wearing and taking off of XR headsets.

[0364] Fig. 9 is a structural block diagram of a device communication apparatus provided by an exemplary embodiment of the present application, such as Fig. 9 As shown, the device includes the following parts:

[0365] A connection establishing module 910, configured to establish a first long connection with a first control device through an Internet network, and to establish a second long connection with an extended reality device through an Internet network, wherein a pairing relationship exists between the first control device and the extended reality device;

[0366] An information receiving module 920 is configured to receive real-time status information sent by the extended reality device through the second persistent connection when the extended reality device executes a status change process related to the first application;

[0367] The information sending module 930 is used to send the real-time status information to the first control device through the first long connection based on the pairing relationship, and the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0368] In an optional embodiment, the information sending module 930 is also used to receive the real-time status information sent by the extended reality device through the second long connection when the extended reality device performs a state change process related to the first application based on a state adjustment operation; the state adjustment operation is an operation received by the extended reality device and performed on the first application.

[0369] In an optional embodiment, the information sending module 930 is also used to receive a first control instruction sent by the first control device, the first control instruction including control execution information corresponding to the first application and a second device identifier corresponding to the extended reality device; based on the second device identifier, the first control instruction is sent to the extended reality device, and the first control instruction is used to control the extended reality device to execute the state change process related to the first application based on the control execution information.

[0370] In an optional embodiment, the information sending module 930 is also used to determine the extended reality device controlled by the first control device based on the second device identifier; obtain the instruction sending level in the first control instruction, and the instruction sending level is used to characterize the priority requirement for sending the first control instruction; and send the first control instruction to the extended reality device based on the instruction sending level.

[0371] In an optional embodiment, the information sending module 930 is also used to send the first control instruction to the extended reality device once when the instruction sending level is at a first preset level; cache the first control instruction to a database when the instruction sending level is at a second preset level; call the first control instruction from the database, and send the first control instruction to the extended reality device at least once.

[0372] In an optional embodiment, the information sending module 930 is also used to obtain instruction retransmission information in the first control instruction, the instruction retransmission information is used to indicate the number of repeated transmissions when the first control instruction is not successfully sent; call the first control instruction from the database, and send the first control instruction once to the extended reality device; if the first control instruction is not successfully sent to the extended reality device, repeatedly send the first control instruction to the extended device based on the number of repeated transmissions indicated by the instruction retransmission information.

[0373] In an optional embodiment, the information receiving module 920 is also used to receive the program download progress sent by the extended device through the second long connection as the real-time status information when the extended reality device executes a program download process for the first application, and the state change process includes the program download process; or, when the extended reality device executes a program update process for the first application, receive the program update progress sent by the extended device through the second long connection as the real-time status information, and the state change process includes the program update process; or, when the extended reality device executes a program running adjustment process for the first application, receive the running screen data sent by the extended device through the second long connection as the real-time status information, and the state change process includes the program running adjustment process.

[0374] In an optional embodiment, the information receiving module 920 is also used to, when receiving multiple control instructions, determine the instruction sending order based on the sending priorities corresponding to the multiple control instructions respectively, and the multiple control instructions include the first control instruction; and send the multiple control instructions to the extended reality device in accordance with the instruction sending order.

[0375] In an optional embodiment, the connection establishment module 910 is also used to receive a first connection request sent by the first control device through the Internet network, the first connection request including a first device identifier corresponding to the first control device and a first account identifier of a first account logged in to the first control device; receive a second connection request sent by the extended reality device through the Internet network, the second connection request including a second device identifier corresponding to the extended reality device and a second account identifier of a second account logged in to the extended reality device; if the first connection request meets a preset connection condition, establish the first long connection with the first control device; if the second connection request meets the preset connection condition, establish a second long connection with the extended reality device.

[0376] In an optional embodiment, the connection establishment module 910 is also used to parse the first connection establishment request to obtain the first device identifier and the first account identifier; when the first device identifier and the first account identifier meet the preset connection establishment conditions, the first connection establishment information in the first connection establishment request is updated to the database to establish the first long connection with the first control device, and the first connection establishment information includes at least one of the first device identifier and the first account identifier.

[0377] In an optional embodiment, the connection establishment module 910 is also used to establish the first long connection with the first control device with a first connection duration based on the first program identifier in the first connection request indicating the second application, when the first connection request meets the preset connection condition, and the second application is an application installed on the first control device with a connection requirement; or, when the first connection request meets the preset connection condition, establish the first long connection with the first control device with a second connection duration based on the first program identifier in the first connection request indicating the third application, the third application is an application installed on the first control device with a connection requirement, the third application is different from the second application, and the first connection duration is different from the second connection duration.

[0378] In an optional embodiment, the information sending module 930 is also used to obtain a first device identifier in the real-time status information, where the first device identifier is the device identifier of the first control device, and the first device identifier is used to establish the pairing relationship between the first control device and the extended reality device; based on the first device identifier, the real-time status information is sent to the first control device; or, based on the pre-established pairing relationship between the first control device and the extended reality device, the real-time status information is sent to the first control device through the first long connection.

[0379] In an optional embodiment, the information sending module 930 is also used to disconnect the second long connection with the extended reality device in response to the number of pieces of status real-time information received from the first control device reaching a preset number threshold; or, in response to the connection duration of the second long connection reaching a preset duration threshold, disconnect the second long connection with the extended reality device.

[0380] In an optional embodiment, the information sending module 930 is also used to receive first display information sent by the first control device through the first long connection at a first moment, and the first display information is information rendered and displayed on a first interface of the first control device based on the real-time status information; receive second display information sent by the extended reality device through the second long connection at the first moment, and the second display information is information rendered and displayed on a second interface of the extended reality device based on the real-time status information; in response to the information difference between the first display information and the second display information reaching a preset difference condition, disconnect at least one of the first long connection and the second long connection.

[0381] Fig.10is a structural block diagram of a device communication apparatus provided by another exemplary embodiment of the present application, such as Fig.10 As shown, the device includes the following parts:

[0382] A connection establishing module 1010 is used to establish a first long connection with a server through an Internet network, the first control device and the extended reality device are paired, and the extended reality device has a second long connection with the server through the Internet network;

[0383] an instruction sending module 1020, configured to send a first control instruction to the server through the first persistent connection in response to receiving a device control operation, wherein the device control operation is used to control the operation of the extended reality device, and the first control instruction is used to control the extended reality device to execute a state change process related to the first application;

[0384] The information receiving module 1030 is used to receive real-time status information corresponding to the extended reality device, and the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0385] In an optional embodiment, the first control instruction further includes at least one of an instruction sending parameter, an instruction receiving parameter, an instruction sending level, and instruction retransmission information;

[0386] The instruction sending parameter includes a first device identifier and a first account identifier corresponding to the first control device, where the first account identifier is an identifier of a first account logged in to the first control device;

[0387] The instruction receiving parameter includes a second device identifier and a second account identifier corresponding to the extended reality device, where the second account identifier is an identifier of a second account logged in to the extended reality device;

[0388] The instruction sending level is used to represent the priority requirement for sending the first control instruction;

[0389] The instruction retransmission information is used to indicate the number of times the first control instruction is sent repeatedly when the first control instruction is not sent successfully.

[0390] In an optional embodiment, the connection establishment module 1010 is also used to send a first connection establishment request to the server through the Internet, wherein the first connection establishment request includes a first device identifier corresponding to the first control device and a first account identifier of a first account logged in to the first control device; in response to receiving the first request feedback information fed back by the server, the first long connection with the server is established.

[0391] In an optional embodiment, the information receiving module 1030 is also used to receive the real-time status information sent by the extended reality device through short connection technology when the extended reality device is within the range of the first device corresponding to the first control device. The short connection technology is a technology for performing data transmission within the range of the first device.

[0392] Fig.11 is a structural block diagram of a device communication apparatus provided by another exemplary embodiment of the present application, such as Fig.11 As shown, the device includes the following parts:

[0393] A connection establishing module 1110 is used to establish a second long connection with the server through the Internet, the extended reality device is paired with the first control device, and the first control device has a first long connection with the server through the Internet;

[0394] The information generation module 1120 is used to generate real-time status information when executing a status change process related to the first application;

[0395] The information sending module 1130 is used to send the real-time status information to the server through the second long connection, and the server is used to send the real-time status information to the first control device through the first long connection based on the pairing relationship, and the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

[0396] To sum up, the long connection communication method makes device interaction more stable, which helps to ensure that the extended reality device sends real-time status information to the first control device more promptly and accurately, reduces network burden, saves resource consumption, and makes device interaction suitable for application scenarios with strong real-time and frequent interactions. It can also be applied to remote operation scenarios where the distance between the first control device and the extended reality device is far, thereby improving the application range of device interaction under the device pairing relationship.

[0397] It should be noted that the device communication apparatus provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device communication apparatus and the device communication method embodiment provided in the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0398] Fig.12 The schematic diagram of the structure of a server provided by an exemplary embodiment of the present application is shown, which specifically includes the following structure.

[0399] The server 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including a random access memory (RAM) 1202 and a read only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the central processing unit 1201. The server 1200 also includes a mass storage device 1206 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0400] The mass storage device 1206 is connected to the central processing unit 1201 through a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1206 and its associated computer-readable media provide non-volatile storage for the server 1200. That is, the mass storage device 1206 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0401] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1204 and mass storage device 1206 can be collectively referred to as memory.

[0402] According to various embodiments of the present application, the server 1200 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 1200 can be connected to the network 1212 through the network interface unit 1211 connected to the system bus 1205, or the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).

[0403] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0404] The embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the device communication method provided by the above-mentioned method embodiments. Optionally, the computer device can be a terminal or a server.

[0405] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the device communication method provided by the above-mentioned method embodiments.

[0406] The embodiments of the present application also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the device communication method described in any of the above embodiments.

[0407] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0408] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0409] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A device communication method, characterized in that: The method is executed by a server, and comprises: Establishing a first long connection with a first control device through an Internet network, and establishing a second long connection with an extended reality device through the Internet network, wherein a pairing relationship exists between the first control device and the extended reality device; When the extended reality device executes a state change process related to the first application, receiving real-time state information sent by the extended reality device through the second persistent connection; The real-time status information is sent to the first control device through the first long connection based on the pairing relationship, where the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

2. The method according to claim 1, characterized in that When the extended reality device executes a state change process related to the first application, receiving the real-time state information sent by the extended reality device through the second long connection includes: When the extended reality device performs a state change process related to the first application based on a state adjustment operation, the real-time state information sent by the extended reality device through the second long connection is received; the state adjustment operation is an operation received by the extended reality device and performed on the first application.

3. The method according to claim 1, characterized in that In the case where the extended reality device executes a state change process related to the first application, before receiving the real-time state information sent by the extended reality device through the second long connection, the method further includes: receiving a first control instruction sent by the first control device, where the first control instruction includes control execution information corresponding to the first application and a second device identifier corresponding to the extended reality device; The first control instruction is sent to the extended reality device based on the second device identifier, where the first control instruction is used to control the extended reality device to execute the state change process related to the first application based on the control execution information.

4. The method according to claim 3, characterized in that The sending the first control instruction to the extended reality device based on the second device identifier includes: Determine the extended reality device controlled by the first control device based on the second device identifier; Acquire an instruction sending level in the first control instruction, where the instruction sending level is used to represent a priority requirement for sending the first control instruction; The first control instruction is sent to the extended reality device based on the instruction sending level.

5. The method according to claim 4, characterized in that The sending the first control instruction to the extended reality device based on the instruction sending level includes: When the instruction sending level is at a first preset level, sending the first control instruction to the extended reality device once; When the instruction sending level is at a second preset level, the first control instruction is cached in a database; the first control instruction is called from the database, and the first control instruction is sent to the extended reality device at least once.

6. The method according to claim 5, characterized in that The calling the first control instruction from the database, and sending the first control instruction to the extended reality device at least once, includes: Acquire instruction retransmission information in the first control instruction, where the instruction retransmission information is used to indicate the number of times the first control instruction is repeatedly sent when the first control instruction is not successfully sent; Recalling the first control instruction from the database, and sending the first control instruction to the extended reality device once; In the case where the first control instruction is not successfully sent to the extended reality device, the first control instruction is repeatedly sent to the extended reality device based on the number of repeated sending times indicated by the instruction retransmission information.

7. The method according to any one of claims 1 to 6, characterized in that: When the extended reality device executes a state change process related to the first application, receiving the real-time state information sent by the extended reality device through the second long connection includes: In a case where the extended reality device executes a program download process for the first application, receiving a program download progress sent by the extended reality device through the second long connection as the status real-time information, and the status change process includes the program download process; or In a case where the extended reality device executes a program update process for the first application, receiving a program update progress sent by the extended reality device through the second long connection as the status real-time information, and the status change process includes the program update process; or When the extended reality device executes a program running adjustment process for the first application, the running screen data sent by the extended device through the second long connection is received as the status real-time information, and the status change process includes the program running adjustment process.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In the case of receiving multiple control instructions, determining the instruction sending order based on the sending priorities respectively corresponding to the multiple control instructions, the multiple control instructions including the first control instruction; The plurality of control instructions are sent to the extended reality device according to the instruction sending order.

9. The method according to any one of claims 1 to 6, characterized in that: The establishing of a first long connection with the first control device through the Internet, and the establishing of a second long connection with the extended reality device through the Internet, include: receiving, through the Internet, a first connection establishment request sent by the first control device, where the first connection establishment request includes a first device identifier corresponding to the first control device and a first account identifier of a first account logged in to the first control device; receiving, through the Internet, a second connection establishment request sent by the extended reality device, wherein the second connection establishment request includes a second device identifier corresponding to the extended reality device and a second account identifier of a second account logged in to the extended reality device; When the first connection request meets the preset connection condition, the first long connection with the first control device is established; when the second connection request meets the preset connection condition, the second long connection with the extended reality device is established.

10. The method according to claim 9, characterized in that The establishing the first long connection with the first control device when the first connection establishment request meets a preset connection establishment condition includes: Parsing the first connection establishment request to obtain the first device identifier and the first account identifier; When the first device identifier and the first account identifier meet the preset connection conditions, the first connection information in the first connection request is updated to the database to establish the first long connection with the first control device, and the first connection information includes at least one of the first device identifier and the first account identifier.

11. The method according to claim 9, characterized in that The establishing the first long connection with the first control device when the first connection establishment request meets a preset connection establishment condition includes: In the case where the first connection request meets the preset connection condition, the first long connection with the first control device having a first connection duration is established based on the first program identifier in the first connection request indicating the second application, where the second application is an application with a connection requirement installed on the first control device; or When the first connection request meets the preset connection condition, the first long connection with the first control device is established based on the first program identifier in the first connection request indicating the third application, and the third application is an application with a connection requirement installed on the first control device. The third application is different from the second application, and the first connection duration is different from the second connection duration.

12. The method according to any one of claims 1 to 6, characterized in that: The sending the real-time status information to the first control device through the first long connection based on the pairing relationship includes: Acquire a first device identifier in the real-time status information, where the first device identifier is a device identifier of the first control device, and the first device identifier is used to establish the pairing relationship between the first control device and the extended reality device; send the real-time status information to the first control device based on the first device identifier; or, Based on the pre-established pairing relationship between the first control device and the extended reality device, the status real-time information is sent to the first control device through the first long connection.

13. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In response to the number of pieces of the real-time status information sent by the first control device reaching a preset number threshold, disconnecting the second long connection with the extended reality device; or, In response to the connection duration of the second long connection reaching a preset duration threshold, the second long connection with the extended reality device is disconnected.

14. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: receiving, at a first moment, first display information sent by the first control device through the first persistent connection, the first display information being information rendered and displayed on a first interface of the first control device based on the real-time status information; receiving, at the first moment, second display information sent by the extended reality device through the second persistent connection, where the second display information is information rendered and displayed on a second interface of the extended reality device based on the real-time status information; In response to an information difference between the first display information and the second display information reaching a preset difference condition, at least one of the first long connection and the second long connection is disconnected.

15. A device communication method, characterized in that: The method is performed by a first control device, and the method includes: A first long connection is established with a server via an Internet network, wherein the first control device is paired with an extended reality device, and a second long connection is established between the extended reality device and the server via the Internet network; In response to receiving a device control operation, sending a first control instruction to the server through the first persistent connection, wherein the device control operation is used to control the operation of the extended reality device, and the first control instruction is used to control the extended reality device to execute a state change process related to the first application; Receive real-time status information corresponding to the extended reality device, where the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

16. A device communication method, characterized in that: The method is performed by an extended reality device, and the method includes: A second long connection is established with the server through the Internet, the extended reality device is paired with the first control device, and the first control device has a first long connection established with the server through the Internet; generating status real-time information in the case of executing a status change process associated with the first application; The real-time status information is sent to the server through the second long connection, and the server is used to send the real-time status information to the first control device through the first long connection based on the pairing relationship, and the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

17. A device communication apparatus, characterized in that: The device comprises: a connection establishment module, configured to establish a first long connection with a first control device through an Internet network, and to establish a second long connection with an extended reality device through the Internet network, wherein a pairing relationship exists between the first control device and the extended reality device; an information receiving module, configured to receive real-time status information sent by the extended reality device through the second persistent connection when the extended reality device executes a status change process related to the first application; An information sending module is used to send the real-time status information to the first control device through the first long connection based on the pairing relationship, and the real-time status information is used to instruct the first control device to synchronously display the status change process related to the first application.

18. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the device communication method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the device communication method according to any one of claims 1 to 16.

20. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the device communication method as described in any one of claims 1 to 16.

Images