Interactive display method and device, augmented reality equipment, terminal equipment and medium

Abstract

The invention relates to an interactive display method and device, augmented reality equipment, terminal equipment and a medium. The interactive display method comprises the steps that communication connection is established between the augmented reality equipment and target terminal equipment; and based on the communication connection, displaying a first virtual interface in a display picture of the augmented reality device, the first virtual interface being generated by the target terminal device. According to the method and the device, interconnection and intercommunication between the augmented reality device and the target terminal device are realized, the large-view characteristic of the augmented reality device is exerted, an extended display effect is achieved on a real screen of the target terminal device, the display content area of the target terminal device is increased, the use efficiency of a user is improved, and the use experience of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of extended reality technology, and in particular to an interactive display method, apparatus, extended reality device, terminal device, and medium. Background Art

[0002] Extended reality technology refers to the use of computers to combine the real and the virtual to create a virtual scene that allows human-computer interaction. Extended reality technology is a general term for various technologies, including AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality). With the increasing power of VR devices and the expansion of external batteries, AR devices are becoming increasingly lightweight and thinner, and users can wear extended reality devices for longer periods of time. This inevitably leads to the simultaneous use of extended reality devices and terminal devices such as smartphones. However, during use, the extended reality device and terminal device are independent of each other, no different from daily terminal use, and cannot fully utilize the extended reality device's features such as multiple screens, a large field of view, and gesture interaction. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides an interactive display method, apparatus, extended reality device, terminal device and medium.

[0004] According to a first aspect of an embodiment of the present disclosure, an interactive display method is provided, which is applied to an extended reality device. The interactive display method includes:

[0005] Establishing a communication connection with the target terminal device;

[0006] Based on the communication connection, a first virtual interface is displayed in the display screen of the extended reality device, where the first virtual interface is generated by the target terminal device.

[0007] In some exemplary embodiments of the present disclosure, displaying the first virtual interface on the display screen of the extended reality device includes:

[0008] Determining a first display position of the target terminal device in the display image;

[0009] Based on the first display position, determining a second display position of the first virtual interface in the display screen;

[0010] The first virtual interface is displayed at the second display position.

[0011] In some exemplary embodiments of the present disclosure, determining the first display position of the target terminal device in the display screen includes:

[0012] Receiving first image information sent by the target terminal device, where the first image information includes a current screen display image of the target terminal device;

[0013] The first display position is determined based on the first image information.

[0014] In some exemplary embodiments of the present disclosure, the interactive display method further includes:

[0015] In response to a change in the first display position of the target terminal device, the second display position of the first virtual interface is updated based on the changed first display position.

[0016] In some exemplary embodiments of the present disclosure, establishing a communication connection with a target terminal device includes:

[0017] Sending a broadcast message, wherein the broadcast message is used to request establishment of a communication connection;

[0018] receiving a connection request sent by the target terminal device in response to the broadcast message;

[0019] In response to a user confirming a connection operation for the connection request, a communication connection is established with the target terminal device.

[0020] In some exemplary embodiments of the present disclosure, the sending of the broadcast message includes:

[0021] Obtaining second image information captured by a camera of the extended reality device;

[0022] When it is determined based on the second image information that a terminal device is in use, the broadcast message is sent out.

[0023] In some exemplary embodiments of the present disclosure, when it is determined according to the second image information that a terminal device is in use, issuing the broadcast message includes:

[0024] When there is a highlighted area in the second image information, it is determined that there is a terminal device in use, and the broadcast message is sent.

[0025] In some exemplary embodiments of the present disclosure, the interactive display method further includes:

[0026] Based on a first preset operation of the user on the first virtual interface, generating a control instruction corresponding to the first preset operation;

[0027] The control instruction is sent to the target terminal device, so that the target terminal device executes the control instruction.

[0028] In some exemplary embodiments of the present disclosure, the interactive display method further includes:

[0029] receiving a second virtual interface sent by the target terminal device, where the second virtual interface is generated by the target terminal device executing the control instruction;

[0030] The first virtual interface is updated to the second virtual interface.

[0031] According to a second aspect of an embodiment of the present disclosure, an interactive display method is provided, which is applied to a target terminal device. The interactive display method includes:

[0032] Establishing a communication connection with an extended reality device;

[0033] In response to a second preset operation of the user, a first virtual interface is generated, and the first virtual interface is sent to the extended reality device so as to be displayed on a display screen of the extended reality device.

[0034] In some exemplary embodiments of the present disclosure, the interactive display method further includes:

[0035] First image information is sent to the extended reality device so that the extended reality device determines a first display position of the target terminal device in the display screen according to the first image information, wherein the first image information includes a current screen display screen of the target terminal device.

[0036] In some exemplary embodiments of the present disclosure, the interactive display method further includes:

[0037] In response to the broadcast message sent by the extended reality device, sending a connection request to the extended reality device;

[0038] In response to the confirmation connection instruction sent by the extended reality device, a communication connection is established with the extended reality device.

[0039] In some exemplary embodiments of the present disclosure, the interactive display method further includes:

[0040] Receiving a control instruction sent by the extended reality device;

[0041] Executing the control instruction to generate a second virtual interface;

[0042] The second virtual interface is sent to the extended reality device, so that the extended reality device updates the first virtual interface to the second virtual interface.

[0043] According to a third aspect of an embodiment of the present disclosure, an interactive display device is provided, which is applied to an extended reality device. The interactive display device includes:

[0044] A first connection module, configured to establish a communication connection with a target terminal device;

[0045] A first execution module is configured to display a first virtual interface in a display screen of the extended reality device based on the communication connection, where the first virtual interface is generated by the target terminal device.

[0046] According to a fourth aspect of an embodiment of the present disclosure, an interactive display device is provided, which is applied to a target terminal device, and the interactive display device includes:

[0047] A second connection module is used to establish a communication connection with the extended reality device;

[0048] The second execution module is used to generate a first virtual interface in response to a second preset operation of the user, and send the first virtual interface to the extended reality device to display the first virtual interface on the display screen of the extended reality device.

[0049] According to a fifth aspect of an embodiment of the present disclosure, an extended reality device is provided, the extended reality device comprising:

[0050] processor;

[0051] a memory for storing processor-executable instructions;

[0052] The processor is configured to execute executable instructions in the memory to implement the interactive display method provided in the first aspect of the present disclosure.

[0053] According to a sixth aspect of an embodiment of the present disclosure, a target terminal device is provided, the target terminal device including:

[0054] processor;

[0055] a memory for storing processor-executable instructions;

[0056] The processor is configured to execute the executable instructions in the memory to implement the interactive display method provided in the second aspect of the present disclosure.

[0057] According to the seventh aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program that can be executed is stored. When the computer program is executed by a processor of an extended reality device, the interactive display method provided by the first aspect of the present disclosure is implemented.

[0058] According to the eighth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which a computer program that can be executed is stored. When the computer program is executed by a processor of a target terminal device, the interactive display method provided in the second aspect of the present disclosure is implemented.

[0059] According to the ninth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; the processor of an extended reality device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the extended reality device executes the interactive display method of messages provided in the first aspect of the present disclosure.

[0060] According to the tenth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium; a processor of a target terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the target terminal executes the interactive display method of messages provided in the second aspect of the present disclosure.

[0061] The above-mentioned method of the present invention has the following beneficial effects: the present invention realizes the interconnection and interoperability of the extended reality device and the target terminal device, gives play to the large field of view characteristics of the extended reality device, has the effect of expanding the display of the real screen of the target terminal device, increases the display content area of ​​the target terminal device, improves the user's usage efficiency, and improves the user's usage experience.

[0062] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0064] Figure 1 The figure is a flowchart of an interactive display method according to an exemplary embodiment.

[0065] Figure 2 The figure is a flowchart of an interactive display method according to an exemplary embodiment.

[0066] Figure 3 The figure is a flowchart of an interactive display method according to an exemplary embodiment.

[0067] Figure 4 The figure is a flowchart of an interactive display method according to an exemplary embodiment.

[0068] Figure 5 It is a schematic diagram showing an interactive display method according to an exemplary embodiment.

[0069] Figure 6 It is a block diagram of an interactive display device according to an exemplary embodiment.

[0070] Figure 7 It is a block diagram of an interactive display device according to an exemplary embodiment.

[0071] Figure 8 is a block diagram of an extended reality device according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure.

[0073] As XR devices become increasingly portable and increasingly optimized, users are wearing them for longer periods of time, sometimes using them simultaneously with devices like smartphones. However, the XR devices and devices are independent of each other during use, preventing them from fully utilizing features like multiple screens, a wide field of view, and gesture interaction.

[0074] To address the aforementioned issues, the present disclosure provides an interactive display method, which is applied to an extended reality device. The extended reality device establishes a communication connection with a target terminal device. Based on the communication connection, a first virtual interface generated by the target terminal device is displayed on a display screen of the extended reality device. The present disclosure achieves interconnection between the extended reality device and the target terminal device, leveraging the extended reality device's wide field of view to achieve an extended display effect on the target terminal device's real screen, increasing the target terminal device's display content area, improving user efficiency, and enhancing the user experience.

[0075] The exemplary embodiment of the present disclosure provides an interactive display method, which is applied to an extended reality device, which includes a VR device, an AR device, an MR device, etc. Figure 1 As shown, the interactive display method shown in the present disclosure includes:

[0076] S101, establishing a communication connection with a target terminal device;

[0077] S102: Based on the communication connection, display a first virtual interface on a display screen of the extended reality device, where the first virtual interface is generated by the target terminal device.

[0078] In step S101, the target terminal device can be any electronic device with a display screen, such as a smartphone, smartwatch, tablet, air conditioner, refrigerator, etc. The communication connection function between the extended reality device and the target terminal device is enabled. Based on the information exchange between the extended reality device and the target terminal device, the extended reality device and the target terminal device establish a communication connection. The communication connection includes wireless connection methods and wired connection methods. The wireless connection can be a Bluetooth connection, ZigBee connection, WiFi connection, NFC connection, etc.

[0079] The present disclosure does not limit the number of target terminal devices that can be communicated with the extended reality device. The extended reality device can be communicated with one or more target terminal devices. For example, at time t1, the extended reality device only establishes a communication connection with the smartphone, at time t2, the extended reality device establishes a communication connection with both the smartphone and the smartwatch, and at time t3, the extended reality device establishes a communication connection with the smartphone, the smartwatch, and the air conditioner.

[0080] In step S102, after the extended reality device establishes a communication connection with the target terminal device, the target terminal device sends a first virtual interface to the extended reality device via the communication connection. The first virtual interface is displayed on the display screen of the extended reality device, and the extended reality device achieves the effect of expanding the display of the target terminal device's real screen. The user can use the extended reality device to simultaneously view the content of multiple screens of the target terminal device. The display screen of the extended reality device can display multiple first virtual interfaces. The display content of the first virtual interface can be any display screen of the target terminal device's screen, for example, it can be a different screen display of the same application as the current screen of the target terminal device, or it can be a display screen of a different application. Therefore, the display content of the multiple first virtual interfaces can be different screen displays of the same application, or it can be a display screen of different applications. The user can directly adjust the display screen of the first virtual interface through gesture commands, or can change the display screen of the first virtual interface by adjusting the display screen of the target terminal device. When the communication connection between the target terminal device and the extended reality device is disconnected or closed, the display screen of the extended reality device no longer displays the first virtual interface, and only displays the image generated by the extended reality device. In addition, when the target terminal device is in a screen-off state, that is, the user is not using the target terminal device, the display screen of the extended reality device no longer displays the first virtual interface.

[0081] The present disclosure does not limit the display position and display size of the first virtual interface. The first virtual interface can be displayed at any position in the display screen of the extended reality device, and the user can move the position of the first virtual interface according to personal usage habits. For example, in order to achieve an extended display effect, in the display screen of the extended reality device, the first virtual interface can be displayed around the display screen of the screen of the target terminal device, without covering or blocking the display screen of the screen of the target terminal device. The display size of the first virtual interface can be consistent with the screen size of the target terminal device, or the display size of the first virtual interface can be set to be larger than the screen size of the target terminal device so that the user can view the details of the first virtual interface. The display size of the first virtual interface can also be set to be smaller than the screen size of the target terminal device so that the display screen of the extended reality device can display multiple first virtual interfaces.

[0082] like Figure 5 As shown, the target terminal device is mobile phone 2. The extended reality device establishes a communication connection with mobile phone 2. Mobile phone 2 sends multiple first virtual interfaces to the extended reality device, so that the display screen 1 of the extended reality device displays four first virtual interfaces. The display size of the four first virtual interfaces is the same as the display size of the display interface 3 of the target terminal device. The four first virtual interfaces are displayed to the right of the display interface 3 of the target terminal device. Among them, the display content of the display interface 3 of the target terminal device is a car image in the album application, the first virtual interface 4 displays the display screen of the weather application, the first virtual interface 5 displays a landscape image in the album application, the first virtual interface 6 displays a chat interface with "YY" in the instant messaging application, and the first virtual interface displays a chat interface with "ZZ" in the instant messaging application. When the screen of mobile phone 2 is turned off, the first virtual interface 4, the first virtual interface 5, the first virtual interface 6, and the first virtual interface 7 are no longer displayed.

[0083] In the present disclosure, the extended reality device and the target terminal device are interconnected, giving full play to the large field of view characteristics of the extended reality device, achieving the effect of expanding the display of the real screen of the target terminal device, increasing the display content area of ​​the target terminal device, improving the user's usage efficiency, and enhancing the user's usage experience.

[0084] In some embodiments, displaying the first virtual interface on a display screen of an augmented reality device includes:

[0085] Determining a first display position of the target terminal device in the display image;

[0086] Based on the first display position, determining a second display position of the first virtual interface in the display screen;

[0087] The first virtual interface is displayed at the second display position.

[0088] Since the first virtual interface is generated by the target terminal device, the display screen of the first virtual interface is the content that the user wants to view. In order to enable the user to view the display interface of the target terminal device and the first virtual interface at the same time, the display position of the first virtual interface is determined based on the display position of the target terminal device. The extended reality device uses a camera to locate the target terminal device, determines the first display position of the target terminal device in the display screen, and determines the second display position of the first virtual interface in the display screen based on the first display position. The second display position can be one side, both sides, or all four sides of the upper, lower, left, or right side of the first display position, and the first virtual interface is displayed at the second display position. For example, if the target terminal device is determined to be at position A on the display screen, if the left side of position A is used as the second display position, then all first virtual interfaces are displayed on the left side of position A; if the surroundings of position A are used as the second display position, then all first virtual interfaces are displayed around position A.

[0089] In some embodiments, determining a first display position of the target terminal device in the display image includes:

[0090] Receiving first image information sent by a target terminal device, where the first image information includes a current screen display image of the target terminal device;

[0091] Based on the first image information, a first display position is determined.

[0092] The extended reality device is equipped with sensors, such as cameras and lidar, and uses these sensors to collect image information of the current environment and construct a map using this information. The extended reality device receives first image information, including the current screen display of the target terminal device, and compares the first image information with the image information of the current environment. If the image information of the current environment contains content that is consistent with the first image information, the location of the target terminal device on the map is determined, which is the first display position of the target terminal device on the display.

[0093] To improve the accuracy of determining the first display position, the extended reality device's sensor is always on, collecting real-time image information of the current environment. When the user repositions the extended reality device, the extended reality device uses this real-time image information to construct a map that matches the current environment. The extended reality device also receives the first image information sent by the target terminal device in real time and tracks the first display position of the target terminal device in real time by comparing the first image information with the image information of the current environment.

[0094] In some embodiments, the interactive display method further includes:

[0095] In response to a change in the first display position of the target terminal device, the second display position of the first virtual interface is updated based on the changed first display position.

[0096] Since the target terminal device or the extended reality device is movable, in order to avoid visual obstruction caused by the movement of the target terminal device or the extended reality device to the user, it is necessary to ensure that the relative position between the first display position and the second display position remains unchanged. When the first display position of the target terminal device changes, the second display position will also change based on the change of the first display position, and the first virtual interface will be displayed at the updated second display position. For example Figure 5 As shown, the four first virtual interfaces are displayed on the right side of the display interface 3 of the target terminal device. If the first display position changes, the second display position will also change synchronously, but after the change, the second display position is still on the right side of the first display position, that is, the four first virtual interfaces will still be displayed on the right side of the display interface 3 of the target terminal device.

[0097] There are two situations in which the first display position of the target terminal device changes: one is that the target terminal device moves but the extended reality device does not; the other is that the extended reality device moves but the target terminal device does not. For example, the extended reality device is a VR device that the user wears on their head, and the target terminal device is a mobile phone that the user holds in their hand. If the user keeps their head fixed and moves their hand to change the position of the phone, i.e., the VR device does not move, but the phone moves, the extended reality device can determine based on the first image information that the first display position has moved five centimeters to the left, and the second display position will also move five centimeters to the left, displaying the first virtual interface at the updated second display position. If the user keeps their hand fixed and turns their head to the left to change the position of the VR device, i.e., the phone does not move, but the VR device moves, the extended reality device can determine based on the first image information that the first display position has moved three centimeters to the right, and the second display position will also move three centimeters to the right, displaying the first virtual interface at the updated second display position.

[0098] In some embodiments, establishing a communication connection with a target terminal device includes:

[0099] Sending a broadcast message, which is used to request the establishment of a communication connection;

[0100] receiving a connection request sent by a target terminal device in response to the broadcast message;

[0101] In response to the user's confirming connection operation for the connection request, a communication connection is established with the target terminal device.

[0102] The extended reality device sends a broadcast message to scan the surrounding terminal devices for target terminal devices that can establish a communication connection. After receiving the broadcast message of the extended reality device, the target terminal device will respond to the broadcast message and send a connection request to the extended reality device. After receiving the connection request from the target terminal device, the extended reality device can ask the user whether to establish a communication connection in the form of a small window on the display screen. If the user takes a confirmation connection operation in response to the connection request, the target terminal device responds to the confirmation connection instruction sent by the extended reality device, the extended reality device establishes a communication connection with the target terminal device, and the first virtual interface is displayed on the display screen of the extended reality device; if the user takes a rejection connection operation in response to the connection request, the target terminal device responds to the rejection connection instruction sent by the extended reality device, the extended reality device does not establish a communication connection with the target terminal device, and the first virtual interface is not displayed on the display screen of the extended reality device.

[0103] If the extended reality device receives connection requests from multiple target terminal devices simultaneously, the user can select one or two of the target terminal devices to confirm the connection operation based on their personal needs. In addition, to simplify user operation, a small window may appear in the form of a prompt to the user to establish a communication connection with the target terminal device for the first time. For target terminal devices that are not the first time to connect, the extended reality device does not require the user to confirm the connection operation and automatically establishes a communication connection with the target terminal device after receiving the connection request.

[0104] In some embodiments, issuing a broadcast message includes:

[0105] Obtaining second image information captured by a camera of the extended reality device;

[0106] When it is determined based on the second image information that a terminal device is in use, a broadcast message is issued.

[0107] The XR device can use a camera to capture a second image in real time. This second image contains information about the environment in front of the user while wearing the XR device. The XR device uses this second image to scan for any active devices nearby. Specifically, it analyzes the second image to determine if any devices have active screens. If the second image determines that a device is in use, indicating that the user is using both the XR device and the device simultaneously, the XR device broadcasts a message to promptly display the extended screen on the device.

[0108] In addition, when the extended reality device is communicating with a target terminal device, the extended reality device can obtain second image information at preset intervals to confirm whether other terminal devices have the need for expanded screen display. Of course, the extended reality device can also choose not to obtain the second image information, not to send a broadcast message, and reject the communication connection with other terminal devices, and only display the first virtual interface generated by the connected target terminal device on the display screen.

[0109] In some embodiments, when it is determined based on the second image information that a terminal device is in use, a broadcast message is issued, including:

[0110] When there is a highlighted area in the second image information, it is determined that there is a terminal device in use, and a broadcast message is sent.

[0111] If the second image information has a highlighted area, that is, there is a bright screen, it means that there is a terminal device in use. In the image, the higher the brightness, the larger the grayscale value, so the extended reality device can determine whether there is a terminal device in use by analyzing the grayscale value in the second image information. If the grayscale value of a certain area in the second image information exceeds a preset value, there is a highlighted area, that is, there is a terminal device in use. For example, the user is using a smartphone and a smartwatch, and both the smartphone and the smartwatch are in the bright screen state. The extended reality device uses the camera to collect the second image information. The second image information will have two areas with grayscale values ​​higher than the preset value, that is, there are two highlighted areas, so the extended reality device sends a broadcast message to the smartphone and the smartwatch.

[0112] The second image information can be either an RGB image or a grayscale image. When the second image information is an RGB image, since the RGB image must be converted to a grayscale image, the data processing capacity of the extended reality device is higher than that of the grayscale image. Furthermore, the extended reality device can also use UWB technology to determine whether a terminal device is in use. UWB technology is currently available and will not be further described here.

[0113] In some embodiments, the interactive display method further includes:

[0114] Based on a first preset operation of the user on the first virtual interface, generating a control instruction corresponding to the first preset operation;

[0115] The control instruction is sent to the target terminal device so that the target terminal device executes the control instruction.

[0116] Because the extended reality device is a virtual screen, users cannot control it using the same touch controls they use to control the target terminal device. Therefore, the first preset operation is often a gesture. The extended reality device uses a camera to capture the user's first preset operation on the first virtual interface, identifies the control instruction corresponding to the first preset operation, and sends the generated control instruction to the target terminal device, enabling the target terminal device to execute the same control instruction, thereby changing the content of the first virtual interface on the display screen of the extended reality device.

[0117] In some embodiments, the interactive display method further includes:

[0118] Receiving a second virtual interface sent by the target terminal device, where the second virtual interface is generated by the target terminal device executing a control instruction;

[0119] The first virtual interface is updated to the second virtual interface.

[0120] The target terminal device executes the control instruction, generates another screen interface corresponding to the control instruction, and sends the newly generated another screen interface to the extended reality device. The extended reality device can then receive the second virtual interface and update the first virtual interface to the second virtual interface, completing the user's first preset operation on the first virtual interface. For example, the display content of the first virtual interface is the title of an article. The user uses a downward sliding gesture on the first virtual interface. After the extended reality device uses the camera to recognize the first preset operation, it generates a sliding instruction corresponding to the first preset downward sliding operation and sends the sliding instruction to the target terminal device. The target terminal device executes the sliding instruction, generates and sends the second virtual interface with the new display content after sliding to the extended reality device. Since the display content of the second virtual interface is a specific paragraph of the article, the display screen of the extended reality device changes to the second virtual interface, that is, changes from displaying the title of the article to displaying the specific paragraph of the article.

[0121] According to an exemplary embodiment, Figure 2 As shown, the interactive display method in this embodiment includes:

[0122] S201: Acquire second image information captured by a camera of an extended reality device;

[0123] S202: When the second image information contains a highlighted area, it is determined that a terminal device is in use, and a broadcast message is sent, where the broadcast message is used to request establishment of a communication connection;

[0124] S203, receiving a connection request sent by the target terminal device in response to the broadcast message;

[0125] S204: In response to the user's confirmation operation for the connection request, establishing a communication connection with the target terminal device;

[0126] S205: Receive first image information sent by the target terminal device, where the first image information includes a current screen display image of the target terminal device;

[0127] S206. Determine a first display position of the target terminal device in the display image based on the first image information;

[0128] S207: Determine a second display position of the first virtual interface in the display screen based on the first display position;

[0129] S208, displaying the first virtual interface at the second display position;

[0130] S209: In response to a change in the first display position of the target terminal device, updating the second display position of the first virtual interface based on the changed first display position;

[0131] S210: Based on a first preset operation of a user on a first virtual interface, generating a control instruction corresponding to the first preset operation;

[0132] S211, sending the control instruction to the target terminal device, so that the target terminal device executes the control instruction;

[0133] S212: Receive a second virtual interface sent by the target terminal device, where the second virtual interface is generated by the target terminal device executing a control instruction;

[0134] S213: Update the first virtual interface to a second virtual interface.

[0135] Regarding the interactive display method in the above embodiment, the specific manner of each step has been described in detail in the embodiment related to the step and will not be elaborated here.

[0136] In this disclosure, the communication connection between the extended reality device and the target terminal device enables the coordinated display of the extended reality device's virtual screen and the target terminal device's real screen. By locating the target terminal device's first display position, the extended reality device's large-screen display is linked to smart devices and smart appliances, meeting the user's multi-screen viewing needs. The user can interact with the extended reality device using first preset operations such as gestures to adjust the extended reality device's display, improving user convenience and playability.

[0137] The exemplary embodiment of the present disclosure provides an interactive display method, which is applied to a target terminal device. The target terminal device can be any electronic device with a screen, such as a smart phone, a smart watch, a tablet computer, a refrigerator with a screen, an air conditioner, etc. Figure 3 As shown, the interactive display method shown in the present disclosure includes:

[0138] S301, establishing a communication connection with an extended reality device;

[0139] S302: In response to a second preset operation of the user, generate a first virtual interface, and send the first virtual interface to the extended reality device so as to display the first virtual interface on a display screen of the extended reality device.

[0140] In step S301, there may be one or more target terminal devices, which establish a wireless or wired connection with the augmented reality device to achieve communication between the two parties. The wireless connection may be a Bluetooth connection, a ZigBee connection, a WiFi connection, an NFC connection, or the like.

[0141] In step 302, the user performs a second preset operation on the target terminal device, selecting the target terminal device's current screen display as the display content of the first virtual interface, thereby generating a first virtual interface. After the extended reality device establishes a communication connection with the target terminal device, the target terminal device transmits the first virtual interface to the extended reality device via the communication connection. The first virtual interface is then displayed on the extended reality device's display screen, achieving the effect of expanding the target terminal device's real screen. The user can view the content that should be displayed on the target terminal device on the extended reality device's display screen. The first virtual interface can be one or more.

[0142] For example, the target terminal devices include a smartphone, a smart watch, and a smart refrigerator with a screen. The extended reality device establishes communication connections with the smartphone, smart watch, and smart refrigerator, and receives multiple first virtual interfaces sent by the three. The first virtual interface sent by the smartphone is the product details of the shopping application, the first virtual interface sent by the smart watch is the user's health information, and the first virtual interface sent by the smart refrigerator is the refrigerator's refrigeration information. The extended reality device will simultaneously display the product details, health information, and refrigerator's refrigeration information that the user wants to purchase on the display screen.

[0143] In some embodiments, the interactive display method further includes:

[0144] In response to the broadcast message sent by the extended reality device, sending a connection request to the extended reality device;

[0145] In response to the confirmation connection instruction sent by the extended reality device, a communication connection is established with the extended reality device.

[0146] In some embodiments, the interactive display method further includes:

[0147] First image information is sent to the extended reality device, so that the extended reality device determines a first display position of the target terminal device in the display screen according to the first image information, where the first image information includes a current screen display screen of the target terminal device.

[0148] In some embodiments, the interactive display method further includes:

[0149] Receive control instructions sent by the extended reality device;

[0150] Executing the control instruction to generate a second virtual interface;

[0151] The second virtual interface is sent to the extended reality device, so that the extended reality device updates the first virtual interface to the second virtual interface.

[0152] The specific methods of the above embodiments have been described in detail in the corresponding embodiments of the extended reality device side, and will not be elaborated here.

[0153] According to an exemplary embodiment, Figure 4 As shown, the interactive display method in this embodiment includes:

[0154] S401: In response to a broadcast message sent by an extended reality device, sending a connection request to the extended reality device;

[0155] S402: Establishing a communication connection with the extended reality device in response to a connection confirmation instruction sent by the extended reality device;

[0156] S403: Sending first image information to the extended reality device, so that the extended reality device determines a first display position of the target terminal device in the display screen according to the first image information;

[0157] S404: In response to a second preset operation of the user, generate a first virtual interface, and send the first virtual interface to the extended reality device so as to display the first virtual interface on a display screen of the extended reality device;

[0158] S405: Receive a control instruction sent by the extended reality device;

[0159] S406, executing the control instruction to generate a second virtual interface;

[0160] S407: Send the second virtual interface to the extended reality device, so that the extended reality device updates the first virtual interface to the second virtual interface.

[0161] Regarding the interactive display method in the above embodiment, the specific method of each step has been described in detail in the embodiment of the corresponding extended reality device side of the step, and will not be elaborated here.

[0162] The interactive method disclosed herein not only does not hinder the use of the target terminal device, but also brings benefits to the use of the target terminal device. The communication connection between the target terminal device and the extended reality device achieves an expansion effect on the target terminal device's real screen, solving the problem of the two devices, such as smartphones, being independent of each other when the user wears the extended reality device all day.

[0163] The exemplary embodiment of the present disclosure provides an interactive display device, which is applied to an extended reality device. Figure 6 As shown, a block diagram of an interactive display device shown in the present disclosure.

[0164] The block diagram includes: a first connection module 61, which is used to establish a communication connection with a target terminal device; and a first execution module 62, which is used to display a first virtual interface on the display screen of the extended reality device based on the communication connection. The first virtual interface is generated by the target terminal device.

[0165] In an exemplary embodiment of the present disclosure, the first execution module 62 is also used to: determine a first display position of the target terminal device in the display screen; based on the first display position, determine a second display position of the first virtual interface in the display screen; and display the first virtual interface at the second display position.

[0166] In an exemplary embodiment of the present disclosure, the first execution module 62 is further configured to: receive first image information sent by the target terminal device, the first image information including the current screen display image of the target terminal device; and determine a first display position based on the first image information.

[0167] In an exemplary embodiment of the present disclosure, the first execution module 62 is further configured to: in response to a change in the first display position of the target terminal device, update the second display position of the first virtual interface based on the changed first display position.

[0168] In an exemplary embodiment of the present disclosure, the first connection module 61 is also used to: send a broadcast message, which is used to request the establishment of a communication connection; receive a connection request sent by the target terminal device in response to the broadcast message; and establish a communication connection with the target terminal device in response to the user's confirmation connection operation for the connection request.

[0169] In an exemplary embodiment of the present disclosure, the first connection module 61 is further used to: obtain second image information captured by the camera of the extended reality device; and send a broadcast message when it is determined based on the second image information that a terminal device is in use.

[0170] In an exemplary embodiment of the present disclosure, the first connection module 61 is further configured to: when there is a highlighted area in the second image information, determine that there is a terminal device in use, and send a broadcast message.

[0171] In an exemplary embodiment of the present disclosure, the first execution module 62 is also used to: generate a control instruction corresponding to the first preset operation based on the user's first preset operation on the first virtual interface; and send the control instruction to the target terminal device so that the target terminal device executes the control instruction.

[0172] In an exemplary embodiment of the present disclosure, the first execution module 62 is further configured to: receive a second virtual interface sent by the target terminal device, where the second virtual interface is generated by the target terminal device executing a control instruction; and update the first virtual interface to the second virtual interface.

[0173] Regarding the interactive display device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0174] The exemplary embodiment of the present disclosure provides an interactive display device, which is applied to a target terminal device. Figure 7 As shown, a block diagram of an interactive display device shown in the present disclosure.

[0175] The block diagram includes a second connection module 71 and a second execution module 72. The second connection module 71 is configured to establish a communication connection with the extended reality device; the second execution module 72 is configured to generate a first virtual interface in response to a second preset user operation, and transmit the first virtual interface to the extended reality device for display on the display screen of the extended reality device.

[0176] In an exemplary embodiment of the present disclosure, the second execution module 72 is also used to: send first image information to the extended reality device, so that the extended reality device determines the first display position of the target terminal device in the display screen based on the first image information, and the first image information includes the current screen display screen of the target terminal device.

[0177] In an exemplary embodiment of the present disclosure, the second connection module 71 is further configured to: send a connection request to the extended reality device in response to a broadcast message sent by the extended reality device; and establish a communication connection with the extended reality device in response to a confirmation connection instruction sent by the extended reality device.

[0178] In an exemplary embodiment of the present disclosure, the second execution module 72 is also used to: receive control instructions sent by the extended reality device; execute the control instructions to generate a second virtual interface; and send the second virtual interface to the extended reality device so that the extended reality device updates the first virtual interface to the second virtual interface.

[0179] Regarding the interactive display device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0180] Figure 8 FIG1 is a block diagram of an extended reality device 800 according to an exemplary embodiment. For example, the extended reality device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0181] Reference Figure 8 , the extended reality device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0182] The processing component 802 generally controls the overall operation of the extended reality device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0183] The memory 804 is configured to store various types of data to support operations on the extended reality device 800. Examples of such data include instructions for any application or method operating on the extended reality device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0184] The power supply component 806 provides power to the various components of the extended reality device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the extended reality device 800.

[0185] The multimedia component 808 includes a screen that provides an output interface between the extended reality device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the extended reality device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0186] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the extended reality device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0187] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0188] The sensor assembly 814 includes one or more sensors for providing various aspects of the state assessment of the extended reality device 800. For example, the sensor assembly 814 can detect the open / closed state of the extended reality device 800, the relative positioning of components, such as the display and keypad of the extended reality device 800. The sensor assembly 814 can also detect changes in the position of the extended reality device 800 or a component of the extended reality device 800, the presence or absence of user contact with the extended reality device 800, the orientation or acceleration / deceleration of the extended reality device 800, and changes in the temperature of the extended reality device 800. The sensor assembly 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 can also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0189] The communication component 816 is configured to facilitate wired or wireless communication between the extended reality device 800 and other devices. The extended reality device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0190] In an exemplary embodiment, the extended reality device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described methods.

[0191] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions. The instructions can be executed by the processor 820 of the augmented reality device 800 to perform the above-described noise reduction method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0192] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an extended reality device, enables a processing device of the extended reality device to perform the interactive display method provided by an exemplary embodiment of the present disclosure.

[0193] A computer program product includes computer instructions stored in a computer-readable storage medium; a processor of an extended reality device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the extended reality device to perform the interactive display method of messages provided in an exemplary embodiment of the present disclosure.

[0194] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a target terminal device, enables a processing device of the target terminal device to execute the interactive display method provided by an exemplary embodiment of the present disclosure.

[0195] A computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a target terminal device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the target terminal device executes the interactive display method of messages provided by the exemplary embodiment of the present disclosure.

[0196] Those skilled in the art will readily recognize other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.

[0197] It will be understood that the present disclosure is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. An interactive display method, characterized in that: Applied to an extended reality device, the interactive display method includes: Establishing a communication connection with the target terminal device; Based on the communication connection, a first virtual interface is displayed in the display screen of the extended reality device, where the first virtual interface is generated by the target terminal device.

2. The interactive display method according to claim 1, characterized in that: Displaying the first virtual interface on the display screen of the extended reality device includes: Determining a first display position of the target terminal device in the display image; Based on the first display position, determining a second display position of the first virtual interface in the display screen; The first virtual interface is displayed at the second display position.

3. The interactive display method according to claim 2, characterized in that: The determining a first display position of the target terminal device in the display screen includes: Receiving first image information sent by the target terminal device, where the first image information includes a current screen display image of the target terminal device; The first display position is determined based on the first image information.

4. The interactive display method according to claim 2, characterized in that: The interactive display method further includes: In response to a change in the first display position of the target terminal device, the second display position of the first virtual interface is updated based on the changed first display position.

5. The interactive display method according to claim 1, characterized in that: The establishing of a communication connection with the target terminal device includes: Sending a broadcast message, wherein the broadcast message is used to request establishment of a communication connection; receiving a connection request sent by the target terminal device in response to the broadcast message; In response to a user confirming a connection operation for the connection request, a communication connection is established with the target terminal device.

6. The interactive display method according to claim 5, characterized in that: The sending of the broadcast message comprises: Obtaining second image information captured by a camera of the extended reality device; When it is determined based on the second image information that a terminal device is in use, the broadcast message is sent out.

7. The interactive display method according to claim 6, characterized in that: When it is determined according to the second image information that a terminal device is in use, sending the broadcast message includes: When there is a highlighted area in the second image information, it is determined that there is a terminal device in use, and the broadcast message is sent.

8. The interactive display method according to any one of claims 1 to 7, characterized in that: The interactive display method further includes: Based on a first preset operation of the user on the first virtual interface, generating a control instruction corresponding to the first preset operation; The control instruction is sent to the target terminal device, so that the target terminal device executes the control instruction.

9. The interactive display method according to claim 8, characterized in that: The interactive display method further includes: receiving a second virtual interface sent by the target terminal device, where the second virtual interface is generated by the target terminal device executing the control instruction; The first virtual interface is updated to the second virtual interface.

10. An interactive display method, characterized in that: Applied to a target terminal device, the interactive display method includes: Establishing a communication connection with an extended reality device; In response to a second preset operation of the user, a first virtual interface is generated, and the first virtual interface is sent to the extended reality device so as to be displayed on a display screen of the extended reality device.

11. The interactive display method according to claim 10, characterized in that: The interactive display method further includes: First image information is sent to the extended reality device so that the extended reality device determines a first display position of the target terminal device in the display screen according to the first image information, wherein the first image information includes a current screen display screen of the target terminal device.

12. The interactive display method according to claim 10, characterized in that: The interactive display method further includes: In response to the broadcast message sent by the extended reality device, sending a connection request to the extended reality device; In response to the confirmation connection instruction sent by the extended reality device, a communication connection is established with the extended reality device.

13. The interactive display method according to claim 10, characterized in that: The interactive display method further includes: Receiving a control instruction sent by the extended reality device; Executing the control instruction to generate a second virtual interface; The second virtual interface is sent to the extended reality device, so that the extended reality device updates the first virtual interface to the second virtual interface.

14. An interactive display device, characterized in that: Applied to an extended reality device, the interactive display device includes: A first connection module, configured to establish a communication connection with a target terminal device; A first execution module is configured to display a first virtual interface in a display screen of the extended reality device based on the communication connection, where the first virtual interface is generated by the target terminal device.

15. An interactive display device, characterized in that: Applied to a target terminal device, the interactive display device includes: A second connection module is used to establish a communication connection with the extended reality device; The second execution module is used to generate a first virtual interface in response to a second preset operation of the user, and send the first virtual interface to the extended reality device to display the first virtual interface on the display screen of the extended reality device.

16. An extended reality device, characterized in that The extended reality device includes: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions in the memory to implement the interactive display method according to any one of claims 1 to 9.

17. A target terminal device, characterized in that: The target terminal device includes: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions in the memory to implement the interactive display method according to any one of claims 10 to 13.

18. A non-transitory computer-readable storage medium, characterized in that Executable instructions are stored thereon, and when the executable instructions are executed by the processor of the extended reality device, the interactive display method as described in any one of claims 1 to 9 is implemented.

19. A non-transitory computer-readable storage medium, characterized in that Executable instructions are stored thereon, and when the executable instructions are executed by a processor of a target terminal device, the interactive display method according to any one of claims 10 to 13 is implemented.

20. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the processor of the extended reality device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the extended reality device performs the interactive display method as described in any one of claims 1 to 9.

21. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the processor of the target terminal device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the target terminal device executes the interactive display method as described in any one of claims 10 to 13.

Images