Device interaction method, wearable device, storage medium and program product

Abstract

The invention discloses a device interaction method, a wearable device, a storage medium and a program product, and belongs to the technical field of intelligent wearable devices.The device interaction method comprises the steps that under the condition that an external environment image captured by the wearable device comprises a first display interface of an external device, the first display interface of the external device is displayed on the wearable device; establishing a communication connection between the wearable device and the external device; creating a virtual interface corresponding to the first display interface when the wearable device is in a state that the video perspective function is turned on; and the wearable device receives a first input of a user through the virtual interface, and generates first control information for controlling the external device.

Description

Technical Field

[0001] The present application relates to the technical field of smart wearable devices, and in particular to a device interaction method, a wearable device, a storage medium, and a program product. Background Art

[0002] With the development of smart wearable devices, the functions of smart wearable devices are becoming more and more perfect, and the usage scenarios are becoming more and more abundant. In the related art, users wearing head-mounted wearable devices may need to temporarily perform some operations on a nearby personal computer (PC) in some scenarios. When a user wearing a head-mounted wearable device needs to operate the PC, it is usually necessary to take off the head-mounted wearable device first, then operate the PC, and then put on the head-mounted wearable device again after the operation is completed. In this process, since the user needs to take off and put on the head-mounted wearable device again, the operation process is relatively cumbersome. It can be seen that in the related art, there is a problem of cumbersome operation process when a user wearing a head-mounted wearable device operates the PC. Summary of the Invention

[0003] The embodiments of the present application provide a device interaction method, a wearable device, a storage medium, and a program product, which can solve the problem that when a user wearing a head-mounted wearable device operates a PC, the operation process is relatively cumbersome.

[0004] In a first aspect, a device interaction method is provided, which is applied to a wearable device with a video see-through function, the method comprising:

[0005] In a case where the external environment image captured by the wearable device includes a first display interface of an external device, establishing a communication connection between the wearable device and the external device;

[0006] When the wearable device is in a state where the video see-through function is turned on, creating a virtual interface corresponding to the first display interface;

[0007] The wearable device receives a first input from a user through the virtual interface and generates first control information for controlling the external device.

[0008] In a second aspect, a wearable device is provided, comprising:

[0009] a connection module, configured to establish a communication connection between the wearable device and the external device when the external environment image captured by the wearable device includes a first display interface of the external device;

[0010] A creation module, configured to create a virtual interface corresponding to the first display interface when the video see-through function of the wearable device is turned on;

[0011] A generating module is used for the wearable device to receive a first input from a user through the virtual interface and generate first control information for controlling the external device.

[0012] In a third aspect, an embodiment of the present application provides a wearable device, which includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0015] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0016] In an embodiment of the present application, when an image of an external environment captured by a wearable device includes a first display interface of an external device, a communication connection is established between the wearable device and the external device, and a virtual interface corresponding to the first display interface is created when the wearable device is in a state where a video see-through function is enabled. In this way, when a user wearing the wearable device needs to operate the external device, since the user can directly see the content displayed on the display screen through the video see-through function, and the virtual interface corresponds to the first display interface, the user can directly input operation information for the external device through the virtual interface. The wearable device generates corresponding first control information based on the operation information input by the user on the virtual interface, and sends the first control information to the external device to control the external device, so that the external device performs the operation corresponding to the user input in response to the first control information. In this process, since the user can directly operate the external device while wearing the wearable device, it is helpful to simplify the operation process of the external device by the user wearing the wearable device. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is one of the flow diagrams of a device interaction method provided in an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of the field of view of a user wearing a wearable device in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a user selecting point P in the first display interface based on a virtual ray in an embodiment of the present application;

[0020] Figure 4 Schematic diagram of three ways of triggering virtual rays in an embodiment of the present application;

[0021] Figure 5 This is a flow chart of a user operating a PC operating interface in an embodiment of the present application;

[0022] Figure 6 This is a schematic diagram of a control system provided by an embodiment of the present application;

[0023] Figure 7 This is a second flow chart of a device interaction method provided in an embodiment of the present application;

[0024] Figure 8 This is one of the structural diagrams of a wearable device provided in an embodiment of the present application;

[0025] Figure 9 This is a second structural diagram of a wearable device provided in an embodiment of the present application;

[0026] Figure 10 A schematic diagram of the hardware structure of a wearable device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0028] The terms "first," "second," and the like in the specification of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification indicates at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0029] Below, in conjunction with the accompanying drawings, a device interaction method, wearable device, storage medium and program product provided by the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0030] See Figure 1 , Figure 1 A schematic diagram of a flow chart of a device interaction method provided in an embodiment of the present application, wherein the device interaction method is applied to a wearable device with a video see-through (VST) function, and the device interaction method comprises the following steps:

[0031] Step 101: When the external environment image captured by the wearable device includes a first display interface of an external device, establish a communication connection between the wearable device and the external device;

[0032] Step 102: When the wearable device is in a state where the video see-through function is turned on, create a virtual interface corresponding to the first display interface;

[0033] Step 103: The wearable device receives a first input from the user through the virtual interface and generates first control information for controlling the external device.

[0034] The wearable device can be any head-mounted smart wearable device with a video see-through function, for example, any extended reality (XR) head-mounted product or artificial intelligence (AI) glasses with a video see-through function. For ease of understanding, the following uses an XR device as an example to further explain the method provided in the embodiment of the present application. The external device can be a PC device near the user.

[0035] It can be understood that, in the case where the external environment image captured by the wearable device includes the first display interface of the external device, establishing a communication connection between the wearable device and the external device may refer to: when the user wears the wearable device, the wearable device turns on the video perspective function, and the field of view of the wearable device includes the first display interface of the external device, establishing a communication connection with the external device. The communication connection may be a local area network connection or a Bluetooth connection. The field of view may be the range of the external environment that the user wearing the wearable device sees through the video perspective function. For example, see Figure 2 , Figure 2 The shaded area in the figure represents the field of view.

[0036] The above-mentioned video perspective function may refer to a function that enables users to view augmented reality content through wearable devices by superimposing virtual images on a real-time video stream in the real world.

[0037] The virtual interface may be invisible to the user, or the virtual interface may be a semi-transparent floating window, so that the user can view the display content in the first display interface through the virtual interface.

[0038] The first input can be various types of input that a user can perform based on the wearable device. For example, the first input can be input based on a handle that is communicatively connected to the wearable device, or eye movement input, or gesture input, etc. It is understandable that when performing the first input, the user does not need to actually touch the external device, the first display interface, the mouse of the external device, or other physical objects related to the external device.

[0039] The first control information may include the user's interaction position in the first display interface, as well as an interaction event, wherein the interaction event may be an interaction event in various scenarios, such as a single-click event, a double-click event, a drag event, a long-press event, etc. The correspondence between various interaction actions and interaction events may be preset. In this way, the user may input a corresponding interaction action through the first input to generate first control information including the expected interaction event, and by sending the first control information to the external device, the external device performs an operation corresponding to the preset interaction event, thereby realizing the user's operation process on the external device while wearing the wearable device.

[0040] In this embodiment, when the external environment image captured by the wearable device includes a first display interface of the external device, a communication connection is established between the wearable device and the external device, and when the wearable device is in a state where the video perspective function is turned on, a virtual interface corresponding to the first display interface is created. In this way, when a user wearing the wearable device needs to operate the electronic device, since the user can directly see the content displayed on the first display interface through the video perspective function, and the virtual interface corresponds to the first display interface, the user can directly input operation information for the electronic device through the virtual interface. The wearable device generates corresponding first control information based on the operation information input by the user on the virtual interface, and sends the first control information to the electronic device, so that the electronic device responds to the first control information and performs the operation corresponding to the user input. In this process, since the user can directly operate the electronic device while wearing the wearable device, it is helpful to simplify the operation process of the electronic device by the user wearing the wearable device.

[0041] Optionally, when the wearable device displays the external environment image through the video perspective function, the virtual interface overlaps with the first display interface.

[0042] The shape and size of the virtual interface are the same as the display area of the first display interface. In this way, the virtual interface can completely cover the display area of the first display interface. Figure 3 FIG. 3 is a schematic diagram of the modeled virtual interface 301 , wherein the four vertex corners P0 , P1 , P2 and P3 of the virtual interface 301 coincide with the four vertex corners of the display area of the first display interface 302 .

[0043] In this embodiment, since the user can see the real picture of the outside world through the video perspective function, and the position of the virtual interface in the field of view coincides with the position of the first display interface in the field of view, for the user, the input in the virtual interface is the input in the first display interface in the real picture, so that the user can operate the external device more intuitively and naturally, which is conducive to improving the user experience of the operation process.

[0044] Optionally, when the external environment image captured by the wearable device includes a first display interface of an external device, establishing a communication connection between the wearable device and the external device includes:

[0045] The wearable device turns on the video perspective function to capture the external environment image, and when the external environment image includes the first display interface, establishes a communication connection between the wearable device and the external device.

[0046] Specifically, after the user turns on the video perspective function of the wearable device, the camera management module in the wearable device can continuously obtain images captured by the video perspective camera, and input the acquired images into the AI module, and use the AI algorithm to identify whether the received images include the image of the first display interface, and output the corresponding recognition result. The external environment image includes the image captured by the camera of the wearable device. The camera can be the red, green, and blue (RGB) camera of the video perspective module itself, or it can be other cameras of the wearable device.

[0047] The above-mentioned establishing a communication connection with the external device includes:

[0048] When scanning is performed based on the wireless network module or the Bluetooth module of the wearable device and the external device is scanned, a communication connection with the external device is established.

[0049] In some embodiments of the present application, when the above recognition result indicates that the first display interface of the external device is included in the field of view of the wearable device, the wireless network module of the wearable device can be used to scan within the Wi-Fi local area network to determine whether the local area network includes the external device. If so, it is determined whether the external device is a device that the wearable device has paired with. If so, a connection request is directly sent to the external device to establish a wireless connection between the wearable device and the external device. If not, a pairing request is first sent to the external device to perform pairing, and then a wireless connection is established, and the pairing information is saved.

[0050] In some embodiments of the present application, when the above recognition result indicates that the first display interface of the external device is included in the field of view of the wearable device, the Bluetooth module of the wearable device can be used to scan to determine whether the external device is included in the scanning range. If so, it is determined whether the external device is a device that the wearable device has paired with. If so, a connection request is directly sent to the external device to establish a Bluetooth connection between the wearable device and the external device. If not, a pairing request is first sent to the external device to perform pairing first, and then a Bluetooth connection is established, and the pairing information is saved.

[0051] In this embodiment, the wearable device turns on the video perspective function to capture the external environment image, and when the external environment image includes the first display interface, a communication connection is established between the wearable device and the external device, that is, when the first display interface appears in the field of view, the wearable device actively establishes a communication connection with the external device of the first display interface. In this way, when the user subsequently needs to operate the external device, there is no need to manually establish a connection between the wearable device and the external device, thereby further improving the user's operating experience of the external device.

[0052] Optionally, the creating a virtual interface corresponding to the first display interface includes:

[0053] Acquiring external information collected by the wearable device, wherein the external information includes modeling information used to model the first display interface;

[0054] The virtual interface is created based on the external information.

[0055] Among them, the external information may be information such as the shape, size and location of the first display interface, and the virtual interface overlaps with the first display interface. Therefore, the virtual interface can be modeled based on information such as the shape, size and location of the first display interface in the external information to realize the creation process of the virtual interface.

[0056] In this embodiment, the virtual interface creation process can be achieved by acquiring the external information collected by the wearable device and creating the virtual interface based on the external information.

[0057] Optionally, the external information includes an image of the external environment captured by a camera in the wearable device; or

[0058] The external information includes: an external environment image collected by a camera in the wearable device, and depth information corresponding to the external environment image collected by a depth sensor in the wearable device.

[0059] Among them, the wearable device usually includes multiple RGB cameras. Therefore, the external information may include the external environment image captured by each RGB camera in the wearable device. It is understandable that the external environment image includes the image content of the first display interface. The depth sensor can be any common sensor for users to collect depth information of the screen, for example, it can be a time-of-flight (TOF) sensor.

[0060] Since a wearable device may or may not include a TOF sensor, when the wearable device includes a TOF sensor, the external information includes: an image of the external environment captured by a camera, and depth information corresponding to the image of the external environment captured by a depth sensor. Correspondingly, when the wearable device does not include a TOF sensor, the external information includes an image of the external environment captured by a camera.

[0061] In this embodiment, since the external information includes an image of the external environment captured by the camera in the wearable device; or the external information includes: an image of the external environment captured by the camera in the wearable device, and depth information corresponding to the image of the external environment captured by the depth sensor in the wearable device, and the virtual interface overlaps with the first display interface, the first display interface can be modeled based on the shape, size, position, and other information of the first display interface in the external environment image. Alternatively, the first display interface can be modeled based on the shape, size, position, and other information of the first display interface in the external environment image, and the distance information between the wearable device and the first display interface in the depth information. This enables the creation process of the first display interface.

[0062] Optionally, the creating the virtual interface based on the external information includes:

[0063] Inputting the external information into a three-dimensional reconstruction model for three-dimensional reconstruction to obtain coordinate information output by the three-dimensional reconstruction model, wherein the coordinate information includes position coordinates corresponding to four vertices of the first display interface;

[0064] The virtual interface is created based on the coordinate information.

[0065] For ease of understanding, the embodiment of the present application takes the example of the external information including: the external environment image captured by the camera in the wearable device, and the depth information corresponding to the external environment image captured by the depth sensor in the wearable device, to further explain the process of creating the virtual interface:

[0066] The creation of the virtual interface based on the external information may include: inputting the RGB image and TOF depth information into a three-dimensional (3D) reconstruction algorithm to calculate the plane of the first display interface in 3D space, as well as the coordinates of the four corners of the first display interface in 3D space, namely, the upper left corner P0 (x0, y0, z0), the upper right corner P1 (x1, y1, z1), the lower left corner P2 (x2, y2, z2), and the lower right corner P3 (x3, y3, z3). The 3D rendering engine reads the 3D plane of the first display interface calculated by the 3D reconstruction algorithm and establishes a model of the 3D plane in the virtual space. The established 3D plane model is the virtual interface. The RGB image is the external environment image, and the TOF depth information is the depth information corresponding to the external environment image. The 3D space may be a three-dimensional coordinate system space pre-established in the field of view. The virtual space is the space corresponding to the field of view.

[0067] In this embodiment, the external information is input into a three-dimensional reconstruction model for three-dimensional reconstruction, coordinate information output by the three-dimensional reconstruction model is obtained, and the virtual interface is created based on the coordinate information, thereby realizing the creation process of the virtual interface.

[0068] Optionally, the first input includes at least one of the following input modes: eye movement input, gesture input, and input based on a handle communicatively connected to the wearable device.

[0069] It is understandable that the user can choose to use at least one of the following input methods: eye movement input, gesture input, and input based on a handle that is communicatively connected to the wearable device to complete the first input according to their own preferences. For example, the user can select the location where interaction is required through eye movement input, and then input the interaction event through gesture input to implement the input process of the first input. In this case, the first input includes eye movement input and gesture input. For another example, the user can select the location where interaction is required by pointing a finger, and then input the interaction event through gesture input. In this case, the first input only includes gesture input. For another example, the user can select the location where interaction is required through eye movement input, and then input the interaction event through handle input. In this case, the first input only includes eye movement input and handle input. The user can select the corresponding input method according to their needs, and there is no limitation to this.

[0070] See Figure 4 (a) In some embodiments of the present application, the user can perform the above-mentioned first input through hand-eye interaction. In this case, the first input includes eye movement input and gesture input. Figure 4(b) In some embodiments of the present application, the user may perform the first input by means of gesture input. Figure 4 (c) In some embodiments of the present application, the user may perform the above-mentioned first input by inputting through the handle 401.

[0071] In this embodiment, since the first input includes at least one of the following input methods: eye movement input, gesture input, and input based on a handle connected to the communication with the wearable device, the user can choose to use at least one of eye movement, gesture and handle to perform the above-mentioned first input according to his or her own preferences, which is conducive to facilitating the user to perform the above-mentioned first input in the virtual interface, thereby improving the user experience of the operation process.

[0072] Optionally, the generating the first control information includes:

[0073] Determining, based on the first input, an interaction position and an interaction event of the first input in the plane where the virtual interface is located;

[0074] The first control information is generated based on the interaction position and the interaction event.

[0075] The interaction event may be an interaction event in various scenarios, such as a single-click event, a double-click event, a drag event, a long-press event, etc. The correspondence between various interaction actions and interaction events may be preset, so that the user may input the corresponding interaction action through the first input to generate first control information including the expected interaction event, and send the first control information to the external device to cause the external device to perform the operation corresponding to the preset interaction event, thereby realizing the user's operation process on the external device while wearing the wearable device.

[0076] In this embodiment, since the object to be operated by the first input can be determined according to the interaction position, and the operation to be performed by the first input can be determined according to the interaction event, the first control information is generated based on the interaction position and the interaction event. In this way, the external device can perform the operation indicated by the first input on the operation object indicated by the first input according to the first control information, thereby realizing the operation process of the external device when wearing the wearable device.

[0077] Optionally, determining an interaction position of the first input in the plane where the virtual interface is located based on the first input includes:

[0078] Determining an extension direction of a virtual ray based on the indication direction of the first input, and determining an interaction position of the first input in the plane where the virtual interface is located; the interaction position is an intersection point of the virtual ray and the plane where the virtual interface is located;

[0079] In the case where the first input is an eye movement input, the extension direction of the virtual ray is the gaze direction of the user;

[0080] In the case where the first input is a gesture input, the extension direction of the virtual ray is the direction of the first position of the user's hand;

[0081] When the first input is a handle input, the extension direction of the virtual ray is the direction of the handle.

[0082] Specifically, the virtual ray can be triggered by the user based on eye movement input, or by the user based on gesture input, or by the user based on a handle. Before performing the above-mentioned first input, the user can first trigger the virtual ray to select the location that needs to be interacted in the first display interface. For example, see Figure 4 (a), the user can gaze at position point P' in the first display interface through eye movement input. When the wearable device detects that there is an intersection between the user's gaze direction and the virtual interface 403, it determines that the user needs to operate the external device. At this time, a virtual ray 402 can be generated along the user's gaze direction. In this way, the user can move the intersection position between the virtual ray 402 and the virtual interface 403 through eye movement input, thereby realizing the selection of different positions in the virtual interface 403. It can be understood that since the virtual interface 403 overlaps with the display area of the first display interface, the position selected by the user based on the virtual ray 402 can be determined as the position selected by the user in the first display interface.

[0083] For example, see Figure 4(b) The user can turn the palm of his hand toward the first display interface. In this way, when the wearable device detects that there is an intersection between the direction of the user's palm and the virtual interface 403, it is determined that the user needs to operate the external device. At this time, a virtual ray 402 can be generated along the direction of the user's palm. In this way, the user can move the intersection position between the virtual ray 402 and the virtual interface 403 by moving his palm, thereby realizing the selection of different positions in the virtual interface 403. It can be understood that since the virtual interface 403 overlaps with the display area of the first display interface, the position selected by the user based on the virtual ray 402 can be determined as the position selected by the user in the first display interface, wherein the palm of the user is the above-mentioned first position. In addition, in other embodiments of the present application, the user can also trigger the virtual ray 402 by pointing a finger. At this time, the first position is the user's finger.

[0084] For example, see Figure 4 (c) The user can also point the handle 401 toward the first display interface. In this way, when the wearable device detects that there is an intersection between the direction of the handle 401 and the virtual interface 403, it is determined that the user needs to operate the external device. At this time, a virtual ray 402 can be generated along the direction of the handle 401. In this way, the user can move the handle 401 to move the intersection position between the virtual ray 402 and the virtual interface 403, thereby realizing the selection of different positions in the virtual interface 403. It can be understood that since the virtual interface 403 overlaps with the display area of the first display interface, the position selected by the user based on the virtual ray 402 can be determined as the position selected by the user in the first display interface, wherein the palm of the user is the above-mentioned first position.

[0085] In this embodiment, the process of determining the interaction position can be achieved by determining the extension direction of the virtual ray based on the indication direction of the first input, and determining the interaction position of the first input in the plane where the virtual interface is located.

[0086] Optionally, determining an interaction position and an interaction event of the first input in the plane where the virtual interface is located based on the first input includes:

[0087] Determining coordinate information of an interaction position of the first input in the plane where the virtual interface is located;

[0088] Determining the interaction event based on the first input, where the interaction event includes an eye movement, a gesture, or a handle interaction;

[0089] The generating the first control information based on the interaction position and the interaction event includes: generating the first control information based on coordinate information of the interaction position and the interaction event.

[0090] It can be understood that after the user selects the interaction position based on the above-mentioned virtual ray, the interaction management module of the wearable device can project the coordinates P0, P1, P2, P3 of the four vertex corners of the first display interface and the above-mentioned intersection coordinate P into the same coordinate system XOY, and then calculate the relative position of the intersection and the coordinate point P, (x,, y,) in the upper left corner of the first display interface based on the relative position relationship, and determine the relative position as the above-mentioned interaction position. Then, read the interaction event of the gesture or handle. And transmit the interaction event and the calculated relative position information to the external device through the interconnection assistant module in the wearable device. In this way, the external device can parse the received first control information to determine the user's interaction position and interaction event, and can determine the corresponding interaction position in the first display interface, and execute the interaction event to realize the user's operation process on the external device.

[0091] See Figure 5 (a) and 5(b), in some embodiments of the present application, the user can select a control 502 in the PC operation interface 501 through eye movement input, so that the mouse icon of the external device hovers to the position of the control 502 selected by the user, and then, see Figure 5 (c) By pinching twice in succession, a double-click input can be performed on the selected control 502 to implement the operation process of the external device.

[0092] The generating of the first control information based on the coordinate information of the interaction position and the interaction event may refer to encoding the coordinate information of the interaction position and the interaction event together, and determining the encoded information as the first control information.

[0093] In this embodiment, by determining the coordinate information of the interaction position of the first input in the plane where the virtual interface is located, and generating the first control information based on the coordinate information of the interaction position and the interaction event, after receiving the first control information, the external device can determine the object to be operated according to the coordinate information, and can determine the operation to be performed by the first input according to the interaction event, so that the operation indicated by the first input can be performed on the operation object indicated by the first input according to the first control information, thereby realizing the operation process of the external device when wearing a wearable device.

[0094] Optionally, the method further includes:

[0095] The external device receives the first control information, and based on the first control information, the external device obtains interaction information corresponding to the interaction event and coordinate information of an associated position in the first display interface corresponding to the coordinate information of the interaction position;

[0096] The external device performs an operation based on the coordinate information of the associated position and the interaction information.

[0097] After the user selects the interaction location in the first display interface based on the virtual ray, the user can input the interaction action corresponding to the interaction event based on eye movement input, gesture input, or handle input. For example, when the user blinks once, the corresponding interaction event can be a click event, and when the user blinks twice, the corresponding interaction event can be a double-click event. Please see Table 1 below for a schematic diagram of the corresponding relationship between interaction actions and interaction events in some embodiments of the present application:

[0098] Table 1:

[0099]

[0100] The “pinching” may be performed by pinching the user’s thumb and index finger, for example, Figure 5 (c). "Pinch and drag" may refer to a user pinching and then moving their entire hand. The aforementioned "press and hold the trigger and move" may refer to a user pressing and holding the trigger of a controller and then moving the entire controller.

[0101] In some embodiments of the present application, the method further includes: generating second control information when the intersection point of the virtual ray and the virtual interface moves within the virtual interface, wherein the second control information includes an end point position after the intersection point moves;

[0102] The second control information is sent to the external device to control the first marker displayed in the first display interface to move to a position corresponding to the end position.

[0103] The first identifier may be an identifier of an object selected by the user to be operated on the external device. For example, the first identifier may be a mouse icon. It is understood that when the user controls the movement of the virtual ray relative to the virtual interface, the XR device may continuously generate and send second control information to the external device. In this way, the external device may move the position of the first identifier in the first display interface based on the received second control information, so that the position of the first identifier in the first display interface remains consistent with the position of the intersection.

[0104] The controlling the first marker displayed in the first display interface to move to a position corresponding to the end position may refer to controlling the first marker displayed in the first display interface to move to a position overlapping with the intersection.

[0105] Among them, the external device can continuously receive data sent by the XR device through the interconnection module, and input the decoded interaction information into the driver module to convert the interaction information into mouse events and report it to the operating system, so that the application (Application, APP) in the external device can respond to the user's interaction actions. When the virtual ray in the XR device moves in different areas, the mouse icon of the external device will also move in the corresponding area. When receiving an interaction event, operations such as clicking and dragging can also be performed.

[0106] In this implementation, after receiving the first control information, the external device can obtain the interaction information corresponding to the interaction event and the coordinate information of the associated position in the first display interface corresponding to the coordinate information of the interaction position based on the first control information; and perform operations based on the coordinate information of the associated position and the interaction information, thereby realizing the operation process of the external device based on the first control information.

[0107] See Figure 6 , is a schematic diagram of a framework of a control system provided in an embodiment of the present application, wherein the control system includes an XR terminal and a PC terminal, and the XR terminal includes:

[0108] Camera management module, used to open and read camera images.

[0109] AI module for display recognition;

[0110] 3D modeling module, used to build models of virtual interfaces;

[0111] The eye movement, gesture, and controller module is used to input eye movement, gesture, or controller interaction information, such as rays, pinching, and button presses;

[0112] 3D engine module, used to calculate the intersection of the virtual interface and the virtual rays of eye movement, gestures, and handles;

[0113] An interaction management module for identifying interaction operations on the PC;

[0114] The interconnection assistant module is used to manage the connection between the XR headset and the PC and transmit interaction information to the PC.

[0115] The above-mentioned PC terminals include:

[0116] The interconnection assistant module is used to manage the connection between the XR headset and the PC and receive interaction information;

[0117] The input driver module is used to simulate a mouse and convert the interactive data transmitted by the XR headset into mouse event input.

[0118] In some embodiments of the present application, the external device PC side can be identified based on the video perspective function of the XR side, that is, the wearable device, and the PC screen model is reconstructed through the 3D reconstruction algorithm to obtain the above-mentioned virtual interface. However, the 3D rendering engine calculates the coordinates of the intersection of the ray of the interactor on the XR side and the virtual interface, and converts the intersection coordinates into PC screen coordinates. The generated interaction command and the intersection coordinates are transmitted to the PC side, and the PC side reports the received interaction information to the application APP through the underlying driver, and the application APP responds, thereby converting the unique spatial operation mode of the XR side into the plane operation mode of the PC side. The above-mentioned XR side supports multi-touch, and users can use multi-finger operations to achieve more complex interactions, such as zooming, rotating and multi-tasking.

[0119] See Figure 7 , is a flow chart of a device interaction method provided in an embodiment of the present application, the method comprising the following steps:

[0120] Turning on the video perspective function, that is, turning on the wearable device and turning on the video perspective function in the wearable device;

[0121] AI identifies the PC in the field of view, that is, based on the AI module in the wearable device, identifies whether the image taken by the VST camera includes the first display interface of the PC;

[0122] The interconnection assistant connects to the PC, that is, when the first display interface of the PC is included in the field of view of the wearable device, a communication connection is established between the wearable device and the PC based on the interconnection assistant in the wearable device;

[0123] PC screen plane recognition, i.e., identifying the screen where the PC display is located and creating the virtual interface within the field of view;

[0124] Interaction processing, i.e. receiving a first input from a user in the virtual interface and generating first control information for controlling the PC;

[0125] Synchronously interacting with the PC, that is, sending the first control information to the PC, so that the PC performs a corresponding interactive action, thereby realizing a control process of the PC.

[0126] The device interaction method provided in the embodiment of the present application has at least the following beneficial effects:

[0127] Because it can accommodate a variety of interactive input methods, eye-tracking technology allows users to select or activate interface elements on the PC screen simply by looking at them. This reduces the number of physical operations and makes the experience more natural. Furthermore, users can use a virtual controller or gamepad in the XR environment to control the PC interface. This virtual controller or gamepad can simulate the functions of a mouse and keyboard, enhancing the user's operational experience.

[0128] It can be understood that the method provided in the embodiment of the present application is also applicable to controlling other devices with screens and screenless devices. Through quick connection and 3D reconstruction of the first display interface, the interactive information is captured, allowing users to perform tasks on other ends more intuitively and naturally through the air.

[0129] The device interaction method provided in the embodiment of the present application can be executed by a control device. In the embodiment of the present application, the control device provided in the embodiment of the present application is described by taking the control device executing the device interaction method as an example.

[0130] See Figure 8 , Figure 8 The present invention provides a structural diagram of a wearable device 800 according to an embodiment of the present invention. The wearable device 800 is applied to a wearable device 800, and the wearable device 800 includes:

[0131] a connection module 801 for establishing a communication connection between the wearable device 800 and the external device when the external environment image captured by the wearable device 800 includes a first display interface of the external device;

[0132] A creation module 802 is configured to create a virtual interface corresponding to the first display interface when the wearable device 800 is in a state where the video see-through function is turned on;

[0133] The generating module 803 is configured to enable the wearable device 800 to receive a first input from a user through the virtual interface and to generate first control information for controlling the external device.

[0134] Optionally, when the wearable device 800 displays the external environment image through the video perspective function, the virtual interface overlaps with the first display interface.

[0135] Optionally, the connection module 801 is specifically used for the wearable device 800 to turn on the video perspective function to capture the external environment image, and to establish a communication connection between the wearable device 800 and the external device when the external environment image includes the first display interface.

[0136] Optionally, the creation module 802 includes:

[0137] an acquisition submodule, configured to acquire external information collected by the wearable device 800, wherein the external information includes modeling information used to model the first display interface;

[0138] A creation submodule is configured to create the virtual interface based on the external information.

[0139] Optionally, the external information includes an image of the external environment captured by a camera in the wearable device 800; or,

[0140] The external information includes: an external environment image collected by a camera in the wearable device 800, and depth information corresponding to the external environment image collected by a depth sensor in the wearable device 800.

[0141] Optionally, the creating submodule includes:

[0142] a reconstruction unit, configured to input the external information into a three-dimensional reconstruction model for three-dimensional reconstruction, and obtain coordinate information output by the three-dimensional reconstruction model, wherein the coordinate information includes position coordinates corresponding to four vertices of the first display interface;

[0143] A creating unit is configured to create the virtual interface based on the coordinate information.

[0144] Optionally, the first input includes at least one of the following input methods: eye movement input, gesture input, and input based on a handle that is communicatively connected to the wearable device 800.

[0145] Optionally, the generating module 803 includes:

[0146] a determination submodule, configured to determine, based on the first input, an interaction position and an interaction event of the first input in the plane where the virtual interface is located;

[0147] A generating submodule is configured to generate the first control information based on the interaction position and the interaction event.

[0148] Optionally, the determining submodule is specifically configured to determine an extension direction of the virtual ray based on the indication direction of the first input, and determine an interaction position of the first input in the plane where the virtual interface is located; the interaction position is an intersection point of the virtual ray and the plane where the virtual interface is located;

[0149] In the case where the first input is an eye movement input, the extension direction of the virtual ray is the gaze direction of the user;

[0150] In the case where the first input is a gesture input, the extension direction of the virtual ray is the direction of the first position of the user's hand;

[0151] When the first input is a handle input, the extension direction of the virtual ray is the direction of the handle.

[0152] Optionally, the determining submodule is specifically configured to determine coordinate information of an interaction position of the first input in the plane where the virtual interface is located;

[0153] The determining submodule is specifically configured to determine the interaction event based on the first input, where the interaction event includes an eye movement, a gesture, or a handle interaction;

[0154] The generating submodule is specifically configured to generate the first control information based on the coordinate information of the interaction position and the interaction event.

[0155] Optionally, the external device receives the first control information, and based on the first control information, the external device obtains interaction information corresponding to the interaction event and coordinate information of an associated position in the first display interface corresponding to the coordinate information of the interaction position;

[0156] The external device performs an operation based on the coordinate information of the associated position and the interaction information.

[0157] In this embodiment, when the external environment image captured by the wearable device 800 includes a first display interface of an external device, a communication connection is established between the wearable device 800 and the external device, and when the wearable device 800 is in a state where the video perspective function is turned on, a virtual interface corresponding to the first display interface is created. In this way, when a user wearing the wearable device 800 needs to operate the external device, since the user can directly see the content displayed on the first display interface through the video perspective function, and the virtual interface corresponds to the first display interface, the user can directly input operation information for the external device through the virtual interface. The wearable device 800 generates corresponding first control information based on the operation information input by the user on the virtual interface, and sends the first control information to the external device, so that the external device responds to the first control information and performs the operation corresponding to the user input. In this process, since the user can directly operate the external device while wearing the wearable device 800, it is helpful to simplify the operation process of the user wearing the wearable device 800 on the external device.

[0158] The wearable device 800 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0159] The wearable device 800 provided in the embodiment of the present application can achieve Figure 1 The various processes implemented in the method embodiment achieve the same technical effect and will not be described again here to avoid repetition.

[0160] In some embodiments, as Figure 9 As shown, an embodiment of the present application also provides a wearable device 900, including a processor 901, a memory 902, and a program or instruction stored in the memory 902 and executable on the processor 901. When the program or instruction is executed by the processor 901, each process of the above-mentioned device interaction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0161] Figure 10 A schematic diagram of the hardware structure of a wearable device according to an embodiment of the present application.

[0162] The wearable device 1000 includes but is not limited to components such as a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010.

[0163] The network module 1002 is configured to establish a communication connection between the wearable device and the external device when the external environment image captured by the wearable device includes a first display interface of the external device;

[0164] The processor 1010 is configured to create a virtual interface corresponding to the first display interface when the wearable device is in a state where the video see-through function is turned on;

[0165] The processor 1010 is configured to enable the wearable device to receive a first input from a user through the virtual interface and generate first control information for controlling the external device.

[0166] Optionally, when the wearable device displays the external environment image through the video perspective function, the virtual interface overlaps with the first display interface.

[0167] Optionally, the network module 1002 is used for the wearable device to enable the video perspective function to capture the external environment image, and to establish a communication connection between the wearable device and the external device when the external environment image includes the first display interface.

[0168] Optionally, the processor 1010 is configured to obtain external information collected by the wearable device, wherein the external information includes modeling information used to model the first display interface;

[0169] The processor 1010 is configured to create the virtual interface based on the external information.

[0170] Optionally, the external information includes an image of the external environment captured by a camera in the wearable device; or

[0171] The external information includes: an external environment image collected by a camera in the wearable device, and depth information corresponding to the external environment image collected by a depth sensor in the wearable device.

[0172] Optionally, the processor 1010 is configured to input the external information into a three-dimensional reconstruction model for three-dimensional reconstruction, and obtain coordinate information output by the three-dimensional reconstruction model, wherein the coordinate information includes position coordinates corresponding to four vertices of the first display interface;

[0173] The processor 1010 is configured to create the virtual interface based on the coordinate information.

[0174] Optionally, the first input includes at least one of the following input modes: eye movement input, gesture input, and input based on a handle communicatively connected to the wearable device.

[0175] Optionally, the processor 1010 is configured to determine, based on the first input, an interaction position and an interaction event of the first input in the plane where the virtual interface is located;

[0176] The processor 1010 is configured to generate the first control information based on the interaction position and the interaction event.

[0177] Optionally, the processor 1010 is configured to determine an extension direction of a virtual ray based on the indication direction of the first input, and determine an interaction position of the first input in the plane where the virtual interface is located; the interaction position is an intersection of the virtual ray and the plane where the virtual interface is located;

[0178] In the case where the first input is an eye movement input, the extension direction of the virtual ray is the gaze direction of the user;

[0179] In the case where the first input is a gesture input, the extension direction of the virtual ray is the direction of the first position of the user's hand;

[0180] When the first input is a handle input, the extension direction of the virtual ray is the direction of the handle.

[0181] Optionally, the processor 1010 is configured to determine coordinate information of an interaction position of the first input in the plane where the virtual interface is located;

[0182] The processor 1010 is configured to determine the interaction event based on the first input, where the interaction event includes an eye movement, a gesture, or a handle interaction;

[0183] The processor 1010 is configured to generate the first control information based on the coordinate information of the interaction position and the interaction event.

[0184] Optionally, the external device receives the first control information, and based on the first control information, the external device obtains interaction information corresponding to the interaction event and coordinate information of an associated position in the first display interface corresponding to the coordinate information of the interaction position;

[0185] The external device performs an operation based on the coordinate information of the associated position and the interaction information.

[0186] Those skilled in the art will understand that the wearable device 1000 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 10 The wearable device structure shown in the figure does not constitute a limitation on the wearable device. The wearable device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0187] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0188] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0189] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0190] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned device interaction method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0191] The processor is the processor in the wearable device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0192] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned device interaction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0193] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0194] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0196] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A device interaction method, applied to a wearable device with a video see-through function, characterized in that: The method comprises: In a case where the external environment image captured by the wearable device includes a first display interface of an external device, establishing a communication connection between the wearable device and the external device; When the wearable device is in a state where the video see-through function is turned on, creating a virtual interface corresponding to the first display interface; The wearable device receives a first input from a user through the virtual interface and generates first control information for controlling the external device.

2. The device interaction method according to claim 1, characterized in that: When the wearable device displays the external environment image through the video perspective function, the virtual interface overlaps with the first display interface.

3. The device interaction method according to claim 1, characterized in that: The step of establishing a communication connection between the wearable device and the external device when the external environment image captured by the wearable device includes a first display interface of the external device includes: The wearable device turns on the video perspective function to capture the external environment image, and when the external environment image includes the first display interface, establishes a communication connection between the wearable device and the external device.

4. The device interaction method according to claim 1, wherein: The creating a virtual interface corresponding to the first display interface includes: Acquiring external information collected by the wearable device, wherein the external information includes modeling information used to model the first display interface; The virtual interface is created based on the external information.

5. The device interaction method according to claim 4, characterized in that: The creating the virtual interface based on the external information includes: Inputting the external information into a three-dimensional reconstruction model for three-dimensional reconstruction to obtain coordinate information output by the three-dimensional reconstruction model, wherein the coordinate information includes position coordinates corresponding to four vertices of the first display interface; The virtual interface is created based on the coordinate information.

6. The device interaction method according to claim 1, characterized in that: The first input includes at least one of the following input modes: eye movement input, gesture input, and input based on a handle communicatively connected to the wearable device.

7. The device interaction method according to claim 1, characterized in that: The generating of the first control information includes: Determining, based on the first input, an interaction position and an interaction event of the first input in the plane where the virtual interface is located; The first control information is generated based on the interaction position and the interaction event.

8. The device interaction method according to claim 7, characterized in that: The determining, based on the first input, an interaction position of the first input in the plane where the virtual interface is located includes: Determining an extension direction of a virtual ray based on the indication direction of the first input, and determining an interaction position of the first input in the plane where the virtual interface is located; the interaction position is an intersection point of the virtual ray and the plane where the virtual interface is located; In the case where the first input is an eye movement input, the extension direction of the virtual ray is the gaze direction of the user; In the case where the first input is a gesture input, the extension direction of the virtual ray is the direction of the first position of the user's hand; When the first input is a handle input, the extension direction of the virtual ray is the direction of the handle.

9. The device interaction method according to claim 7, characterized in that: The determining, based on the first input, an interaction position and an interaction event of the first input in the plane where the virtual interface is located includes: Determining coordinate information of an interaction position of the first input in the plane where the virtual interface is located; Determining the interaction event based on the first input, where the interaction event includes an eye movement, a gesture, or a handle interaction; The generating the first control information based on the interaction position and the interaction event includes: generating the first control information based on coordinate information of the interaction position and the interaction event.

10. The device interaction method according to any one of claims 1 to 9, characterized in that: The method further comprises: The external device receives the first control information, and based on the first control information, the external device obtains interaction information corresponding to the interaction event and coordinate information of an associated position in the first display interface corresponding to the coordinate information of the interaction position; The external device performs an operation based on the coordinate information of the associated position and the interaction information.

11. A wearable device, characterized in that: include: a connection module, configured to establish a communication connection between the wearable device and the external device when the external environment image captured by the wearable device includes a first display interface of the external device; A creation module, configured to create a virtual interface corresponding to the first display interface when the video see-through function of the wearable device is turned on; A generating module is used for the wearable device to receive a first input from a user through the virtual interface and generate first control information for controlling the external device.

12. The wearable device according to claim 11, wherein: When the wearable device displays the external environment image through the video perspective function, the virtual interface overlaps with the first display interface.

13. The wearable device according to claim 11, wherein: The connection module is specifically used for the wearable device to enable the video perspective function to capture the external environment image, and to establish a communication connection between the wearable device and the external device when the external environment image includes the first display interface.

14. The wearable device according to claim 11, wherein: The creation module includes: an acquisition submodule, configured to acquire external information collected by the wearable device, wherein the external information includes modeling information used to model the first display interface; A creation submodule is configured to create the virtual interface based on the external information.

15. The wearable device according to claim 14, wherein: The creation submodule includes: a reconstruction unit, configured to input the external information into a three-dimensional reconstruction model for three-dimensional reconstruction, and obtain coordinate information output by the three-dimensional reconstruction model, wherein the coordinate information includes position coordinates corresponding to four vertices of the first display interface; A creating unit is configured to create the virtual interface based on the coordinate information.

16. The wearable device according to claim 11, wherein: The generation module includes: a determination submodule, configured to determine, based on the first input, an interaction position and an interaction event of the first input in the plane where the virtual interface is located; A generating submodule is configured to generate the first control information based on the interaction position and the interaction event.

17. The wearable device according to claim 16, wherein: The determination submodule is specifically configured to determine an extension direction of the virtual ray based on the indication direction of the first input, and determine an interaction position of the first input in the plane where the virtual interface is located; the interaction position is an intersection point between the virtual ray and the plane where the virtual interface is located; In the case where the first input is an eye movement input, the extension direction of the virtual ray is the gaze direction of the user; In the case where the first input is a gesture input, the extension direction of the virtual ray is the direction of the first position of the user's hand; When the first input is a handle input, the extension direction of the virtual ray is the direction of the handle.

18. The wearable device according to claim 16, wherein: The determining submodule is specifically configured to determine coordinate information of an interaction position of the first input in the plane where the virtual interface is located; The determining submodule is specifically configured to determine the interaction event based on the first input, where the interaction event includes an eye movement, a gesture, or a handle interaction; The generating submodule is specifically configured to generate the first control information based on the coordinate information of the interaction position and the interaction event.

19. A readable storage medium, characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the device interaction method according to any one of claims 1 to 10 are implemented.

20. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the device interaction method according to any one of claims 1 to 10.

Images